Nov 01 15:44:59.180853 ip-10-0-128-238 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 01 15:44:59.180865 ip-10-0-128-238 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 01 15:44:59.180872 ip-10-0-128-238 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 01 15:44:59.181144 ip-10-0-128-238 systemd[1]: Failed to start Kubernetes Kubelet. Nov 01 15:45:09.205061 ip-10-0-128-238 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 01 15:45:09.205077 ip-10-0-128-238 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 35dfd4b2c3994bc28cda065fcccc4abc -- Nov 01 15:47:05.725190 ip-10-0-128-238 systemd[1]: Starting Kubernetes Kubelet... Nov 01 15:47:06.084488 ip-10-0-128-238 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 15:47:06.084488 ip-10-0-128-238 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 15:47:06.084488 ip-10-0-128-238 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 15:47:06.084488 ip-10-0-128-238 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 15:47:06.088370 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088290 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:47:06.088370 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088370 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:47:06.088370 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088374 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:47:06.088473 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088378 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:47:06.088473 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088381 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:47:06.088473 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088385 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:47:06.088473 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088389 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:47:06.088473 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088393 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:47:06.088473 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088397 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:47:06.088473 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088400 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:47:06.088473 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088404 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:47:06.088473 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088408 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:47:06.088473 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088411 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:47:06.088473 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088414 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:47:06.088473 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088417 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:47:06.088473 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088420 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:47:06.088473 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088422 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:47:06.088473 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088425 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:47:06.088473 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088427 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:47:06.088473 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088430 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:47:06.088473 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088432 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:47:06.088473 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088435 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:47:06.088914 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088437 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:47:06.088914 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088440 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:47:06.088914 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088443 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:47:06.088914 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088446 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:47:06.088914 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088448 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:47:06.088914 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088451 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:47:06.088914 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088453 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:47:06.088914 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088456 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:47:06.088914 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088458 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:47:06.088914 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088461 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:47:06.088914 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088463 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:47:06.088914 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088466 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:47:06.088914 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088468 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:47:06.088914 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088471 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:47:06.088914 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088475 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:47:06.088914 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088478 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:47:06.088914 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088480 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:47:06.088914 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088483 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:47:06.088914 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088486 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:47:06.089352 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088489 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:47:06.089352 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088492 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:47:06.089352 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088494 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:47:06.089352 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088497 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:47:06.089352 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088499 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:47:06.089352 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088502 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:47:06.089352 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088505 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:47:06.089352 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088507 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:47:06.089352 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088509 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:47:06.089352 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088512 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:47:06.089352 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088514 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:47:06.089352 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088517 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:47:06.089352 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088519 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:47:06.089352 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088522 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:47:06.089352 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088524 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:47:06.089352 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088526 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:47:06.089352 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088529 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:47:06.089352 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088532 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:47:06.089352 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088536 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:47:06.089352 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088538 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:47:06.089810 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088540 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:47:06.089810 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088543 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:47:06.089810 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088546 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:47:06.089810 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088548 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:47:06.089810 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088551 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:47:06.089810 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088553 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:47:06.089810 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088556 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:47:06.089810 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088559 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:47:06.089810 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088562 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:47:06.089810 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088565 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:47:06.089810 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088568 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:47:06.089810 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088571 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:47:06.089810 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088573 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:47:06.089810 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088583 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:47:06.089810 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088586 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:47:06.089810 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088588 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:47:06.089810 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088592 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:47:06.089810 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088594 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:47:06.089810 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088598 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:47:06.090264 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088600 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:47:06.090264 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088603 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:47:06.090264 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088606 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:47:06.090264 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088609 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:47:06.090264 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088611 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:47:06.090264 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088614 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:47:06.090264 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088617 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:47:06.090264 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088619 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:47:06.090264 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088622 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:47:06.090264 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088624 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:47:06.090264 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088627 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:47:06.090264 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088630 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:47:06.090264 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088633 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:47:06.090264 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088635 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:47:06.090264 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088638 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:47:06.090264 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088641 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:47:06.090264 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088643 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:47:06.090264 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088646 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:47:06.090264 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088649 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:47:06.090264 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088652 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:47:06.090739 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088655 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:47:06.090739 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088658 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:47:06.090739 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088660 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:47:06.090739 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088663 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:47:06.090739 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088665 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:47:06.090739 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.088668 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:47:06.090739 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089438 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:47:06.090739 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089447 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:47:06.090739 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089451 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:47:06.090739 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089455 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:47:06.090739 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089458 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:47:06.090739 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089460 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:47:06.090739 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089463 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:47:06.090739 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089466 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:47:06.090739 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089468 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:47:06.090739 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089471 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:47:06.090739 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089473 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:47:06.090739 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089484 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:47:06.090739 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089487 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:47:06.091216 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089490 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:47:06.091216 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089492 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:47:06.091216 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089495 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:47:06.091216 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089498 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:47:06.091216 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089506 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:47:06.091216 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089509 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:47:06.091216 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089511 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:47:06.091216 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089514 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:47:06.091216 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089517 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:47:06.091216 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089520 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:47:06.091216 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089522 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:47:06.091216 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089525 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:47:06.091216 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089528 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:47:06.091216 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089531 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:47:06.091216 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089533 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:47:06.091216 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089536 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:47:06.091216 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089539 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:47:06.091216 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089541 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:47:06.091216 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089544 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:47:06.091675 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089546 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:47:06.091675 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089549 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:47:06.091675 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089921 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:47:06.091675 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089925 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:47:06.091675 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089928 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:47:06.091675 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089931 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:47:06.091675 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089933 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:47:06.091675 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089936 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:47:06.091675 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089938 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:47:06.091675 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089941 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:47:06.091675 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089943 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:47:06.091675 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089946 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:47:06.091675 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089949 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:47:06.091675 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089952 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:47:06.091675 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089954 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:47:06.091675 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089956 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:47:06.091675 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089959 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:47:06.091675 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089961 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:47:06.091675 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089964 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:47:06.091675 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089967 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:47:06.092253 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089969 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:47:06.092253 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089972 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:47:06.092253 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089974 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:47:06.092253 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089977 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:47:06.092253 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089979 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:47:06.092253 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089982 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:47:06.092253 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089985 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:47:06.092253 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089987 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:47:06.092253 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089990 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:47:06.092253 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089993 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:47:06.092253 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089996 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:47:06.092253 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.089999 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:47:06.092253 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.090001 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:47:06.092253 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.090005 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:47:06.092253 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.090008 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:47:06.092253 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.090011 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:47:06.092253 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.090014 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:47:06.092253 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.090016 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:47:06.092253 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.090019 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:47:06.092253 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.090022 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:47:06.092716 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.090024 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:47:06.092716 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.090027 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:47:06.092716 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.090029 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:47:06.092716 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.090032 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:47:06.092716 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.090034 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:47:06.092716 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.090037 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:47:06.092716 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.090039 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:47:06.092716 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.090042 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:47:06.092716 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.090046 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:47:06.092716 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.090050 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:47:06.092716 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.090053 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:47:06.092716 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.090055 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:47:06.092716 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.090058 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:47:06.092716 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.090060 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:47:06.092716 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.090063 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:47:06.092716 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.090066 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:47:06.092716 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.090068 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:47:06.092716 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.090071 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:47:06.092716 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.090074 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:47:06.093214 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.090076 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:47:06.093214 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.090079 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:47:06.093214 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.090081 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:47:06.093214 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.090084 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:47:06.093214 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.090087 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:47:06.093214 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.090089 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:47:06.093214 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.090093 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:47:06.093214 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.090096 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:47:06.093214 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.090099 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:47:06.093214 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.090102 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:47:06.093214 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.090105 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:47:06.093214 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.090108 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:47:06.093214 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.090110 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:47:06.093214 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.090113 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:47:06.093214 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.090115 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:47:06.093214 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091206 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:47:06.093214 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091213 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:47:06.093214 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091216 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:47:06.093214 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091220 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:47:06.093214 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091223 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:47:06.093800 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091225 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:47:06.093800 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091228 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:47:06.093800 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091231 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:47:06.093800 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091234 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:47:06.093800 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091236 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:47:06.093800 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091239 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:47:06.093800 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091241 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:47:06.093800 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091244 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:47:06.093800 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091246 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:47:06.093800 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091249 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:47:06.093800 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091252 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:47:06.093800 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091255 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:47:06.093800 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091257 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:47:06.093800 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091260 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:47:06.093800 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091263 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:47:06.093800 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091265 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:47:06.093800 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091268 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:47:06.093800 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091270 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:47:06.093800 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091274 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:47:06.094493 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091276 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:47:06.094493 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091279 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:47:06.094493 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091284 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:47:06.094493 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091287 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:47:06.094493 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091290 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:47:06.094493 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091292 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:47:06.094493 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091295 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:47:06.094493 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091297 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:47:06.094493 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091300 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:47:06.094493 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091303 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:47:06.094493 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091305 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:47:06.094493 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091308 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:47:06.094493 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091310 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:47:06.094493 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091314 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:47:06.094493 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091318 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:47:06.094493 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091321 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:47:06.094493 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091324 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:47:06.094493 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091326 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:47:06.094493 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091329 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:47:06.094493 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091331 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:47:06.094986 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091334 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:47:06.094986 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091337 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:47:06.094986 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091339 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:47:06.094986 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091343 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:47:06.094986 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091345 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:47:06.094986 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091348 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:47:06.094986 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091351 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:47:06.094986 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091353 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:47:06.094986 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091356 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:47:06.094986 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091358 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:47:06.094986 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091362 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:47:06.094986 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091364 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:47:06.094986 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091366 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:47:06.094986 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091369 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:47:06.094986 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091371 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:47:06.094986 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091375 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:47:06.094986 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091377 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:47:06.094986 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091380 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:47:06.094986 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091382 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:47:06.094986 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091386 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:47:06.095480 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091388 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:47:06.095480 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091391 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:47:06.095480 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091393 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:47:06.095480 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091396 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:47:06.095480 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091398 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:47:06.095480 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091400 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:47:06.095480 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091403 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:47:06.095480 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091405 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:47:06.095480 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091408 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:47:06.095480 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091410 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:47:06.095480 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091413 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:47:06.095480 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091415 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:47:06.095480 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091418 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:47:06.095480 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091421 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:47:06.095480 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091423 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:47:06.095480 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091426 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:47:06.095480 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091429 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:47:06.095480 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091431 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:47:06.095480 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091434 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:47:06.095480 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091436 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:47:06.095959 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091439 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:47:06.095959 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091441 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:47:06.095959 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091443 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:47:06.095959 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091446 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:47:06.095959 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091448 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:47:06.095959 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091451 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:47:06.095959 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091453 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:47:06.095959 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091456 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:47:06.095959 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091458 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:47:06.095959 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091461 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:47:06.095959 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091463 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:47:06.095959 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091466 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:47:06.095959 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091468 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:47:06.095959 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091471 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:47:06.095959 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091473 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:47:06.095959 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091477 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:47:06.095959 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091480 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:47:06.095959 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091483 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:47:06.095959 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091486 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:47:06.096403 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091489 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:47:06.096403 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091492 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:47:06.096403 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.091494 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:47:06.096403 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092324 2576 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 15:47:06.096403 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092334 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 15:47:06.096403 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092341 2576 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 15:47:06.096403 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092346 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 15:47:06.096403 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092350 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 15:47:06.096403 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092354 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 15:47:06.096403 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092358 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 15:47:06.096403 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092363 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 15:47:06.096403 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092366 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 15:47:06.096403 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092369 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 15:47:06.096403 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092373 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 15:47:06.096403 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092376 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 15:47:06.096403 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092379 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 15:47:06.096403 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092382 2576 flags.go:64] FLAG: --cgroup-root="" Nov 01 15:47:06.096403 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092384 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 15:47:06.096403 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092388 2576 flags.go:64] FLAG: --client-ca-file="" Nov 01 15:47:06.096403 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092390 2576 flags.go:64] FLAG: --cloud-provider="external" Nov 01 15:47:06.096403 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092393 2576 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 15:47:06.096403 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092397 2576 flags.go:64] FLAG: --cluster-domain="" Nov 01 15:47:06.096403 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092400 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 15:47:06.096403 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092405 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 15:47:06.096958 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092409 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 15:47:06.096958 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092412 2576 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 15:47:06.096958 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092416 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 15:47:06.096958 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092419 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 15:47:06.096958 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092422 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 15:47:06.096958 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092425 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 15:47:06.096958 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092428 2576 flags.go:64] FLAG: --contention-profiling="false" Nov 01 15:47:06.096958 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092431 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 15:47:06.096958 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092434 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 15:47:06.096958 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092437 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 15:47:06.096958 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092440 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 15:47:06.096958 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092444 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 15:47:06.096958 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092447 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 15:47:06.096958 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092450 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 15:47:06.096958 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092453 2576 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 15:47:06.096958 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092458 2576 flags.go:64] FLAG: --enable-server="true" Nov 01 15:47:06.096958 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092461 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 15:47:06.096958 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092465 2576 flags.go:64] FLAG: --event-burst="100" Nov 01 15:47:06.096958 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092468 2576 flags.go:64] FLAG: --event-qps="50" Nov 01 15:47:06.096958 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092471 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 15:47:06.096958 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092474 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 15:47:06.096958 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092477 2576 flags.go:64] FLAG: --eviction-hard="" Nov 01 15:47:06.096958 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092486 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 15:47:06.096958 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092489 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 15:47:06.097581 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092492 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 15:47:06.097581 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092495 2576 flags.go:64] FLAG: --eviction-soft="" Nov 01 15:47:06.097581 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092498 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 15:47:06.097581 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092501 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 15:47:06.097581 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092504 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 15:47:06.097581 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092507 2576 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 15:47:06.097581 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092510 2576 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 15:47:06.097581 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092515 2576 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 15:47:06.097581 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092518 2576 flags.go:64] FLAG: --feature-gates="" Nov 01 15:47:06.097581 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092522 2576 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 15:47:06.097581 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092525 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 15:47:06.097581 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092528 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 15:47:06.097581 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092531 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 15:47:06.097581 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092534 2576 flags.go:64] FLAG: --healthz-port="10248" Nov 01 15:47:06.097581 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092537 2576 flags.go:64] FLAG: --help="false" Nov 01 15:47:06.097581 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092540 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-128-238.us-east-2.compute.internal" Nov 01 15:47:06.097581 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092543 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 15:47:06.097581 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092546 2576 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 15:47:06.097581 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092549 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 15:47:06.097581 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092553 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 15:47:06.097581 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092556 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 15:47:06.097581 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092559 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 15:47:06.097581 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092562 2576 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 15:47:06.098197 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092566 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 15:47:06.098197 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092569 2576 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 15:47:06.098197 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092572 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 15:47:06.098197 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092575 2576 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 15:47:06.098197 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092578 2576 flags.go:64] FLAG: --kube-reserved="" Nov 01 15:47:06.098197 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092581 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 15:47:06.098197 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092584 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 15:47:06.098197 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092587 2576 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 15:47:06.098197 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092590 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 15:47:06.098197 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092593 2576 flags.go:64] FLAG: --lock-file="" Nov 01 15:47:06.098197 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092595 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 15:47:06.098197 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092599 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 15:47:06.098197 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092602 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 15:47:06.098197 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092607 2576 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 15:47:06.098197 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092610 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 15:47:06.098197 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092613 2576 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 15:47:06.098197 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092617 2576 flags.go:64] FLAG: --logging-format="text" Nov 01 15:47:06.098197 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092620 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 15:47:06.098197 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092624 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 15:47:06.098197 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092627 2576 flags.go:64] FLAG: --manifest-url="" Nov 01 15:47:06.098197 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092630 2576 flags.go:64] FLAG: --manifest-url-header="" Nov 01 15:47:06.098197 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092634 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 15:47:06.098197 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092637 2576 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 15:47:06.098197 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092641 2576 flags.go:64] FLAG: --max-pods="110" Nov 01 15:47:06.098197 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092644 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 15:47:06.098810 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092647 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 15:47:06.098810 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092649 2576 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 15:47:06.098810 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092652 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 15:47:06.098810 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092655 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 15:47:06.098810 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092658 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 15:47:06.098810 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092661 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 15:47:06.098810 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092668 2576 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 15:47:06.098810 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092672 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 15:47:06.098810 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092675 2576 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 15:47:06.098810 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092678 2576 flags.go:64] FLAG: --pod-cidr="" Nov 01 15:47:06.098810 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092681 2576 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 15:47:06.098810 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092684 2576 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 15:47:06.098810 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092687 2576 flags.go:64] FLAG: --pods-per-core="0" Nov 01 15:47:06.098810 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092690 2576 flags.go:64] FLAG: --port="10250" Nov 01 15:47:06.098810 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092693 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 15:47:06.098810 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092696 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-07a90eca50fbde4de" Nov 01 15:47:06.098810 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092699 2576 flags.go:64] FLAG: --qos-reserved="" Nov 01 15:47:06.098810 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092702 2576 flags.go:64] FLAG: --read-only-port="10255" Nov 01 15:47:06.098810 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092705 2576 flags.go:64] FLAG: --register-node="true" Nov 01 15:47:06.098810 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092708 2576 flags.go:64] FLAG: --register-with-taints="" Nov 01 15:47:06.098810 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092711 2576 flags.go:64] FLAG: --registry-burst="10" Nov 01 15:47:06.098810 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092714 2576 flags.go:64] FLAG: --registry-qps="5" Nov 01 15:47:06.098810 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092718 2576 flags.go:64] FLAG: --reserved-cpus="" Nov 01 15:47:06.098810 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092722 2576 flags.go:64] FLAG: --reserved-memory="" Nov 01 15:47:06.098810 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092726 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 15:47:06.098810 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092730 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 15:47:06.099457 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092733 2576 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 15:47:06.099457 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092736 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 15:47:06.099457 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092739 2576 flags.go:64] FLAG: --runonce="false" Nov 01 15:47:06.099457 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092742 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 15:47:06.099457 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092745 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 15:47:06.099457 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092748 2576 flags.go:64] FLAG: --seccomp-default="false" Nov 01 15:47:06.099457 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092750 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 15:47:06.099457 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092753 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 15:47:06.099457 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092756 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 15:47:06.099457 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092759 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 15:47:06.099457 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092762 2576 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 15:47:06.099457 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092765 2576 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 15:47:06.099457 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092768 2576 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 15:47:06.099457 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092772 2576 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 15:47:06.099457 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092775 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 15:47:06.099457 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092778 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 15:47:06.099457 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092781 2576 flags.go:64] FLAG: --system-cgroups="" Nov 01 15:47:06.099457 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092785 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 15:47:06.099457 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092790 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 15:47:06.099457 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092793 2576 flags.go:64] FLAG: --tls-cert-file="" Nov 01 15:47:06.099457 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092796 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 15:47:06.099457 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092799 2576 flags.go:64] FLAG: --tls-min-version="" Nov 01 15:47:06.099457 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092802 2576 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 15:47:06.099457 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092805 2576 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 15:47:06.099457 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092808 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 15:47:06.100060 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092811 2576 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 15:47:06.100060 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092814 2576 flags.go:64] FLAG: --v="2" Nov 01 15:47:06.100060 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092818 2576 flags.go:64] FLAG: --version="false" Nov 01 15:47:06.100060 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092822 2576 flags.go:64] FLAG: --vmodule="" Nov 01 15:47:06.100060 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092828 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 15:47:06.100060 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.092831 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 15:47:06.100060 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093662 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:47:06.100060 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093686 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:47:06.100060 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093692 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:47:06.100060 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093696 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:47:06.100060 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093707 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:47:06.100060 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093711 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:47:06.100060 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093717 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:47:06.100060 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093721 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:47:06.100060 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093725 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:47:06.100060 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093729 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:47:06.100060 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093734 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:47:06.100060 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093741 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:47:06.100060 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093747 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:47:06.100060 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093752 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:47:06.100060 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093761 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:47:06.100559 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093765 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:47:06.100559 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093776 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:47:06.100559 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093780 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:47:06.100559 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093785 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:47:06.100559 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093789 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:47:06.100559 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093793 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:47:06.100559 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093797 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:47:06.100559 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093802 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:47:06.100559 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093808 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:47:06.100559 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093815 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:47:06.100559 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093819 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:47:06.100559 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093824 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:47:06.100559 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093828 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:47:06.100559 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093832 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:47:06.100559 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093841 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:47:06.100559 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093846 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:47:06.100559 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093851 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:47:06.100559 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093855 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:47:06.100559 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093859 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:47:06.100559 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093864 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:47:06.101043 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093868 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:47:06.101043 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093873 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:47:06.101043 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093877 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:47:06.101043 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093881 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:47:06.101043 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093885 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:47:06.101043 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093889 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:47:06.101043 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093899 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:47:06.101043 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093918 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:47:06.101043 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093923 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:47:06.101043 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093927 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:47:06.101043 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093932 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:47:06.101043 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093937 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:47:06.101043 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093942 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:47:06.101043 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093945 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:47:06.101043 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093948 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:47:06.101043 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093951 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:47:06.101043 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.093987 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:47:06.101043 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094007 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:47:06.101043 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094013 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:47:06.101043 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094021 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:47:06.101517 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094027 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:47:06.101517 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094032 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:47:06.101517 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094037 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:47:06.101517 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094041 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:47:06.101517 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094061 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:47:06.101517 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094067 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:47:06.101517 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094071 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:47:06.101517 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094074 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:47:06.101517 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094077 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:47:06.101517 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094080 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:47:06.101517 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094223 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:47:06.101517 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094228 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:47:06.101517 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094230 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:47:06.101517 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094233 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:47:06.101517 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094236 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:47:06.101517 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094239 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:47:06.101517 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094242 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:47:06.101517 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094245 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:47:06.101517 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094247 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:47:06.101517 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094250 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:47:06.102036 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094252 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:47:06.102036 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094255 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:47:06.102036 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094258 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:47:06.102036 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094261 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:47:06.102036 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094263 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:47:06.102036 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094266 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:47:06.102036 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094268 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:47:06.102036 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094271 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:47:06.102036 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094273 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:47:06.102036 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094277 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:47:06.102036 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094279 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:47:06.102036 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094282 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:47:06.102036 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094284 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:47:06.102036 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094287 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:47:06.102036 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094289 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:47:06.102036 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094292 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:47:06.102036 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094295 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:47:06.102036 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094297 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:47:06.102036 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094300 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:47:06.102498 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094302 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:47:06.102498 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094305 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:47:06.102498 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094308 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:47:06.102498 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094310 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:47:06.102498 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094313 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:47:06.102498 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094316 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:47:06.102498 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094319 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:47:06.102498 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094322 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:47:06.102498 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094324 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:47:06.102498 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094327 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:47:06.102498 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094329 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:47:06.102498 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094332 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:47:06.102498 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.094368 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 15:47:06.102498 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094421 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:47:06.102498 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094426 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:47:06.102498 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094430 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:47:06.102860 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094433 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:47:06.102860 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094435 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:47:06.102860 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094438 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:47:06.102860 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094441 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:47:06.102860 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094444 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:47:06.102860 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094446 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:47:06.102860 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094449 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:47:06.102860 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094452 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:47:06.102860 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094454 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:47:06.102860 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094457 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:47:06.102860 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094460 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:47:06.102860 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094463 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:47:06.102860 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094465 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:47:06.102860 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094468 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:47:06.102860 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094470 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:47:06.102860 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094473 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:47:06.102860 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094476 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:47:06.102860 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094478 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:47:06.102860 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094480 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:47:06.102860 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094483 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:47:06.103345 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094487 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:47:06.103345 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094489 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:47:06.103345 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094492 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:47:06.103345 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094494 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:47:06.103345 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094497 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:47:06.103345 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094500 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:47:06.103345 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094503 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:47:06.103345 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094506 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:47:06.103345 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094508 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:47:06.103345 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094511 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:47:06.103345 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094514 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:47:06.103345 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094516 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:47:06.103345 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094519 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:47:06.103345 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094521 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:47:06.103345 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094523 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:47:06.103345 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094526 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:47:06.103345 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094529 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:47:06.103345 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094531 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:47:06.103345 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094534 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:47:06.103779 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094536 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:47:06.103779 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094539 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:47:06.103779 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094542 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:47:06.103779 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094544 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:47:06.103779 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094547 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:47:06.103779 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094550 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:47:06.103779 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094553 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:47:06.103779 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094555 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:47:06.103779 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094558 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:47:06.103779 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094561 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:47:06.103779 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094563 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:47:06.103779 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094566 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:47:06.103779 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094568 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:47:06.103779 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094571 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:47:06.103779 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094573 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:47:06.103779 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094575 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:47:06.103779 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094578 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:47:06.103779 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094580 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:47:06.103779 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094583 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:47:06.103779 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094585 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:47:06.104287 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094588 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:47:06.104287 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094590 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:47:06.104287 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094593 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:47:06.104287 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094598 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:47:06.104287 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094600 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:47:06.104287 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094603 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:47:06.104287 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094606 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:47:06.104287 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094608 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:47:06.104287 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094611 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:47:06.104287 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094613 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:47:06.104287 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094616 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:47:06.104287 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094618 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:47:06.104287 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094621 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:47:06.104287 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094624 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:47:06.104287 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094627 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:47:06.104287 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094630 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:47:06.104287 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094633 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:47:06.104287 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094635 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:47:06.104287 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094637 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:47:06.104732 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094640 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:47:06.104732 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094644 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:47:06.104732 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094647 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:47:06.104732 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094649 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:47:06.104732 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094652 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:47:06.104732 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094656 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:47:06.104732 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094659 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:47:06.104732 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094662 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:47:06.104732 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094665 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:47:06.104732 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094667 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:47:06.104732 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094670 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:47:06.104732 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094673 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:47:06.104732 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094675 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:47:06.104732 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094678 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:47:06.104732 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094681 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:47:06.104732 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094683 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:47:06.104732 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094687 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:47:06.104732 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094692 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:47:06.104732 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094696 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:47:06.105190 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094699 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:47:06.105190 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094701 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:47:06.105190 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094704 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:47:06.105190 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094707 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:47:06.105190 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094709 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:47:06.105190 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.094712 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:47:06.105190 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.101048 2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 15:47:06.105190 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.101065 2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 15:47:06.105190 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.101082 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 15:47:06.105190 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.101086 2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 15:47:06.105190 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101141 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:47:06.105190 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101147 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:47:06.105190 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101150 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:47:06.105190 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101154 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:47:06.105190 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101156 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:47:06.105190 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101160 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:47:06.105190 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101163 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:47:06.105190 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101166 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:47:06.105190 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101169 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:47:06.105190 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101172 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:47:06.105663 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101174 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:47:06.105663 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101177 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:47:06.105663 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101180 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:47:06.105663 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101183 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:47:06.105663 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101185 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:47:06.105663 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101188 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:47:06.105663 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101191 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:47:06.105663 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101193 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:47:06.105663 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101196 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:47:06.105663 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101199 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:47:06.105663 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101201 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:47:06.105663 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101204 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:47:06.105663 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101207 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:47:06.105663 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101210 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:47:06.105663 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101212 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:47:06.105663 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101215 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:47:06.105663 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101218 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:47:06.105663 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101221 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:47:06.105663 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101224 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:47:06.105663 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101227 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:47:06.106130 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101230 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:47:06.106130 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101232 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:47:06.106130 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101235 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:47:06.106130 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101237 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:47:06.106130 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101240 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:47:06.106130 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101243 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:47:06.106130 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101245 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:47:06.106130 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101248 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:47:06.106130 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101250 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:47:06.106130 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101253 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:47:06.106130 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101256 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:47:06.106130 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101258 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:47:06.106130 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101261 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:47:06.106130 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101263 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:47:06.106130 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101266 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:47:06.106130 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101270 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:47:06.106130 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101274 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:47:06.106130 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101277 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:47:06.106130 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101280 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:47:06.106572 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101283 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:47:06.106572 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101285 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:47:06.106572 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101288 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:47:06.106572 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101290 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:47:06.106572 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101294 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:47:06.106572 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101296 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:47:06.106572 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101299 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:47:06.106572 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101302 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:47:06.106572 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101304 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:47:06.106572 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101307 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:47:06.106572 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101310 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:47:06.106572 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101313 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:47:06.106572 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101316 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:47:06.106572 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101318 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:47:06.106572 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101322 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:47:06.106572 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101324 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:47:06.106572 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101327 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:47:06.106572 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101329 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:47:06.106572 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101332 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:47:06.106572 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101335 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:47:06.107056 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101337 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:47:06.107056 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101340 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:47:06.107056 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101342 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:47:06.107056 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101345 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:47:06.107056 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101348 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:47:06.107056 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101350 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:47:06.107056 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101353 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:47:06.107056 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101355 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:47:06.107056 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101359 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:47:06.107056 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101362 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:47:06.107056 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101365 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:47:06.107056 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101368 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:47:06.107056 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101371 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:47:06.107056 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101373 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:47:06.107056 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101376 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:47:06.107056 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101378 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:47:06.107056 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101381 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:47:06.107056 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101384 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:47:06.107056 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101386 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:47:06.107498 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101389 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:47:06.107498 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101392 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:47:06.107498 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101395 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:47:06.107498 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101397 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:47:06.107498 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101400 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:47:06.107498 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101402 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:47:06.107498 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101405 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:47:06.107498 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101407 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:47:06.107498 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101410 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:47:06.107498 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101413 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:47:06.107498 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101415 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:47:06.107498 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101418 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:47:06.107498 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101420 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:47:06.107498 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101422 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:47:06.107498 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101425 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:47:06.107498 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101428 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:47:06.107498 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101430 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:47:06.107498 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101432 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:47:06.107920 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.101461 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 15:47:06.107920 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101601 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:47:06.107920 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101607 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:47:06.107920 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101610 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:47:06.107920 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101613 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:47:06.107920 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101615 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:47:06.107920 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101618 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:47:06.107920 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101620 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:47:06.107920 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101623 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:47:06.107920 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101625 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:47:06.107920 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101628 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:47:06.107920 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101631 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:47:06.107920 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101633 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:47:06.107920 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101636 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:47:06.107920 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101638 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:47:06.107920 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101641 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:47:06.108297 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101643 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:47:06.108297 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101646 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:47:06.108297 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101648 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:47:06.108297 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101651 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:47:06.108297 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101653 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:47:06.108297 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101656 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:47:06.108297 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101658 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:47:06.108297 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101661 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:47:06.108297 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101664 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:47:06.108297 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101667 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:47:06.108297 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101669 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:47:06.108297 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101672 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:47:06.108297 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101674 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:47:06.108297 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101677 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:47:06.108297 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101679 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:47:06.108297 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101682 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:47:06.108297 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101684 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:47:06.108297 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101687 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:47:06.108297 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101689 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:47:06.108297 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101692 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:47:06.108764 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101694 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:47:06.108764 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101697 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:47:06.108764 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101699 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:47:06.108764 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101702 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:47:06.108764 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101705 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:47:06.108764 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101707 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:47:06.108764 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101710 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:47:06.108764 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101712 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:47:06.108764 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101715 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:47:06.108764 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101717 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:47:06.108764 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101719 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:47:06.108764 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101722 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:47:06.108764 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101724 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:47:06.108764 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101727 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:47:06.108764 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101730 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:47:06.108764 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101732 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:47:06.108764 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101735 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:47:06.108764 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101737 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:47:06.108764 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101740 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:47:06.108764 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101742 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:47:06.109226 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101749 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:47:06.109226 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101752 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:47:06.109226 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101755 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:47:06.109226 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101757 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:47:06.109226 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101760 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:47:06.109226 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101762 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:47:06.109226 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101764 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:47:06.109226 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101767 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:47:06.109226 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101770 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:47:06.109226 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101772 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:47:06.109226 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101776 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:47:06.109226 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101781 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:47:06.109226 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101784 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:47:06.109226 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101787 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:47:06.109226 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101789 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:47:06.109226 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101792 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:47:06.109226 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101794 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:47:06.109226 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101797 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:47:06.109226 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101799 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:47:06.109683 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101802 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:47:06.109683 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101804 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:47:06.109683 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101806 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:47:06.109683 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101809 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:47:06.109683 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101812 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:47:06.109683 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101814 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:47:06.109683 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101817 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:47:06.109683 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101821 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:47:06.109683 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101824 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:47:06.109683 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101826 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:47:06.109683 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101829 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:47:06.109683 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101831 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:47:06.109683 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101834 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:47:06.109683 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101836 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:47:06.109683 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101839 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:47:06.109683 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101842 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:47:06.109683 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101844 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:47:06.109683 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101847 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:47:06.109683 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101849 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:47:06.110189 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101851 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:47:06.110189 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101854 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:47:06.110189 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101856 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:47:06.110189 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101860 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:47:06.110189 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101863 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:47:06.110189 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101866 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:47:06.110189 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101868 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:47:06.110189 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101871 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:47:06.110189 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101873 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:47:06.110189 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101876 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:47:06.110189 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101878 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:47:06.110189 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101880 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:47:06.110189 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101883 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:47:06.110189 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.101919 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 15:47:06.110189 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101950 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:47:06.110550 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101954 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:47:06.110550 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101957 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:47:06.110550 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101960 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:47:06.110550 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101962 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:47:06.110550 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101965 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:47:06.110550 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101970 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:47:06.110550 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101972 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:47:06.110550 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101975 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:47:06.110550 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101977 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:47:06.110550 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101980 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:47:06.110550 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101982 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:47:06.110550 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101985 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:47:06.110550 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101988 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:47:06.110550 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101991 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:47:06.110550 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101993 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:47:06.110550 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101996 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:47:06.110550 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.101998 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:47:06.110550 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102001 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:47:06.110550 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102003 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:47:06.110550 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102006 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:47:06.111014 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102009 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:47:06.111014 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102011 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:47:06.111014 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102014 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:47:06.111014 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102016 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:47:06.111014 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102019 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:47:06.111014 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102021 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:47:06.111014 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102024 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:47:06.111014 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102026 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:47:06.111014 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102029 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:47:06.111014 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102032 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:47:06.111014 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102034 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:47:06.111014 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102037 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:47:06.111014 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102039 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:47:06.111014 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102042 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:47:06.111014 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102045 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:47:06.111014 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102047 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:47:06.111014 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102051 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:47:06.111014 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102055 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:47:06.111014 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102058 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:47:06.111445 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102060 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:47:06.111445 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102063 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:47:06.111445 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102065 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:47:06.111445 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102068 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:47:06.111445 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102070 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:47:06.111445 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102073 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:47:06.111445 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102076 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:47:06.111445 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102080 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:47:06.111445 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102085 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:47:06.111445 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102088 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:47:06.111445 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102092 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:47:06.111445 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102094 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:47:06.111445 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102097 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:47:06.111445 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102099 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:47:06.111445 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102102 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:47:06.111445 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102104 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:47:06.111445 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102107 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:47:06.111445 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102109 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:47:06.111445 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102112 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:47:06.111932 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102114 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:47:06.111932 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102117 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:47:06.111932 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102119 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:47:06.111932 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102122 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:47:06.111932 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102124 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:47:06.111932 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102127 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:47:06.111932 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102129 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:47:06.111932 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102132 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:47:06.111932 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102134 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:47:06.111932 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102138 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:47:06.111932 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102142 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:47:06.111932 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102146 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:47:06.111932 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102148 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:47:06.111932 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102151 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:47:06.111932 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102153 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:47:06.111932 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102156 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:47:06.111932 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102158 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:47:06.111932 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102161 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:47:06.111932 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102163 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:47:06.112373 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102167 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:47:06.112373 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102171 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:47:06.112373 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102175 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:47:06.112373 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102178 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:47:06.112373 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102180 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:47:06.112373 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102183 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:47:06.112373 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102185 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:47:06.112373 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102188 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:47:06.112373 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102190 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:47:06.112373 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102193 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:47:06.112373 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102196 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:47:06.112373 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102198 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:47:06.112373 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102201 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:47:06.112373 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102204 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:47:06.112373 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102206 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:47:06.112373 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102209 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:47:06.112373 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102211 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:47:06.112373 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102214 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:47:06.112373 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102216 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:47:06.112837 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102219 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:47:06.112837 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102221 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:47:06.112837 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102224 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:47:06.112837 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102227 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:47:06.112837 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102231 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:47:06.112837 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102235 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:47:06.112837 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102237 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:47:06.112837 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102240 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:47:06.112837 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:06.102242 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:47:06.112837 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.102292 2576 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 15:47:06.112837 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.104139 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 01 15:47:06.112837 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.105086 2576 server.go:1014] "Starting client certificate rotation" Nov 01 15:47:06.112837 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.105175 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 15:47:06.112837 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.105216 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 15:47:06.112837 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.106816 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 15:47:06.112837 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.108869 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 15:47:06.121938 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.121923 2576 log.go:25] "Validated CRI v1 runtime API" Nov 01 15:47:06.126057 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.126041 2576 log.go:25] "Validated CRI v1 image API" Nov 01 15:47:06.126867 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.126845 2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 15:47:06.131395 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.131371 2576 fs.go:135] Filesystem UUIDs: map[49d632c8-b5c9-4292-98d3-a47e3e65bfa5:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9df08b24-6cad-4ca2-b0d4-7392efcf0dec:/dev/nvme0n1p4] Nov 01 15:47:06.131463 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.131392 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 01 15:47:06.132939 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.132916 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 15:47:06.136988 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.136867 2576 manager.go:217] Machine: {Timestamp:2026-11-01 15:47:06.135120997 +0000 UTC m=+0.308397504 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100935 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28d0b073fd7323338be4b86c915ee2 SystemUUID:ec28d0b0-73fd-7323-338b-e4b86c915ee2 BootID:35dfd4b2-c399-4bc2-8cda-065fcccc4abc Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e4:d4:00:dd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e4:d4:00:dd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ca:22:96:65:d6:5c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 15:47:06.136988 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.136986 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 15:47:06.137104 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.137082 2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 15:47:06.139218 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.139194 2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 15:47:06.139354 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.139221 2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-238.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 15:47:06.139395 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.139356 2576 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 15:47:06.139395 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.139365 2576 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 15:47:06.139454 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.139402 2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 15:47:06.139454 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.139417 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 15:47:06.140100 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.140091 2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 15:47:06.143531 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.143520 2576 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 15:47:06.143658 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.143648 2576 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 15:47:06.143688 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.143681 2576 kubelet.go:493] "Attempting to sync node with API server" Nov 01 15:47:06.143721 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.143696 2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 15:47:06.143721 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.143714 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 15:47:06.143770 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.143728 2576 kubelet.go:405] "Adding apiserver pod source" Nov 01 15:47:06.143770 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.143740 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 15:47:06.144152 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.144140 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 15:47:06.146008 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.145991 2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 15:47:06.150314 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.150297 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 15:47:06.150405 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.150376 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xbzcz" Nov 01 15:47:06.151016 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.151003 2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 15:47:06.151050 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.151034 2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 15:47:06.151187 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.151177 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 15:47:06.151216 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.151192 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 15:47:06.151216 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.151198 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 15:47:06.151216 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.151204 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 15:47:06.151216 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.151210 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 15:47:06.151216 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.151216 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 15:47:06.151783 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.151775 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 15:47:06.151812 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.151784 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 15:47:06.151812 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.151792 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 15:47:06.151812 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.151798 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 15:47:06.151971 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.151836 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 15:47:06.151971 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.151847 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 15:47:06.152386 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.152377 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 15:47:06.152386 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.152386 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 15:47:06.156130 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.156115 2576 server.go:1263] "Started kubelet" Nov 01 15:47:06.156207 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.156165 2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 15:47:06.156623 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.156545 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 15:47:06.156721 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.156635 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 15:47:06.156814 ip-10-0-128-238 systemd[1]: Started Kubernetes Kubelet. Nov 01 15:47:06.157543 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.157526 2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 15:47:06.158037 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.158018 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xbzcz" Nov 01 15:47:06.158037 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.158035 2576 server.go:317] "Adding debug handlers to kubelet server" Nov 01 15:47:06.164490 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:06.164471 2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 01 15:47:06.164983 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:06.163697 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-238.us-east-2.compute.internal.18e3f66d5aa2eb49 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-238.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-238.us-east-2.compute.internal,},FirstTimestamp:2026-11-01 15:47:06.156084041 +0000 UTC m=+0.329360547,LastTimestamp:2026-11-01 15:47:06.156084041 +0000 UTC m=+0.329360547,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-238.us-east-2.compute.internal,}" Nov 01 15:47:06.166645 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.166624 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 15:47:06.167039 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.167022 2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 15:47:06.167138 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:06.167103 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-238.us-east-2.compute.internal\" not found" Nov 01 15:47:06.168014 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.167783 2576 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 15:47:06.168014 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.167799 2576 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 15:47:06.168014 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.167897 2576 reconstruct.go:97] "Volume reconstruction finished" Nov 01 15:47:06.168014 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.167918 2576 reconciler.go:29] "Reconciler: start to sync state" Nov 01 15:47:06.168691 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.168672 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 15:47:06.168779 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.168771 2576 factory.go:55] Registering systemd factory Nov 01 15:47:06.168842 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.168786 2576 factory.go:223] Registration of the systemd container factory successfully Nov 01 15:47:06.169171 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.169157 2576 factory.go:153] Registering CRI-O factory Nov 01 15:47:06.169244 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.169174 2576 factory.go:223] Registration of the crio container factory successfully Nov 01 15:47:06.169244 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.169222 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 15:47:06.169244 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.169239 2576 factory.go:103] Registering Raw factory Nov 01 15:47:06.169382 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.169252 2576 manager.go:1199] Started watching for new ooms in manager Nov 01 15:47:06.170117 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.170103 2576 manager.go:319] Starting recovery of all containers Nov 01 15:47:06.174693 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.174667 2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 15:47:06.174961 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.174940 2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 15:47:06.175044 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.174966 2576 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-128-238.us-east-2.compute.internal" not found Nov 01 15:47:06.176101 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.176081 2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 15:47:06.177033 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:06.177007 2576 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-128-238.us-east-2.compute.internal\" not found" node="ip-10-0-128-238.us-east-2.compute.internal" Nov 01 15:47:06.179827 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.179812 2576 manager.go:324] Recovery completed Nov 01 15:47:06.187277 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.187261 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 15:47:06.188698 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.188682 2576 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-128-238.us-east-2.compute.internal" not found Nov 01 15:47:06.189318 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.189301 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-238.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 15:47:06.189397 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.189334 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-238.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 15:47:06.189397 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.189348 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-238.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 15:47:06.189894 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.189879 2576 cpu_manager.go:226] "Starting" policy="none" Nov 01 15:47:06.189992 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.189892 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 15:47:06.189992 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.189890 2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 15:47:06.189992 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.189934 2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 15:47:06.190706 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.190690 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-238.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 15:47:06.190777 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.190719 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-238.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 15:47:06.190777 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.190733 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-238.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 15:47:06.191892 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.191881 2576 policy_none.go:50] "Start" Nov 01 15:47:06.191951 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.191896 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 15:47:06.191951 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.191920 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 15:47:06.193842 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.193827 2576 policy_none.go:44] "Start" Nov 01 15:47:06.219474 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.219450 2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 15:47:06.221796 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.221784 2576 manager.go:341] "Starting Device Plugin manager" Nov 01 15:47:06.221845 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:06.221810 2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 15:47:06.221845 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.221818 2576 server.go:87] "Starting device plugin registration server" Nov 01 15:47:06.222075 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.222063 2576 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 15:47:06.222131 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.222075 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 15:47:06.222212 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.222199 2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 15:47:06.222343 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.222277 2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 15:47:06.222343 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.222288 2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 15:47:06.222686 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:06.222665 2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 15:47:06.222686 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:06.222713 2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-238.us-east-2.compute.internal\" not found" Nov 01 15:47:06.242680 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.242663 2576 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-128-238.us-east-2.compute.internal" not found Nov 01 15:47:06.307931 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.307893 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 15:47:06.309456 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.309439 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 15:47:06.309545 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.309462 2576 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 15:47:06.309931 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.309901 2576 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 15:47:06.310010 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:06.309955 2576 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 01 15:47:06.313039 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.313025 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 15:47:06.323143 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.323128 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 15:47:06.327525 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.327510 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-238.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 15:47:06.327612 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.327544 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-238.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 15:47:06.327612 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.327560 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-238.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 15:47:06.327612 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.327593 2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-238.us-east-2.compute.internal" Nov 01 15:47:06.335122 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.335075 2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-238.us-east-2.compute.internal" Nov 01 15:47:06.335122 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:06.335101 2576 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-238.us-east-2.compute.internal\": node \"ip-10-0-128-238.us-east-2.compute.internal\" not found" Nov 01 15:47:06.410955 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.410927 2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-238.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-238.us-east-2.compute.internal"] Nov 01 15:47:06.413223 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.413209 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-238.us-east-2.compute.internal" Nov 01 15:47:06.415112 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.415093 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-238.us-east-2.compute.internal" Nov 01 15:47:06.447459 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.447308 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-238.us-east-2.compute.internal" Nov 01 15:47:06.450096 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.450080 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-238.us-east-2.compute.internal" Nov 01 15:47:06.458260 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.458245 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 15:47:06.458478 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.458467 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 15:47:06.570162 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.570138 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ae861e52d07eb0e1937b2b22576e5cfd-config\") pod \"kube-apiserver-proxy-ip-10-0-128-238.us-east-2.compute.internal\" (UID: \"ae861e52d07eb0e1937b2b22576e5cfd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-238.us-east-2.compute.internal" Nov 01 15:47:06.570294 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.570165 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5d3a91f23ccfbb9c1af1cd6d04b30fd6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-238.us-east-2.compute.internal\" (UID: \"5d3a91f23ccfbb9c1af1cd6d04b30fd6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-238.us-east-2.compute.internal" Nov 01 15:47:06.570294 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.570184 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d3a91f23ccfbb9c1af1cd6d04b30fd6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-238.us-east-2.compute.internal\" (UID: \"5d3a91f23ccfbb9c1af1cd6d04b30fd6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-238.us-east-2.compute.internal" Nov 01 15:47:06.670513 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.670454 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5d3a91f23ccfbb9c1af1cd6d04b30fd6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-238.us-east-2.compute.internal\" (UID: \"5d3a91f23ccfbb9c1af1cd6d04b30fd6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-238.us-east-2.compute.internal" Nov 01 15:47:06.670625 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.670538 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d3a91f23ccfbb9c1af1cd6d04b30fd6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-238.us-east-2.compute.internal\" (UID: \"5d3a91f23ccfbb9c1af1cd6d04b30fd6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-238.us-east-2.compute.internal" Nov 01 15:47:06.670625 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.670540 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d3a91f23ccfbb9c1af1cd6d04b30fd6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-238.us-east-2.compute.internal\" (UID: \"5d3a91f23ccfbb9c1af1cd6d04b30fd6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-238.us-east-2.compute.internal" Nov 01 15:47:06.670625 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.670552 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5d3a91f23ccfbb9c1af1cd6d04b30fd6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-238.us-east-2.compute.internal\" (UID: \"5d3a91f23ccfbb9c1af1cd6d04b30fd6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-238.us-east-2.compute.internal" Nov 01 15:47:06.670625 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.670594 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ae861e52d07eb0e1937b2b22576e5cfd-config\") pod \"kube-apiserver-proxy-ip-10-0-128-238.us-east-2.compute.internal\" (UID: \"ae861e52d07eb0e1937b2b22576e5cfd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-238.us-east-2.compute.internal" Nov 01 15:47:06.670778 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.670641 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ae861e52d07eb0e1937b2b22576e5cfd-config\") pod \"kube-apiserver-proxy-ip-10-0-128-238.us-east-2.compute.internal\" (UID: \"ae861e52d07eb0e1937b2b22576e5cfd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-238.us-east-2.compute.internal" Nov 01 15:47:06.762554 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.762531 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-238.us-east-2.compute.internal" Nov 01 15:47:06.762554 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:06.762545 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-238.us-east-2.compute.internal" Nov 01 15:47:07.105392 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:07.105316 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 01 15:47:07.105836 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:07.105442 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 01 15:47:07.105836 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:07.105490 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 01 15:47:07.105836 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:07.105490 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 01 15:47:07.144180 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:07.144154 2576 apiserver.go:52] "Watching apiserver" Nov 01 15:47:07.153642 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:07.153614 2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 15:47:07.153976 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:07.153956 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-44klx","openshift-network-diagnostics/network-check-target-l26rw","kube-system/kube-apiserver-proxy-ip-10-0-128-238.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-92m6d","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-238.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-f6klc","openshift-multus/network-metrics-daemon-cx4d8","kube-system/global-pull-secret-syncer-ckz9h","openshift-multus/multus-x2wlk","openshift-network-operator/iptables-alerter-86kzp","openshift-ovn-kubernetes/ovnkube-node-jtwhw","kube-system/konnectivity-agent-d7slg","openshift-dns/node-resolver-hkkmh"] Nov 01 15:47:09.007497 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.007477 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-44klx" Nov 01 15:47:09.009529 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.009510 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l26rw" Nov 01 15:47:09.009626 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:09.009568 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l26rw" podUID="34ffae54-fbc4-44f5-b785-a230835742c2" Nov 01 15:47:09.011513 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.011497 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:09.012491 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.012472 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 15:47:09.012578 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.012527 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 15:47:09.014172 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.012752 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 15:47:09.014172 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.012950 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9hxd8\"" Nov 01 15:47:09.014879 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.014851 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" Nov 01 15:47:09.015917 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.015894 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 15:47:09.016420 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.016375 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 15:42:08 +0000 UTC" deadline="2028-06-24 23:50:43.533678942 +0000 UTC" Nov 01 15:47:09.016420 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.016416 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14432h3m34.5172655s" Nov 01 15:47:09.016655 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.016636 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-f6klc" Nov 01 15:47:09.018608 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.018593 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 01 15:47:09.018817 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.018783 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx4d8" Nov 01 15:47:09.018915 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:09.018876 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx4d8" podUID="12f347b0-ed8e-409d-bd18-261844bfeeed" Nov 01 15:47:09.021000 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.020983 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckz9h" Nov 01 15:47:09.021103 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:09.021054 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckz9h" podUID="c7838987-12dc-47bc-9b39-a87ffd11ddaf" Nov 01 15:47:09.023174 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.023157 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.023369 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.023354 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9r89\" (UniqueName: \"kubernetes.io/projected/60e635cd-c1f3-4d34-b795-2f6b16438f8f-kube-api-access-h9r89\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:09.023415 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.023378 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8e87fe52-8f65-43d9-a04a-a86d6227230d-socket-dir\") pod \"aws-ebs-csi-driver-node-cgvz7\" (UID: \"8e87fe52-8f65-43d9-a04a-a86d6227230d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" Nov 01 15:47:09.023415 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.023402 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8e87fe52-8f65-43d9-a04a-a86d6227230d-sys-fs\") pod \"aws-ebs-csi-driver-node-cgvz7\" (UID: \"8e87fe52-8f65-43d9-a04a-a86d6227230d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" Nov 01 15:47:09.023486 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.023459 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4b8758dd-149c-4c37-9d32-a93fc953e548-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f6klc\" (UID: \"4b8758dd-149c-4c37-9d32-a93fc953e548\") " pod="openshift-multus/multus-additional-cni-plugins-f6klc" Nov 01 15:47:09.023520 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.023488 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-lib-modules\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:09.023520 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.023513 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60e635cd-c1f3-4d34-b795-2f6b16438f8f-tmp\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:09.023588 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.023543 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-etc-systemd\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:09.023588 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.023567 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b8758dd-149c-4c37-9d32-a93fc953e548-cnibin\") pod \"multus-additional-cni-plugins-f6klc\" (UID: \"4b8758dd-149c-4c37-9d32-a93fc953e548\") " pod="openshift-multus/multus-additional-cni-plugins-f6klc" Nov 01 15:47:09.023655 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.023595 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b8758dd-149c-4c37-9d32-a93fc953e548-cni-binary-copy\") pod \"multus-additional-cni-plugins-f6klc\" (UID: \"4b8758dd-149c-4c37-9d32-a93fc953e548\") " pod="openshift-multus/multus-additional-cni-plugins-f6klc" Nov 01 15:47:09.023655 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.023613 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4b8758dd-149c-4c37-9d32-a93fc953e548-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f6klc\" (UID: \"4b8758dd-149c-4c37-9d32-a93fc953e548\") " pod="openshift-multus/multus-additional-cni-plugins-f6klc" Nov 01 15:47:09.023655 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.023629 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7838987-12dc-47bc-9b39-a87ffd11ddaf-original-pull-secret\") pod \"global-pull-secret-syncer-ckz9h\" (UID: \"c7838987-12dc-47bc-9b39-a87ffd11ddaf\") " pod="kube-system/global-pull-secret-syncer-ckz9h" Nov 01 15:47:09.023796 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.023704 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njf2l\" (UniqueName: \"kubernetes.io/projected/12f347b0-ed8e-409d-bd18-261844bfeeed-kube-api-access-njf2l\") pod \"network-metrics-daemon-cx4d8\" (UID: \"12f347b0-ed8e-409d-bd18-261844bfeeed\") " pod="openshift-multus/network-metrics-daemon-cx4d8" Nov 01 15:47:09.023796 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.023745 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8e87fe52-8f65-43d9-a04a-a86d6227230d-registration-dir\") pod \"aws-ebs-csi-driver-node-cgvz7\" (UID: \"8e87fe52-8f65-43d9-a04a-a86d6227230d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" Nov 01 15:47:09.023796 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.023771 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12f347b0-ed8e-409d-bd18-261844bfeeed-metrics-certs\") pod \"network-metrics-daemon-cx4d8\" (UID: \"12f347b0-ed8e-409d-bd18-261844bfeeed\") " pod="openshift-multus/network-metrics-daemon-cx4d8" Nov 01 15:47:09.023942 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.023795 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-etc-modprobe-d\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:09.023942 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.023817 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-run\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:09.023942 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.023842 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkkf9\" (UniqueName: \"kubernetes.io/projected/8e87fe52-8f65-43d9-a04a-a86d6227230d-kube-api-access-wkkf9\") pod \"aws-ebs-csi-driver-node-cgvz7\" (UID: \"8e87fe52-8f65-43d9-a04a-a86d6227230d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" Nov 01 15:47:09.023942 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.023865 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b8758dd-149c-4c37-9d32-a93fc953e548-system-cni-dir\") pod \"multus-additional-cni-plugins-f6klc\" (UID: \"4b8758dd-149c-4c37-9d32-a93fc953e548\") " pod="openshift-multus/multus-additional-cni-plugins-f6klc" Nov 01 15:47:09.023942 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.023888 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/56e8fbab-06d3-4421-9995-822247d94d90-serviceca\") pod \"node-ca-44klx\" (UID: \"56e8fbab-06d3-4421-9995-822247d94d90\") " pod="openshift-image-registry/node-ca-44klx" Nov 01 15:47:09.023942 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.023911 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kmfx\" (UniqueName: \"kubernetes.io/projected/56e8fbab-06d3-4421-9995-822247d94d90-kube-api-access-4kmfx\") pod \"node-ca-44klx\" (UID: \"56e8fbab-06d3-4421-9995-822247d94d90\") " pod="openshift-image-registry/node-ca-44klx" Nov 01 15:47:09.024200 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.023952 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-etc-sysctl-conf\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:09.024200 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.023990 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-host\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:09.024200 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.024027 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b8758dd-149c-4c37-9d32-a93fc953e548-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f6klc\" (UID: \"4b8758dd-149c-4c37-9d32-a93fc953e548\") " pod="openshift-multus/multus-additional-cni-plugins-f6klc" Nov 01 15:47:09.024200 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.024057 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/56e8fbab-06d3-4421-9995-822247d94d90-host\") pod \"node-ca-44klx\" (UID: \"56e8fbab-06d3-4421-9995-822247d94d90\") " pod="openshift-image-registry/node-ca-44klx" Nov 01 15:47:09.024200 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.024085 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-etc-sysconfig\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:09.024200 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.024099 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-etc-sysctl-d\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:09.024200 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.024165 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8e87fe52-8f65-43d9-a04a-a86d6227230d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cgvz7\" (UID: \"8e87fe52-8f65-43d9-a04a-a86d6227230d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" Nov 01 15:47:09.024430 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.024200 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8e87fe52-8f65-43d9-a04a-a86d6227230d-device-dir\") pod \"aws-ebs-csi-driver-node-cgvz7\" (UID: \"8e87fe52-8f65-43d9-a04a-a86d6227230d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" Nov 01 15:47:09.024430 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.024223 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8e87fe52-8f65-43d9-a04a-a86d6227230d-etc-selinux\") pod \"aws-ebs-csi-driver-node-cgvz7\" (UID: \"8e87fe52-8f65-43d9-a04a-a86d6227230d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" Nov 01 15:47:09.024430 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.024255 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c7838987-12dc-47bc-9b39-a87ffd11ddaf-kubelet-config\") pod \"global-pull-secret-syncer-ckz9h\" (UID: \"c7838987-12dc-47bc-9b39-a87ffd11ddaf\") " pod="kube-system/global-pull-secret-syncer-ckz9h" Nov 01 15:47:09.024430 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.024272 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-sys\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:09.024430 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.024313 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-var-lib-kubelet\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:09.024430 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.024341 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl7rt\" (UniqueName: \"kubernetes.io/projected/4b8758dd-149c-4c37-9d32-a93fc953e548-kube-api-access-xl7rt\") pod \"multus-additional-cni-plugins-f6klc\" (UID: \"4b8758dd-149c-4c37-9d32-a93fc953e548\") " pod="openshift-multus/multus-additional-cni-plugins-f6klc" Nov 01 15:47:09.024430 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.024357 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c7838987-12dc-47bc-9b39-a87ffd11ddaf-dbus\") pod \"global-pull-secret-syncer-ckz9h\" (UID: \"c7838987-12dc-47bc-9b39-a87ffd11ddaf\") " pod="kube-system/global-pull-secret-syncer-ckz9h" Nov 01 15:47:09.024430 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.024376 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlwtm\" (UniqueName: \"kubernetes.io/projected/34ffae54-fbc4-44f5-b785-a230835742c2-kube-api-access-nlwtm\") pod \"network-check-target-l26rw\" (UID: \"34ffae54-fbc4-44f5-b785-a230835742c2\") " pod="openshift-network-diagnostics/network-check-target-l26rw" Nov 01 15:47:09.024430 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.024417 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-etc-kubernetes\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:09.024430 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.024434 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/60e635cd-c1f3-4d34-b795-2f6b16438f8f-etc-tuned\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:09.025302 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.025290 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-86kzp" Nov 01 15:47:09.027616 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.027603 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.029863 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.029845 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d7slg" Nov 01 15:47:09.032365 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.032346 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hkkmh" Nov 01 15:47:09.051130 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.051113 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 15:47:09.051200 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.051160 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-b7lkr" Nov 01 15:47:09.058732 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.058714 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-b7lkr" Nov 01 15:47:09.118602 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.118578 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 15:47:09.118734 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.118715 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 15:47:09.118734 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.118719 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 15:47:09.118734 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.118739 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 15:47:09.118950 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.118806 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-28hsn\"" Nov 01 15:47:09.118950 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.118939 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6x5r5\"" Nov 01 15:47:09.120634 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.120536 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jsz89\"" Nov 01 15:47:09.120634 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.120545 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 15:47:09.120634 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.120558 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 15:47:09.120634 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.120571 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 15:47:09.120634 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.120580 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 15:47:09.120634 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.120593 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 15:47:09.120634 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.120608 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 15:47:09.120902 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.120644 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5vhjr\"" Nov 01 15:47:09.121951 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.121898 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 15:47:09.121951 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.121944 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-58l9j\"" Nov 01 15:47:09.121951 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.121951 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 15:47:09.122231 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.122043 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 15:47:09.122231 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.122149 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 15:47:09.122231 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.122162 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 15:47:09.122231 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.122168 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bwtj5\"" Nov 01 15:47:09.122231 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.122198 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 15:47:09.122231 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.122220 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 15:47:09.122444 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.122237 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 15:47:09.122444 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.122260 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 15:47:09.122444 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.122290 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pjc9m\"" Nov 01 15:47:09.122444 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.122328 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 15:47:09.122444 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.122364 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 15:47:09.122444 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.122436 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-ttxgn\"" Nov 01 15:47:09.122638 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.122550 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 15:47:09.123644 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.123621 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 15:47:09.123727 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.123705 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 15:47:09.124816 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.124794 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-etc-sysctl-d\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:09.124903 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.124816 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8e87fe52-8f65-43d9-a04a-a86d6227230d-device-dir\") pod \"aws-ebs-csi-driver-node-cgvz7\" (UID: \"8e87fe52-8f65-43d9-a04a-a86d6227230d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" Nov 01 15:47:09.124903 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.124833 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c7838987-12dc-47bc-9b39-a87ffd11ddaf-kubelet-config\") pod \"global-pull-secret-syncer-ckz9h\" (UID: \"c7838987-12dc-47bc-9b39-a87ffd11ddaf\") " pod="kube-system/global-pull-secret-syncer-ckz9h" Nov 01 15:47:09.124903 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.124850 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-host-kubelet\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.124903 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.124871 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-var-lib-kubelet\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:09.125125 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.124920 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl7rt\" (UniqueName: \"kubernetes.io/projected/4b8758dd-149c-4c37-9d32-a93fc953e548-kube-api-access-xl7rt\") pod \"multus-additional-cni-plugins-f6klc\" (UID: \"4b8758dd-149c-4c37-9d32-a93fc953e548\") " pod="openshift-multus/multus-additional-cni-plugins-f6klc" Nov 01 15:47:09.125125 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.124926 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-var-lib-kubelet\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:09.125125 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.124926 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c7838987-12dc-47bc-9b39-a87ffd11ddaf-kubelet-config\") pod \"global-pull-secret-syncer-ckz9h\" (UID: \"c7838987-12dc-47bc-9b39-a87ffd11ddaf\") " pod="kube-system/global-pull-secret-syncer-ckz9h" Nov 01 15:47:09.125125 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.124948 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c7838987-12dc-47bc-9b39-a87ffd11ddaf-dbus\") pod \"global-pull-secret-syncer-ckz9h\" (UID: \"c7838987-12dc-47bc-9b39-a87ffd11ddaf\") " pod="kube-system/global-pull-secret-syncer-ckz9h" Nov 01 15:47:09.125125 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.124969 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/09f28a75-b66e-4933-84e2-eb0941e5f6a3-cni-binary-copy\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.125125 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.124985 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-host-run-multus-certs\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.125125 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.125009 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f13a984b-8706-43d5-859f-abbfcaf58abf-konnectivity-ca\") pod \"konnectivity-agent-d7slg\" (UID: \"f13a984b-8706-43d5-859f-abbfcaf58abf\") " pod="kube-system/konnectivity-agent-d7slg" Nov 01 15:47:09.125125 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.124983 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-etc-sysctl-d\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:09.125125 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.125041 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8e87fe52-8f65-43d9-a04a-a86d6227230d-device-dir\") pod \"aws-ebs-csi-driver-node-cgvz7\" (UID: \"8e87fe52-8f65-43d9-a04a-a86d6227230d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" Nov 01 15:47:09.125125 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.125084 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-etc-kubernetes\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:09.125508 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.125141 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8e87fe52-8f65-43d9-a04a-a86d6227230d-sys-fs\") pod \"aws-ebs-csi-driver-node-cgvz7\" (UID: \"8e87fe52-8f65-43d9-a04a-a86d6227230d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" Nov 01 15:47:09.125508 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.125142 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c7838987-12dc-47bc-9b39-a87ffd11ddaf-dbus\") pod \"global-pull-secret-syncer-ckz9h\" (UID: \"c7838987-12dc-47bc-9b39-a87ffd11ddaf\") " pod="kube-system/global-pull-secret-syncer-ckz9h" Nov 01 15:47:09.125508 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.125158 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-etc-kubernetes\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:09.125508 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.125213 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-system-cni-dir\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.125508 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.125268 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8e87fe52-8f65-43d9-a04a-a86d6227230d-sys-fs\") pod \"aws-ebs-csi-driver-node-cgvz7\" (UID: \"8e87fe52-8f65-43d9-a04a-a86d6227230d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" Nov 01 15:47:09.125508 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.125305 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-host-var-lib-cni-bin\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.125508 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.125338 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.125508 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.125368 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60e635cd-c1f3-4d34-b795-2f6b16438f8f-tmp\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:09.125508 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.125447 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-etc-kubernetes\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.125508 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.125486 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-run-openvswitch\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.125878 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.125533 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/667a7db5-cf73-424c-9d18-53bfbc5f65db-tmp-dir\") pod \"node-resolver-hkkmh\" (UID: \"667a7db5-cf73-424c-9d18-53bfbc5f65db\") " pod="openshift-dns/node-resolver-hkkmh" Nov 01 15:47:09.125878 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.125577 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-etc-systemd\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:09.125878 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.125606 2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 15:47:09.125878 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.125614 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b8758dd-149c-4c37-9d32-a93fc953e548-cnibin\") pod \"multus-additional-cni-plugins-f6klc\" (UID: \"4b8758dd-149c-4c37-9d32-a93fc953e548\") " pod="openshift-multus/multus-additional-cni-plugins-f6klc" Nov 01 15:47:09.125878 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.125652 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-etc-systemd\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:09.125878 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.125669 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b8758dd-149c-4c37-9d32-a93fc953e548-cni-binary-copy\") pod \"multus-additional-cni-plugins-f6klc\" (UID: \"4b8758dd-149c-4c37-9d32-a93fc953e548\") " pod="openshift-multus/multus-additional-cni-plugins-f6klc" Nov 01 15:47:09.125878 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.125695 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b8758dd-149c-4c37-9d32-a93fc953e548-cnibin\") pod \"multus-additional-cni-plugins-f6klc\" (UID: \"4b8758dd-149c-4c37-9d32-a93fc953e548\") " pod="openshift-multus/multus-additional-cni-plugins-f6klc" Nov 01 15:47:09.125878 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.125735 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-multus-conf-dir\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.125878 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.125830 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-host-slash\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.125878 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.125853 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-log-socket\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.125878 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.125880 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-host-var-lib-cni-multus\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.126403 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.125919 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b53051b0-243d-4e03-926a-1f83372e648f-iptables-alerter-script\") pod \"iptables-alerter-86kzp\" (UID: \"b53051b0-243d-4e03-926a-1f83372e648f\") " pod="openshift-network-operator/iptables-alerter-86kzp" Nov 01 15:47:09.126403 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.125950 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f13a984b-8706-43d5-859f-abbfcaf58abf-agent-certs\") pod \"konnectivity-agent-d7slg\" (UID: \"f13a984b-8706-43d5-859f-abbfcaf58abf\") " pod="kube-system/konnectivity-agent-d7slg" Nov 01 15:47:09.126403 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.125983 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12f347b0-ed8e-409d-bd18-261844bfeeed-metrics-certs\") pod \"network-metrics-daemon-cx4d8\" (UID: \"12f347b0-ed8e-409d-bd18-261844bfeeed\") " pod="openshift-multus/network-metrics-daemon-cx4d8" Nov 01 15:47:09.126403 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.126053 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-etc-modprobe-d\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:09.126403 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:09.126101 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 15:47:09.126403 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.126134 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kmfx\" (UniqueName: \"kubernetes.io/projected/56e8fbab-06d3-4421-9995-822247d94d90-kube-api-access-4kmfx\") pod \"node-ca-44klx\" (UID: \"56e8fbab-06d3-4421-9995-822247d94d90\") " pod="openshift-image-registry/node-ca-44klx" Nov 01 15:47:09.126403 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.126196 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-etc-modprobe-d\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:09.126403 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:09.126206 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12f347b0-ed8e-409d-bd18-261844bfeeed-metrics-certs podName:12f347b0-ed8e-409d-bd18-261844bfeeed nodeName:}" failed. No retries permitted until 2026-11-01 15:47:09.626161711 +0000 UTC m=+1.948327613 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12f347b0-ed8e-409d-bd18-261844bfeeed-metrics-certs") pod "network-metrics-daemon-cx4d8" (UID: "12f347b0-ed8e-409d-bd18-261844bfeeed") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 15:47:09.126403 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.126231 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-run-ovn\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.126403 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.126266 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b8758dd-149c-4c37-9d32-a93fc953e548-cni-binary-copy\") pod \"multus-additional-cni-plugins-f6klc\" (UID: \"4b8758dd-149c-4c37-9d32-a93fc953e548\") " pod="openshift-multus/multus-additional-cni-plugins-f6klc" Nov 01 15:47:09.126403 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.126308 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-etc-sysctl-conf\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:09.126403 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.126355 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-host\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:09.126403 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.126387 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b8758dd-149c-4c37-9d32-a93fc953e548-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f6klc\" (UID: \"4b8758dd-149c-4c37-9d32-a93fc953e548\") " pod="openshift-multus/multus-additional-cni-plugins-f6klc" Nov 01 15:47:09.126403 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.126406 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-host\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:09.127182 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.126411 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-cnibin\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.127182 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.126465 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-etc-sysctl-conf\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:09.127182 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.126480 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-multus-socket-dir-parent\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.127182 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.126512 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8254aafa-dd5b-4150-87e4-d165dab6bcb5-ovnkube-script-lib\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.127182 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.126529 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glz5x\" (UniqueName: \"kubernetes.io/projected/8254aafa-dd5b-4150-87e4-d165dab6bcb5-kube-api-access-glz5x\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.127182 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.126547 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-etc-sysconfig\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:09.127182 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.126563 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8e87fe52-8f65-43d9-a04a-a86d6227230d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cgvz7\" (UID: \"8e87fe52-8f65-43d9-a04a-a86d6227230d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" Nov 01 15:47:09.127182 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.126562 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b8758dd-149c-4c37-9d32-a93fc953e548-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f6klc\" (UID: \"4b8758dd-149c-4c37-9d32-a93fc953e548\") " pod="openshift-multus/multus-additional-cni-plugins-f6klc" Nov 01 15:47:09.127182 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.126577 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8e87fe52-8f65-43d9-a04a-a86d6227230d-etc-selinux\") pod \"aws-ebs-csi-driver-node-cgvz7\" (UID: \"8e87fe52-8f65-43d9-a04a-a86d6227230d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" Nov 01 15:47:09.127182 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.126593 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-run-systemd\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.127182 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.126622 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-node-log\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.127182 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.126635 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-etc-sysconfig\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:09.127182 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.126641 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8e87fe52-8f65-43d9-a04a-a86d6227230d-etc-selinux\") pod \"aws-ebs-csi-driver-node-cgvz7\" (UID: \"8e87fe52-8f65-43d9-a04a-a86d6227230d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" Nov 01 15:47:09.127182 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.126683 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8e87fe52-8f65-43d9-a04a-a86d6227230d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cgvz7\" (UID: \"8e87fe52-8f65-43d9-a04a-a86d6227230d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" Nov 01 15:47:09.127182 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.126694 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-host-run-ovn-kubernetes\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.127182 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.126737 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-host-cni-netd\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.127182 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.126779 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8254aafa-dd5b-4150-87e4-d165dab6bcb5-ovnkube-config\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.127793 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.126803 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-sys\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:09.127793 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.126819 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4s2s\" (UniqueName: \"kubernetes.io/projected/667a7db5-cf73-424c-9d18-53bfbc5f65db-kube-api-access-t4s2s\") pod \"node-resolver-hkkmh\" (UID: \"667a7db5-cf73-424c-9d18-53bfbc5f65db\") " pod="openshift-dns/node-resolver-hkkmh" Nov 01 15:47:09.127793 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.126834 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlwtm\" (UniqueName: \"kubernetes.io/projected/34ffae54-fbc4-44f5-b785-a230835742c2-kube-api-access-nlwtm\") pod \"network-check-target-l26rw\" (UID: \"34ffae54-fbc4-44f5-b785-a230835742c2\") " pod="openshift-network-diagnostics/network-check-target-l26rw" Nov 01 15:47:09.127793 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.126860 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/60e635cd-c1f3-4d34-b795-2f6b16438f8f-etc-tuned\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:09.127793 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.126883 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9r89\" (UniqueName: \"kubernetes.io/projected/60e635cd-c1f3-4d34-b795-2f6b16438f8f-kube-api-access-h9r89\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:09.127793 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.126891 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-sys\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:09.127793 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.126908 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8e87fe52-8f65-43d9-a04a-a86d6227230d-socket-dir\") pod \"aws-ebs-csi-driver-node-cgvz7\" (UID: \"8e87fe52-8f65-43d9-a04a-a86d6227230d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" Nov 01 15:47:09.127793 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.126948 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4b8758dd-149c-4c37-9d32-a93fc953e548-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f6klc\" (UID: \"4b8758dd-149c-4c37-9d32-a93fc953e548\") " pod="openshift-multus/multus-additional-cni-plugins-f6klc" Nov 01 15:47:09.127793 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.126977 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-host-run-k8s-cni-cncf-io\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.127793 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.127006 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8254aafa-dd5b-4150-87e4-d165dab6bcb5-env-overrides\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.127793 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.127046 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-lib-modules\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:09.127793 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.127070 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-host-run-netns\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.127793 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.127091 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8e87fe52-8f65-43d9-a04a-a86d6227230d-socket-dir\") pod \"aws-ebs-csi-driver-node-cgvz7\" (UID: \"8e87fe52-8f65-43d9-a04a-a86d6227230d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" Nov 01 15:47:09.127793 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.127177 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-lib-modules\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:09.127793 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.127234 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdpt9\" (UniqueName: \"kubernetes.io/projected/09f28a75-b66e-4933-84e2-eb0941e5f6a3-kube-api-access-gdpt9\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.127793 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.127287 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjb4z\" (UniqueName: \"kubernetes.io/projected/b53051b0-243d-4e03-926a-1f83372e648f-kube-api-access-pjb4z\") pod \"iptables-alerter-86kzp\" (UID: \"b53051b0-243d-4e03-926a-1f83372e648f\") " pod="openshift-network-operator/iptables-alerter-86kzp" Nov 01 15:47:09.127793 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.127345 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-systemd-units\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.128278 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.127374 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4b8758dd-149c-4c37-9d32-a93fc953e548-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f6klc\" (UID: \"4b8758dd-149c-4c37-9d32-a93fc953e548\") " pod="openshift-multus/multus-additional-cni-plugins-f6klc" Nov 01 15:47:09.128278 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.127431 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7838987-12dc-47bc-9b39-a87ffd11ddaf-original-pull-secret\") pod \"global-pull-secret-syncer-ckz9h\" (UID: \"c7838987-12dc-47bc-9b39-a87ffd11ddaf\") " pod="kube-system/global-pull-secret-syncer-ckz9h" Nov 01 15:47:09.128278 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:09.127511 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 15:47:09.128278 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.127507 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4b8758dd-149c-4c37-9d32-a93fc953e548-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f6klc\" (UID: \"4b8758dd-149c-4c37-9d32-a93fc953e548\") " pod="openshift-multus/multus-additional-cni-plugins-f6klc" Nov 01 15:47:09.128278 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:09.127567 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7838987-12dc-47bc-9b39-a87ffd11ddaf-original-pull-secret podName:c7838987-12dc-47bc-9b39-a87ffd11ddaf nodeName:}" failed. No retries permitted until 2026-11-01 15:47:09.627551374 +0000 UTC m=+1.949717277 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c7838987-12dc-47bc-9b39-a87ffd11ddaf-original-pull-secret") pod "global-pull-secret-syncer-ckz9h" (UID: "c7838987-12dc-47bc-9b39-a87ffd11ddaf") : object "kube-system"/"original-pull-secret" not registered Nov 01 15:47:09.128278 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.127612 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-host-var-lib-kubelet\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.128278 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.127637 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/09f28a75-b66e-4933-84e2-eb0941e5f6a3-multus-daemon-config\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.128278 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.127660 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b53051b0-243d-4e03-926a-1f83372e648f-host-slash\") pod \"iptables-alerter-86kzp\" (UID: \"b53051b0-243d-4e03-926a-1f83372e648f\") " pod="openshift-network-operator/iptables-alerter-86kzp" Nov 01 15:47:09.128278 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.127709 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njf2l\" (UniqueName: \"kubernetes.io/projected/12f347b0-ed8e-409d-bd18-261844bfeeed-kube-api-access-njf2l\") pod \"network-metrics-daemon-cx4d8\" (UID: \"12f347b0-ed8e-409d-bd18-261844bfeeed\") " pod="openshift-multus/network-metrics-daemon-cx4d8" Nov 01 15:47:09.128278 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.127752 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8e87fe52-8f65-43d9-a04a-a86d6227230d-registration-dir\") pod \"aws-ebs-csi-driver-node-cgvz7\" (UID: \"8e87fe52-8f65-43d9-a04a-a86d6227230d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" Nov 01 15:47:09.128278 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.127780 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-multus-cni-dir\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.128278 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.127805 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/667a7db5-cf73-424c-9d18-53bfbc5f65db-hosts-file\") pod \"node-resolver-hkkmh\" (UID: \"667a7db5-cf73-424c-9d18-53bfbc5f65db\") " pod="openshift-dns/node-resolver-hkkmh" Nov 01 15:47:09.128278 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.127853 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-run\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:09.128278 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.127880 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkkf9\" (UniqueName: \"kubernetes.io/projected/8e87fe52-8f65-43d9-a04a-a86d6227230d-kube-api-access-wkkf9\") pod \"aws-ebs-csi-driver-node-cgvz7\" (UID: \"8e87fe52-8f65-43d9-a04a-a86d6227230d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" Nov 01 15:47:09.128278 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.127904 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b8758dd-149c-4c37-9d32-a93fc953e548-system-cni-dir\") pod \"multus-additional-cni-plugins-f6klc\" (UID: \"4b8758dd-149c-4c37-9d32-a93fc953e548\") " pod="openshift-multus/multus-additional-cni-plugins-f6klc" Nov 01 15:47:09.128278 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.127915 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4b8758dd-149c-4c37-9d32-a93fc953e548-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f6klc\" (UID: \"4b8758dd-149c-4c37-9d32-a93fc953e548\") " pod="openshift-multus/multus-additional-cni-plugins-f6klc" Nov 01 15:47:09.128795 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.127926 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/56e8fbab-06d3-4421-9995-822247d94d90-serviceca\") pod \"node-ca-44klx\" (UID: \"56e8fbab-06d3-4421-9995-822247d94d90\") " pod="openshift-image-registry/node-ca-44klx" Nov 01 15:47:09.128795 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.127958 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-host-run-netns\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.128795 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.128005 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b8758dd-149c-4c37-9d32-a93fc953e548-system-cni-dir\") pod \"multus-additional-cni-plugins-f6klc\" (UID: \"4b8758dd-149c-4c37-9d32-a93fc953e548\") " pod="openshift-multus/multus-additional-cni-plugins-f6klc" Nov 01 15:47:09.128795 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.128038 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8e87fe52-8f65-43d9-a04a-a86d6227230d-registration-dir\") pod \"aws-ebs-csi-driver-node-cgvz7\" (UID: \"8e87fe52-8f65-43d9-a04a-a86d6227230d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" Nov 01 15:47:09.128795 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.128082 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-run\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:09.128795 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.128115 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8254aafa-dd5b-4150-87e4-d165dab6bcb5-ovn-node-metrics-cert\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.128795 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.128140 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/56e8fbab-06d3-4421-9995-822247d94d90-host\") pod \"node-ca-44klx\" (UID: \"56e8fbab-06d3-4421-9995-822247d94d90\") " pod="openshift-image-registry/node-ca-44klx" Nov 01 15:47:09.128795 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.128165 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-hostroot\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.128795 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.128188 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-var-lib-openvswitch\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.128795 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.128277 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-etc-openvswitch\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.128795 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.128314 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-host-cni-bin\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.128795 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.128410 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/56e8fbab-06d3-4421-9995-822247d94d90-host\") pod \"node-ca-44klx\" (UID: \"56e8fbab-06d3-4421-9995-822247d94d90\") " pod="openshift-image-registry/node-ca-44klx" Nov 01 15:47:09.128795 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.128743 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/56e8fbab-06d3-4421-9995-822247d94d90-serviceca\") pod \"node-ca-44klx\" (UID: \"56e8fbab-06d3-4421-9995-822247d94d90\") " pod="openshift-image-registry/node-ca-44klx" Nov 01 15:47:09.129402 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.129382 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/60e635cd-c1f3-4d34-b795-2f6b16438f8f-etc-tuned\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:09.129481 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.129459 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60e635cd-c1f3-4d34-b795-2f6b16438f8f-tmp\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:09.133490 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.133466 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl7rt\" (UniqueName: \"kubernetes.io/projected/4b8758dd-149c-4c37-9d32-a93fc953e548-kube-api-access-xl7rt\") pod \"multus-additional-cni-plugins-f6klc\" (UID: \"4b8758dd-149c-4c37-9d32-a93fc953e548\") " pod="openshift-multus/multus-additional-cni-plugins-f6klc" Nov 01 15:47:09.135657 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:09.135640 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 15:47:09.135657 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:09.135657 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 15:47:09.135769 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:09.135666 2576 projected.go:196] Error preparing data for projected volume kube-api-access-nlwtm for pod openshift-network-diagnostics/network-check-target-l26rw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 15:47:09.135769 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:09.135714 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34ffae54-fbc4-44f5-b785-a230835742c2-kube-api-access-nlwtm podName:34ffae54-fbc4-44f5-b785-a230835742c2 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:09.635701664 +0000 UTC m=+1.957867568 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nlwtm" (UniqueName: "kubernetes.io/projected/34ffae54-fbc4-44f5-b785-a230835742c2-kube-api-access-nlwtm") pod "network-check-target-l26rw" (UID: "34ffae54-fbc4-44f5-b785-a230835742c2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 15:47:09.136570 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.136547 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kmfx\" (UniqueName: \"kubernetes.io/projected/56e8fbab-06d3-4421-9995-822247d94d90-kube-api-access-4kmfx\") pod \"node-ca-44klx\" (UID: \"56e8fbab-06d3-4421-9995-822247d94d90\") " pod="openshift-image-registry/node-ca-44klx" Nov 01 15:47:09.137448 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.137427 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkkf9\" (UniqueName: \"kubernetes.io/projected/8e87fe52-8f65-43d9-a04a-a86d6227230d-kube-api-access-wkkf9\") pod \"aws-ebs-csi-driver-node-cgvz7\" (UID: \"8e87fe52-8f65-43d9-a04a-a86d6227230d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" Nov 01 15:47:09.137664 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.137646 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9r89\" (UniqueName: \"kubernetes.io/projected/60e635cd-c1f3-4d34-b795-2f6b16438f8f-kube-api-access-h9r89\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:09.138046 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.138009 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njf2l\" (UniqueName: \"kubernetes.io/projected/12f347b0-ed8e-409d-bd18-261844bfeeed-kube-api-access-njf2l\") pod \"network-metrics-daemon-cx4d8\" (UID: \"12f347b0-ed8e-409d-bd18-261844bfeeed\") " pod="openshift-multus/network-metrics-daemon-cx4d8" Nov 01 15:47:09.228728 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.228662 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-host-var-lib-kubelet\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.228728 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.228692 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/09f28a75-b66e-4933-84e2-eb0941e5f6a3-multus-daemon-config\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.228728 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.228708 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b53051b0-243d-4e03-926a-1f83372e648f-host-slash\") pod \"iptables-alerter-86kzp\" (UID: \"b53051b0-243d-4e03-926a-1f83372e648f\") " pod="openshift-network-operator/iptables-alerter-86kzp" Nov 01 15:47:09.228728 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.228730 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-multus-cni-dir\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.229079 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.228752 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/667a7db5-cf73-424c-9d18-53bfbc5f65db-hosts-file\") pod \"node-resolver-hkkmh\" (UID: \"667a7db5-cf73-424c-9d18-53bfbc5f65db\") " pod="openshift-dns/node-resolver-hkkmh" Nov 01 15:47:09.229079 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.228779 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-host-run-netns\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.229079 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.228803 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8254aafa-dd5b-4150-87e4-d165dab6bcb5-ovn-node-metrics-cert\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.229079 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.228814 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b53051b0-243d-4e03-926a-1f83372e648f-host-slash\") pod \"iptables-alerter-86kzp\" (UID: \"b53051b0-243d-4e03-926a-1f83372e648f\") " pod="openshift-network-operator/iptables-alerter-86kzp" Nov 01 15:47:09.229079 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.228827 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-hostroot\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.229079 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.228840 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-host-run-netns\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.229079 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.228801 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-host-var-lib-kubelet\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.229079 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.228839 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/667a7db5-cf73-424c-9d18-53bfbc5f65db-hosts-file\") pod \"node-resolver-hkkmh\" (UID: \"667a7db5-cf73-424c-9d18-53bfbc5f65db\") " pod="openshift-dns/node-resolver-hkkmh" Nov 01 15:47:09.229079 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.228867 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-hostroot\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.229079 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.228861 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-var-lib-openvswitch\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.229079 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.228892 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-var-lib-openvswitch\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.229079 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.228895 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-multus-cni-dir\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.229079 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.228902 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-etc-openvswitch\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.229079 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.228930 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-etc-openvswitch\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.229079 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.228934 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-host-cni-bin\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.229079 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.228957 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-host-kubelet\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.229079 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.228974 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/09f28a75-b66e-4933-84e2-eb0941e5f6a3-cni-binary-copy\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.229079 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.228973 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-host-cni-bin\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.229611 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.228992 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-host-kubelet\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.229611 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.228990 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-host-run-multus-certs\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.229611 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229032 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-host-run-multus-certs\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.229611 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229040 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f13a984b-8706-43d5-859f-abbfcaf58abf-konnectivity-ca\") pod \"konnectivity-agent-d7slg\" (UID: \"f13a984b-8706-43d5-859f-abbfcaf58abf\") " pod="kube-system/konnectivity-agent-d7slg" Nov 01 15:47:09.229611 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229205 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-system-cni-dir\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.229611 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229223 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-host-var-lib-cni-bin\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.229611 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229240 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.229611 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229257 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-etc-kubernetes\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.229611 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229281 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-run-openvswitch\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.229611 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229294 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-host-var-lib-cni-bin\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.229611 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229304 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/667a7db5-cf73-424c-9d18-53bfbc5f65db-tmp-dir\") pod \"node-resolver-hkkmh\" (UID: \"667a7db5-cf73-424c-9d18-53bfbc5f65db\") " pod="openshift-dns/node-resolver-hkkmh" Nov 01 15:47:09.229611 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229309 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.229611 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229327 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-system-cni-dir\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.229611 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229335 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-multus-conf-dir\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.229611 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229356 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-etc-kubernetes\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.229611 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229360 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-host-slash\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.229611 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229369 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-run-openvswitch\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.229611 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229390 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-host-slash\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.230489 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229394 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-log-socket\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.230489 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229404 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-multus-conf-dir\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.230489 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229420 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-log-socket\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.230489 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229424 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-host-var-lib-cni-multus\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.230489 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229446 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b53051b0-243d-4e03-926a-1f83372e648f-iptables-alerter-script\") pod \"iptables-alerter-86kzp\" (UID: \"b53051b0-243d-4e03-926a-1f83372e648f\") " pod="openshift-network-operator/iptables-alerter-86kzp" Nov 01 15:47:09.230489 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229465 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f13a984b-8706-43d5-859f-abbfcaf58abf-agent-certs\") pod \"konnectivity-agent-d7slg\" (UID: \"f13a984b-8706-43d5-859f-abbfcaf58abf\") " pod="kube-system/konnectivity-agent-d7slg" Nov 01 15:47:09.230489 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229445 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-host-var-lib-cni-multus\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.230489 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229493 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-run-ovn\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.230489 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229519 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-cnibin\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.230489 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229541 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-run-ovn\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.230489 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229542 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-multus-socket-dir-parent\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.230489 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229569 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8254aafa-dd5b-4150-87e4-d165dab6bcb5-ovnkube-script-lib\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.230489 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229583 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-multus-socket-dir-parent\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.230489 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229591 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glz5x\" (UniqueName: \"kubernetes.io/projected/8254aafa-dd5b-4150-87e4-d165dab6bcb5-kube-api-access-glz5x\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.230489 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229616 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-cnibin\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.230489 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229619 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-run-systemd\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.230489 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229641 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-node-log\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.230489 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229669 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-host-run-ovn-kubernetes\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.231330 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229672 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-run-systemd\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.231330 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229692 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-host-cni-netd\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.231330 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229710 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-host-run-ovn-kubernetes\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.231330 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229715 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8254aafa-dd5b-4150-87e4-d165dab6bcb5-ovnkube-config\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.231330 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229740 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4s2s\" (UniqueName: \"kubernetes.io/projected/667a7db5-cf73-424c-9d18-53bfbc5f65db-kube-api-access-t4s2s\") pod \"node-resolver-hkkmh\" (UID: \"667a7db5-cf73-424c-9d18-53bfbc5f65db\") " pod="openshift-dns/node-resolver-hkkmh" Nov 01 15:47:09.231330 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229746 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-node-log\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.231330 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229782 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-host-cni-netd\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.231330 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229816 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-host-run-k8s-cni-cncf-io\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.231330 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229783 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-host-run-k8s-cni-cncf-io\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.231330 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229929 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8254aafa-dd5b-4150-87e4-d165dab6bcb5-env-overrides\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.231330 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229955 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-host-run-netns\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.231330 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.229978 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdpt9\" (UniqueName: \"kubernetes.io/projected/09f28a75-b66e-4933-84e2-eb0941e5f6a3-kube-api-access-gdpt9\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.231330 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.230001 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjb4z\" (UniqueName: \"kubernetes.io/projected/b53051b0-243d-4e03-926a-1f83372e648f-kube-api-access-pjb4z\") pod \"iptables-alerter-86kzp\" (UID: \"b53051b0-243d-4e03-926a-1f83372e648f\") " pod="openshift-network-operator/iptables-alerter-86kzp" Nov 01 15:47:09.231330 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.230049 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-systemd-units\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.231330 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.230112 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-systemd-units\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.231330 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.230148 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-host-run-netns\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.231330 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.230768 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/667a7db5-cf73-424c-9d18-53bfbc5f65db-tmp-dir\") pod \"node-resolver-hkkmh\" (UID: \"667a7db5-cf73-424c-9d18-53bfbc5f65db\") " pod="openshift-dns/node-resolver-hkkmh" Nov 01 15:47:09.231330 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.231050 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8254aafa-dd5b-4150-87e4-d165dab6bcb5-ovnkube-script-lib\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.231951 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.231052 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b53051b0-243d-4e03-926a-1f83372e648f-iptables-alerter-script\") pod \"iptables-alerter-86kzp\" (UID: \"b53051b0-243d-4e03-926a-1f83372e648f\") " pod="openshift-network-operator/iptables-alerter-86kzp" Nov 01 15:47:09.231951 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.231213 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f13a984b-8706-43d5-859f-abbfcaf58abf-konnectivity-ca\") pod \"konnectivity-agent-d7slg\" (UID: \"f13a984b-8706-43d5-859f-abbfcaf58abf\") " pod="kube-system/konnectivity-agent-d7slg" Nov 01 15:47:09.231951 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.231405 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8254aafa-dd5b-4150-87e4-d165dab6bcb5-ovn-node-metrics-cert\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.231951 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.231426 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8254aafa-dd5b-4150-87e4-d165dab6bcb5-env-overrides\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.231951 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.231469 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8254aafa-dd5b-4150-87e4-d165dab6bcb5-ovnkube-config\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.231951 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.231639 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/09f28a75-b66e-4933-84e2-eb0941e5f6a3-multus-daemon-config\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.231951 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.231665 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/09f28a75-b66e-4933-84e2-eb0941e5f6a3-cni-binary-copy\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.233344 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.233328 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f13a984b-8706-43d5-859f-abbfcaf58abf-agent-certs\") pod \"konnectivity-agent-d7slg\" (UID: \"f13a984b-8706-43d5-859f-abbfcaf58abf\") " pod="kube-system/konnectivity-agent-d7slg" Nov 01 15:47:09.238449 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.238418 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdpt9\" (UniqueName: \"kubernetes.io/projected/09f28a75-b66e-4933-84e2-eb0941e5f6a3-kube-api-access-gdpt9\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.238449 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.238434 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4s2s\" (UniqueName: \"kubernetes.io/projected/667a7db5-cf73-424c-9d18-53bfbc5f65db-kube-api-access-t4s2s\") pod \"node-resolver-hkkmh\" (UID: \"667a7db5-cf73-424c-9d18-53bfbc5f65db\") " pod="openshift-dns/node-resolver-hkkmh" Nov 01 15:47:09.238928 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.238904 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjb4z\" (UniqueName: \"kubernetes.io/projected/b53051b0-243d-4e03-926a-1f83372e648f-kube-api-access-pjb4z\") pod \"iptables-alerter-86kzp\" (UID: \"b53051b0-243d-4e03-926a-1f83372e648f\") " pod="openshift-network-operator/iptables-alerter-86kzp" Nov 01 15:47:09.239084 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.239070 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glz5x\" (UniqueName: \"kubernetes.io/projected/8254aafa-dd5b-4150-87e4-d165dab6bcb5-kube-api-access-glz5x\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.293339 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:09.293301 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d3a91f23ccfbb9c1af1cd6d04b30fd6.slice/crio-9d5263a747b1d06072586979cb12c5bd28b85c8eb029fe696c87a956fbdf2787 WatchSource:0}: Error finding container 9d5263a747b1d06072586979cb12c5bd28b85c8eb029fe696c87a956fbdf2787: Status 404 returned error can't find the container with id 9d5263a747b1d06072586979cb12c5bd28b85c8eb029fe696c87a956fbdf2787 Nov 01 15:47:09.304315 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.304296 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 15:47:09.311335 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:09.311315 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae861e52d07eb0e1937b2b22576e5cfd.slice/crio-eed1a6e1117b93c7d32aa445a6b08f47f555ec343f4d3007de78f66134168519 WatchSource:0}: Error finding container eed1a6e1117b93c7d32aa445a6b08f47f555ec343f4d3007de78f66134168519: Status 404 returned error can't find the container with id eed1a6e1117b93c7d32aa445a6b08f47f555ec343f4d3007de78f66134168519 Nov 01 15:47:09.338573 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.338548 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-44klx" Nov 01 15:47:09.347769 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:09.347747 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56e8fbab_06d3_4421_9995_822247d94d90.slice/crio-74534d21e666655f844f8a4dce38fb0a3ed04f94c7c214a87da6d6b336832651 WatchSource:0}: Error finding container 74534d21e666655f844f8a4dce38fb0a3ed04f94c7c214a87da6d6b336832651: Status 404 returned error can't find the container with id 74534d21e666655f844f8a4dce38fb0a3ed04f94c7c214a87da6d6b336832651 Nov 01 15:47:09.359728 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.359710 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:09.367571 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:09.367546 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e635cd_c1f3_4d34_b795_2f6b16438f8f.slice/crio-9e31f87a563ee9d4e9b6241af1725ceaed33fdd49e2b1aa14142c7ccaf6fedd4 WatchSource:0}: Error finding container 9e31f87a563ee9d4e9b6241af1725ceaed33fdd49e2b1aa14142c7ccaf6fedd4: Status 404 returned error can't find the container with id 9e31f87a563ee9d4e9b6241af1725ceaed33fdd49e2b1aa14142c7ccaf6fedd4 Nov 01 15:47:09.384887 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.384867 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" Nov 01 15:47:09.392579 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:09.392557 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e87fe52_8f65_43d9_a04a_a86d6227230d.slice/crio-803e85f1b86f85efe617b68cd13e21b22870860367c89256d954973adff1927e WatchSource:0}: Error finding container 803e85f1b86f85efe617b68cd13e21b22870860367c89256d954973adff1927e: Status 404 returned error can't find the container with id 803e85f1b86f85efe617b68cd13e21b22870860367c89256d954973adff1927e Nov 01 15:47:09.406572 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.406554 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-f6klc" Nov 01 15:47:09.408723 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.408706 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-x2wlk" Nov 01 15:47:09.416383 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.416356 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-86kzp" Nov 01 15:47:09.417160 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:09.417140 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b8758dd_149c_4c37_9d32_a93fc953e548.slice/crio-b8db1b3a385d3d6a0dbce8ac88f785a826315f7f35eb317d4177bafa842525a6 WatchSource:0}: Error finding container b8db1b3a385d3d6a0dbce8ac88f785a826315f7f35eb317d4177bafa842525a6: Status 404 returned error can't find the container with id b8db1b3a385d3d6a0dbce8ac88f785a826315f7f35eb317d4177bafa842525a6 Nov 01 15:47:09.417668 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:09.417650 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09f28a75_b66e_4933_84e2_eb0941e5f6a3.slice/crio-a732a83c796496797985edc42d736f58652177e7d9e4f6e3177a4c1a9f1d84e7 WatchSource:0}: Error finding container a732a83c796496797985edc42d736f58652177e7d9e4f6e3177a4c1a9f1d84e7: Status 404 returned error can't find the container with id a732a83c796496797985edc42d736f58652177e7d9e4f6e3177a4c1a9f1d84e7 Nov 01 15:47:09.425179 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:09.425161 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb53051b0_243d_4e03_926a_1f83372e648f.slice/crio-9b717001af95764d3d50972a296153b9f1b650251ab71ecb27e1fdab544fca0b WatchSource:0}: Error finding container 9b717001af95764d3d50972a296153b9f1b650251ab71ecb27e1fdab544fca0b: Status 404 returned error can't find the container with id 9b717001af95764d3d50972a296153b9f1b650251ab71ecb27e1fdab544fca0b Nov 01 15:47:09.445585 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.445564 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:09.454651 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:09.454632 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8254aafa_dd5b_4150_87e4_d165dab6bcb5.slice/crio-4baf3b81c3058ccf6560267398abfaea5af29aeb6a7b9295fee685f4e2bb4fc1 WatchSource:0}: Error finding container 4baf3b81c3058ccf6560267398abfaea5af29aeb6a7b9295fee685f4e2bb4fc1: Status 404 returned error can't find the container with id 4baf3b81c3058ccf6560267398abfaea5af29aeb6a7b9295fee685f4e2bb4fc1 Nov 01 15:47:09.455565 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.455548 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d7slg" Nov 01 15:47:09.458102 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.458085 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hkkmh" Nov 01 15:47:09.463927 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:09.463905 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf13a984b_8706_43d5_859f_abbfcaf58abf.slice/crio-1fd78a54e606c6f9d9e263ddb1349684db7653078d9f5b9524e2debf12cc08f3 WatchSource:0}: Error finding container 1fd78a54e606c6f9d9e263ddb1349684db7653078d9f5b9524e2debf12cc08f3: Status 404 returned error can't find the container with id 1fd78a54e606c6f9d9e263ddb1349684db7653078d9f5b9524e2debf12cc08f3 Nov 01 15:47:09.465802 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:09.465785 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod667a7db5_cf73_424c_9d18_53bfbc5f65db.slice/crio-49011229e0e6d68ef5b69103031362491e74528b2942fd19f7e57d9897009e2a WatchSource:0}: Error finding container 49011229e0e6d68ef5b69103031362491e74528b2942fd19f7e57d9897009e2a: Status 404 returned error can't find the container with id 49011229e0e6d68ef5b69103031362491e74528b2942fd19f7e57d9897009e2a Nov 01 15:47:09.467623 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:09.467606 2576 kuberuntime_manager.go:1707] "Unhandled Error" err="container dns-node-resolver start failed in pod node-resolver-hkkmh_openshift-dns(667a7db5-cf73-424c-9d18-53bfbc5f65db): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Nov 01 15:47:09.467671 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:09.467637 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"dns-node-resolver\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-dns/node-resolver-hkkmh" podUID="667a7db5-cf73-424c-9d18-53bfbc5f65db" Nov 01 15:47:09.633120 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.633050 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12f347b0-ed8e-409d-bd18-261844bfeeed-metrics-certs\") pod \"network-metrics-daemon-cx4d8\" (UID: \"12f347b0-ed8e-409d-bd18-261844bfeeed\") " pod="openshift-multus/network-metrics-daemon-cx4d8" Nov 01 15:47:09.633120 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.633096 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7838987-12dc-47bc-9b39-a87ffd11ddaf-original-pull-secret\") pod \"global-pull-secret-syncer-ckz9h\" (UID: \"c7838987-12dc-47bc-9b39-a87ffd11ddaf\") " pod="kube-system/global-pull-secret-syncer-ckz9h" Nov 01 15:47:09.633288 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:09.633179 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 15:47:09.633288 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:09.633181 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 15:47:09.633288 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:09.633224 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7838987-12dc-47bc-9b39-a87ffd11ddaf-original-pull-secret podName:c7838987-12dc-47bc-9b39-a87ffd11ddaf nodeName:}" failed. No retries permitted until 2026-11-01 15:47:10.633212771 +0000 UTC m=+2.955378663 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c7838987-12dc-47bc-9b39-a87ffd11ddaf-original-pull-secret") pod "global-pull-secret-syncer-ckz9h" (UID: "c7838987-12dc-47bc-9b39-a87ffd11ddaf") : object "kube-system"/"original-pull-secret" not registered Nov 01 15:47:09.633288 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:09.633237 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12f347b0-ed8e-409d-bd18-261844bfeeed-metrics-certs podName:12f347b0-ed8e-409d-bd18-261844bfeeed nodeName:}" failed. No retries permitted until 2026-11-01 15:47:10.63323149 +0000 UTC m=+2.955397376 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12f347b0-ed8e-409d-bd18-261844bfeeed-metrics-certs") pod "network-metrics-daemon-cx4d8" (UID: "12f347b0-ed8e-409d-bd18-261844bfeeed") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 15:47:09.734161 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.734134 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlwtm\" (UniqueName: \"kubernetes.io/projected/34ffae54-fbc4-44f5-b785-a230835742c2-kube-api-access-nlwtm\") pod \"network-check-target-l26rw\" (UID: \"34ffae54-fbc4-44f5-b785-a230835742c2\") " pod="openshift-network-diagnostics/network-check-target-l26rw" Nov 01 15:47:09.734292 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:09.734259 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 15:47:09.734292 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:09.734271 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 15:47:09.734292 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:09.734280 2576 projected.go:196] Error preparing data for projected volume kube-api-access-nlwtm for pod openshift-network-diagnostics/network-check-target-l26rw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 15:47:09.734400 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:09.734325 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34ffae54-fbc4-44f5-b785-a230835742c2-kube-api-access-nlwtm podName:34ffae54-fbc4-44f5-b785-a230835742c2 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:10.734311052 +0000 UTC m=+3.056476954 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nlwtm" (UniqueName: "kubernetes.io/projected/34ffae54-fbc4-44f5-b785-a230835742c2-kube-api-access-nlwtm") pod "network-check-target-l26rw" (UID: "34ffae54-fbc4-44f5-b785-a230835742c2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 15:47:09.811172 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:09.811135 2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 15:47:10.061306 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:10.061194 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 15:42:09 +0000 UTC" deadline="2028-06-04 03:21:26.513526746 +0000 UTC" Nov 01 15:47:10.061306 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:10.061260 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13931h34m16.452270226s" Nov 01 15:47:10.146503 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:10.146460 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 15:47:10.186526 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:10.186441 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-86kzp" event={"ID":"b53051b0-243d-4e03-926a-1f83372e648f","Type":"ContainerStarted","Data":"9b717001af95764d3d50972a296153b9f1b650251ab71ecb27e1fdab544fca0b"} Nov 01 15:47:10.198566 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:10.198494 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-x2wlk" event={"ID":"09f28a75-b66e-4933-84e2-eb0941e5f6a3","Type":"ContainerStarted","Data":"a732a83c796496797985edc42d736f58652177e7d9e4f6e3177a4c1a9f1d84e7"} Nov 01 15:47:10.211238 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:10.211172 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-92m6d" event={"ID":"60e635cd-c1f3-4d34-b795-2f6b16438f8f","Type":"ContainerStarted","Data":"9e31f87a563ee9d4e9b6241af1725ceaed33fdd49e2b1aa14142c7ccaf6fedd4"} Nov 01 15:47:10.214232 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:10.214168 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-44klx" event={"ID":"56e8fbab-06d3-4421-9995-822247d94d90","Type":"ContainerStarted","Data":"74534d21e666655f844f8a4dce38fb0a3ed04f94c7c214a87da6d6b336832651"} Nov 01 15:47:10.223092 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:10.223033 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-238.us-east-2.compute.internal" event={"ID":"ae861e52d07eb0e1937b2b22576e5cfd","Type":"ContainerStarted","Data":"eed1a6e1117b93c7d32aa445a6b08f47f555ec343f4d3007de78f66134168519"} Nov 01 15:47:10.230224 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:10.230191 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" event={"ID":"8254aafa-dd5b-4150-87e4-d165dab6bcb5","Type":"ContainerStarted","Data":"4baf3b81c3058ccf6560267398abfaea5af29aeb6a7b9295fee685f4e2bb4fc1"} Nov 01 15:47:10.240896 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:10.240820 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f6klc" event={"ID":"4b8758dd-149c-4c37-9d32-a93fc953e548","Type":"ContainerStarted","Data":"b8db1b3a385d3d6a0dbce8ac88f785a826315f7f35eb317d4177bafa842525a6"} Nov 01 15:47:10.246459 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:10.246426 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" event={"ID":"8e87fe52-8f65-43d9-a04a-a86d6227230d","Type":"ContainerStarted","Data":"803e85f1b86f85efe617b68cd13e21b22870860367c89256d954973adff1927e"} Nov 01 15:47:10.252241 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:10.252208 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-238.us-east-2.compute.internal" event={"ID":"5d3a91f23ccfbb9c1af1cd6d04b30fd6","Type":"ContainerStarted","Data":"9d5263a747b1d06072586979cb12c5bd28b85c8eb029fe696c87a956fbdf2787"} Nov 01 15:47:10.254801 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:10.254779 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hkkmh" event={"ID":"667a7db5-cf73-424c-9d18-53bfbc5f65db","Type":"ContainerStarted","Data":"49011229e0e6d68ef5b69103031362491e74528b2942fd19f7e57d9897009e2a"} Nov 01 15:47:10.258537 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:10.258085 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"dns-node-resolver\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a8b58f028a9a4a0345ad1f809e47fb9913d7f77c731bfd785819e1c71be9472\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-dns/node-resolver-hkkmh" podUID="667a7db5-cf73-424c-9d18-53bfbc5f65db" Nov 01 15:47:10.259878 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:10.259817 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d7slg" event={"ID":"f13a984b-8706-43d5-859f-abbfcaf58abf","Type":"ContainerStarted","Data":"1fd78a54e606c6f9d9e263ddb1349684db7653078d9f5b9524e2debf12cc08f3"} Nov 01 15:47:10.389136 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:10.389059 2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 15:47:10.640411 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:10.640331 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7838987-12dc-47bc-9b39-a87ffd11ddaf-original-pull-secret\") pod \"global-pull-secret-syncer-ckz9h\" (UID: \"c7838987-12dc-47bc-9b39-a87ffd11ddaf\") " pod="kube-system/global-pull-secret-syncer-ckz9h" Nov 01 15:47:10.640590 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:10.640411 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12f347b0-ed8e-409d-bd18-261844bfeeed-metrics-certs\") pod \"network-metrics-daemon-cx4d8\" (UID: \"12f347b0-ed8e-409d-bd18-261844bfeeed\") " pod="openshift-multus/network-metrics-daemon-cx4d8" Nov 01 15:47:10.640590 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:10.640553 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 15:47:10.640702 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:10.640623 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12f347b0-ed8e-409d-bd18-261844bfeeed-metrics-certs podName:12f347b0-ed8e-409d-bd18-261844bfeeed nodeName:}" failed. No retries permitted until 2026-11-01 15:47:12.64060552 +0000 UTC m=+4.962771421 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12f347b0-ed8e-409d-bd18-261844bfeeed-metrics-certs") pod "network-metrics-daemon-cx4d8" (UID: "12f347b0-ed8e-409d-bd18-261844bfeeed") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 15:47:10.641066 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:10.641046 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 15:47:10.641163 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:10.641105 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7838987-12dc-47bc-9b39-a87ffd11ddaf-original-pull-secret podName:c7838987-12dc-47bc-9b39-a87ffd11ddaf nodeName:}" failed. No retries permitted until 2026-11-01 15:47:12.641089398 +0000 UTC m=+4.963255300 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c7838987-12dc-47bc-9b39-a87ffd11ddaf-original-pull-secret") pod "global-pull-secret-syncer-ckz9h" (UID: "c7838987-12dc-47bc-9b39-a87ffd11ddaf") : object "kube-system"/"original-pull-secret" not registered Nov 01 15:47:10.741786 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:10.741745 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlwtm\" (UniqueName: \"kubernetes.io/projected/34ffae54-fbc4-44f5-b785-a230835742c2-kube-api-access-nlwtm\") pod \"network-check-target-l26rw\" (UID: \"34ffae54-fbc4-44f5-b785-a230835742c2\") " pod="openshift-network-diagnostics/network-check-target-l26rw" Nov 01 15:47:10.741971 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:10.741915 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 15:47:10.741971 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:10.741938 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 15:47:10.741971 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:10.741951 2576 projected.go:196] Error preparing data for projected volume kube-api-access-nlwtm for pod openshift-network-diagnostics/network-check-target-l26rw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 15:47:10.742153 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:10.742011 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34ffae54-fbc4-44f5-b785-a230835742c2-kube-api-access-nlwtm podName:34ffae54-fbc4-44f5-b785-a230835742c2 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:12.741991202 +0000 UTC m=+5.064157103 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nlwtm" (UniqueName: "kubernetes.io/projected/34ffae54-fbc4-44f5-b785-a230835742c2-kube-api-access-nlwtm") pod "network-check-target-l26rw" (UID: "34ffae54-fbc4-44f5-b785-a230835742c2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 15:47:11.061758 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:11.061657 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 15:42:09 +0000 UTC" deadline="2028-07-22 19:07:01.862371654 +0000 UTC" Nov 01 15:47:11.061758 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:11.061718 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15099h19m50.800657237s" Nov 01 15:47:11.163097 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:11.162288 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx4d8" Nov 01 15:47:11.163097 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:11.162427 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx4d8" podUID="12f347b0-ed8e-409d-bd18-261844bfeeed" Nov 01 15:47:11.163097 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:11.162811 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckz9h" Nov 01 15:47:11.163097 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:11.162908 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckz9h" podUID="c7838987-12dc-47bc-9b39-a87ffd11ddaf" Nov 01 15:47:11.163097 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:11.162971 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l26rw" Nov 01 15:47:11.163097 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:11.163060 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l26rw" podUID="34ffae54-fbc4-44f5-b785-a230835742c2" Nov 01 15:47:12.662688 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:12.662375 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12f347b0-ed8e-409d-bd18-261844bfeeed-metrics-certs\") pod \"network-metrics-daemon-cx4d8\" (UID: \"12f347b0-ed8e-409d-bd18-261844bfeeed\") " pod="openshift-multus/network-metrics-daemon-cx4d8" Nov 01 15:47:12.662688 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:12.662454 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7838987-12dc-47bc-9b39-a87ffd11ddaf-original-pull-secret\") pod \"global-pull-secret-syncer-ckz9h\" (UID: \"c7838987-12dc-47bc-9b39-a87ffd11ddaf\") " pod="kube-system/global-pull-secret-syncer-ckz9h" Nov 01 15:47:12.662688 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:12.662547 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 15:47:12.662688 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:12.662581 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 15:47:12.662688 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:12.662620 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12f347b0-ed8e-409d-bd18-261844bfeeed-metrics-certs podName:12f347b0-ed8e-409d-bd18-261844bfeeed nodeName:}" failed. No retries permitted until 2026-11-01 15:47:16.662601602 +0000 UTC m=+8.984767503 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12f347b0-ed8e-409d-bd18-261844bfeeed-metrics-certs") pod "network-metrics-daemon-cx4d8" (UID: "12f347b0-ed8e-409d-bd18-261844bfeeed") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 15:47:12.662688 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:12.662645 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7838987-12dc-47bc-9b39-a87ffd11ddaf-original-pull-secret podName:c7838987-12dc-47bc-9b39-a87ffd11ddaf nodeName:}" failed. No retries permitted until 2026-11-01 15:47:16.662635478 +0000 UTC m=+8.984801366 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c7838987-12dc-47bc-9b39-a87ffd11ddaf-original-pull-secret") pod "global-pull-secret-syncer-ckz9h" (UID: "c7838987-12dc-47bc-9b39-a87ffd11ddaf") : object "kube-system"/"original-pull-secret" not registered Nov 01 15:47:12.764012 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:12.763363 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlwtm\" (UniqueName: \"kubernetes.io/projected/34ffae54-fbc4-44f5-b785-a230835742c2-kube-api-access-nlwtm\") pod \"network-check-target-l26rw\" (UID: \"34ffae54-fbc4-44f5-b785-a230835742c2\") " pod="openshift-network-diagnostics/network-check-target-l26rw" Nov 01 15:47:12.764012 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:12.763555 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 15:47:12.764012 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:12.763581 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 15:47:12.764012 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:12.763594 2576 projected.go:196] Error preparing data for projected volume kube-api-access-nlwtm for pod openshift-network-diagnostics/network-check-target-l26rw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 15:47:12.764012 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:12.763651 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34ffae54-fbc4-44f5-b785-a230835742c2-kube-api-access-nlwtm podName:34ffae54-fbc4-44f5-b785-a230835742c2 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:16.763632478 +0000 UTC m=+9.085798369 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nlwtm" (UniqueName: "kubernetes.io/projected/34ffae54-fbc4-44f5-b785-a230835742c2-kube-api-access-nlwtm") pod "network-check-target-l26rw" (UID: "34ffae54-fbc4-44f5-b785-a230835742c2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 15:47:13.161771 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:13.161726 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckz9h" Nov 01 15:47:13.161927 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:13.161865 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckz9h" podUID="c7838987-12dc-47bc-9b39-a87ffd11ddaf" Nov 01 15:47:13.162286 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:13.162264 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l26rw" Nov 01 15:47:13.162394 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:13.162368 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l26rw" podUID="34ffae54-fbc4-44f5-b785-a230835742c2" Nov 01 15:47:13.162463 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:13.162421 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx4d8" Nov 01 15:47:13.162517 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:13.162500 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx4d8" podUID="12f347b0-ed8e-409d-bd18-261844bfeeed" Nov 01 15:47:13.278204 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:13.278169 2576 generic.go:356] "Generic (PLEG): container finished" podID="4b8758dd-149c-4c37-9d32-a93fc953e548" containerID="8643efd57aa767d3e191eeeadd2a91fdaa4c4dafeff7c800d2e8371fdcc4e490" exitCode=0 Nov 01 15:47:13.278319 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:13.278220 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f6klc" event={"ID":"4b8758dd-149c-4c37-9d32-a93fc953e548","Type":"ContainerDied","Data":"8643efd57aa767d3e191eeeadd2a91fdaa4c4dafeff7c800d2e8371fdcc4e490"} Nov 01 15:47:14.284529 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:14.284230 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-238.us-east-2.compute.internal" event={"ID":"ae861e52d07eb0e1937b2b22576e5cfd","Type":"ContainerStarted","Data":"a8353b93a829549def42b86442c888f2e0d6c41d1372611e7b8188895cb4a281"} Nov 01 15:47:14.289065 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:14.287704 2576 generic.go:356] "Generic (PLEG): container finished" podID="5d3a91f23ccfbb9c1af1cd6d04b30fd6" containerID="c13981f4537784a0dbec6ec40fe85598cf111a0142799131942fd7bce219988a" exitCode=0 Nov 01 15:47:14.289065 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:14.287781 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-238.us-east-2.compute.internal" event={"ID":"5d3a91f23ccfbb9c1af1cd6d04b30fd6","Type":"ContainerDied","Data":"c13981f4537784a0dbec6ec40fe85598cf111a0142799131942fd7bce219988a"} Nov 01 15:47:14.291432 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:14.291405 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d7slg" event={"ID":"f13a984b-8706-43d5-859f-abbfcaf58abf","Type":"ContainerStarted","Data":"f1d4572da40cb6b07af8a7810b2e1a31a8622775273a214d10208a7c9b614b34"} Nov 01 15:47:14.317517 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:14.317452 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-238.us-east-2.compute.internal" podStartSLOduration=6.317409103 podStartE2EDuration="6.317409103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 15:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:14.300618064 +0000 UTC m=+6.622783972" watchObservedRunningTime="2026-11-01 15:47:14.317409103 +0000 UTC m=+6.639575012" Nov 01 15:47:14.336767 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:14.336715 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-d7slg" podStartSLOduration=1.887947213 podStartE2EDuration="6.336695764s" totalImagesPullingTime="4.448748551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:47:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:14.336325611 +0000 UTC m=+6.658491521" watchObservedRunningTime="2026-11-01 15:47:14.336695764 +0000 UTC m=+6.658861673" Nov 01 15:47:14.403037 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:14.402962 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-d7slg" Nov 01 15:47:14.403698 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:14.403572 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-d7slg" Nov 01 15:47:15.161824 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:15.161793 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx4d8" Nov 01 15:47:15.161969 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:15.161944 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx4d8" podUID="12f347b0-ed8e-409d-bd18-261844bfeeed" Nov 01 15:47:15.162397 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:15.162375 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckz9h" Nov 01 15:47:15.162509 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:15.162483 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckz9h" podUID="c7838987-12dc-47bc-9b39-a87ffd11ddaf" Nov 01 15:47:15.162579 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:15.162554 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l26rw" Nov 01 15:47:15.162653 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:15.162624 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l26rw" podUID="34ffae54-fbc4-44f5-b785-a230835742c2" Nov 01 15:47:15.298273 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:15.298239 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-44klx" event={"ID":"56e8fbab-06d3-4421-9995-822247d94d90","Type":"ContainerStarted","Data":"2c76eff67b94abf2b739c9dc5292ff9e509b2fe34dab249bd26b1922625647a3"} Nov 01 15:47:15.305103 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:15.304383 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-238.us-east-2.compute.internal" event={"ID":"5d3a91f23ccfbb9c1af1cd6d04b30fd6","Type":"ContainerStarted","Data":"3ef08cfc3e9507e6623121bf67835e082b661d8525bcb00fbafabf7e73f68ce9"} Nov 01 15:47:15.314662 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:15.314428 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-44klx" podStartSLOduration=1.9844595630000001 podStartE2EDuration="7.314409801s" totalImagesPullingTime="5.329950238s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 15:47:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:15.313342732 +0000 UTC m=+7.635508641" watchObservedRunningTime="2026-11-01 15:47:15.314409801 +0000 UTC m=+7.636575714" Nov 01 15:47:15.331239 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:15.331189 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-238.us-east-2.compute.internal" podStartSLOduration=7.331171057 podStartE2EDuration="7.331171057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 15:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:15.330982246 +0000 UTC m=+7.653148157" watchObservedRunningTime="2026-11-01 15:47:15.331171057 +0000 UTC m=+7.653336969" Nov 01 15:47:16.310462 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:16.309569 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" event={"ID":"8e87fe52-8f65-43d9-a04a-a86d6227230d","Type":"ContainerStarted","Data":"abc665b692c730ba7403dfb7ea39f14c8f0e789dc66e282f92a909dd219674db"} Nov 01 15:47:16.310462 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:16.310109 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 15:47:16.700145 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:16.699331 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12f347b0-ed8e-409d-bd18-261844bfeeed-metrics-certs\") pod \"network-metrics-daemon-cx4d8\" (UID: \"12f347b0-ed8e-409d-bd18-261844bfeeed\") " pod="openshift-multus/network-metrics-daemon-cx4d8" Nov 01 15:47:16.700145 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:16.699415 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7838987-12dc-47bc-9b39-a87ffd11ddaf-original-pull-secret\") pod \"global-pull-secret-syncer-ckz9h\" (UID: \"c7838987-12dc-47bc-9b39-a87ffd11ddaf\") " pod="kube-system/global-pull-secret-syncer-ckz9h" Nov 01 15:47:16.700145 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:16.699574 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 15:47:16.700145 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:16.699645 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7838987-12dc-47bc-9b39-a87ffd11ddaf-original-pull-secret podName:c7838987-12dc-47bc-9b39-a87ffd11ddaf nodeName:}" failed. No retries permitted until 2026-11-01 15:47:24.699626929 +0000 UTC m=+17.021792838 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c7838987-12dc-47bc-9b39-a87ffd11ddaf-original-pull-secret") pod "global-pull-secret-syncer-ckz9h" (UID: "c7838987-12dc-47bc-9b39-a87ffd11ddaf") : object "kube-system"/"original-pull-secret" not registered Nov 01 15:47:16.700145 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:16.699736 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 15:47:16.700145 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:16.699774 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12f347b0-ed8e-409d-bd18-261844bfeeed-metrics-certs podName:12f347b0-ed8e-409d-bd18-261844bfeeed nodeName:}" failed. No retries permitted until 2026-11-01 15:47:24.69976155 +0000 UTC m=+17.021927436 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12f347b0-ed8e-409d-bd18-261844bfeeed-metrics-certs") pod "network-metrics-daemon-cx4d8" (UID: "12f347b0-ed8e-409d-bd18-261844bfeeed") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 15:47:16.792626 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:16.792592 2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 15:47:16.800626 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:16.800593 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlwtm\" (UniqueName: \"kubernetes.io/projected/34ffae54-fbc4-44f5-b785-a230835742c2-kube-api-access-nlwtm\") pod \"network-check-target-l26rw\" (UID: \"34ffae54-fbc4-44f5-b785-a230835742c2\") " pod="openshift-network-diagnostics/network-check-target-l26rw" Nov 01 15:47:16.800762 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:16.800740 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 15:47:16.800831 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:16.800770 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 15:47:16.800831 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:16.800783 2576 projected.go:196] Error preparing data for projected volume kube-api-access-nlwtm for pod openshift-network-diagnostics/network-check-target-l26rw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 15:47:16.800831 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:16.800830 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34ffae54-fbc4-44f5-b785-a230835742c2-kube-api-access-nlwtm podName:34ffae54-fbc4-44f5-b785-a230835742c2 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:24.800811368 +0000 UTC m=+17.122977270 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nlwtm" (UniqueName: "kubernetes.io/projected/34ffae54-fbc4-44f5-b785-a230835742c2-kube-api-access-nlwtm") pod "network-check-target-l26rw" (UID: "34ffae54-fbc4-44f5-b785-a230835742c2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 15:47:17.078329 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:17.078167 2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T15:47:16.792608437Z","UUID":"8d6bba67-0e3d-4b7c-8d8e-6f6e2bbac3aa","Handler":null,"Name":"","Endpoint":""} Nov 01 15:47:17.079875 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:17.079850 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 15:47:17.080005 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:17.079888 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 15:47:17.162217 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:17.162184 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l26rw" Nov 01 15:47:17.162323 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:17.162301 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l26rw" podUID="34ffae54-fbc4-44f5-b785-a230835742c2" Nov 01 15:47:17.162391 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:17.162370 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx4d8" Nov 01 15:47:17.162436 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:17.162421 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx4d8" podUID="12f347b0-ed8e-409d-bd18-261844bfeeed" Nov 01 15:47:17.162486 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:17.162459 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckz9h" Nov 01 15:47:17.162531 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:17.162503 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckz9h" podUID="c7838987-12dc-47bc-9b39-a87ffd11ddaf" Nov 01 15:47:17.312655 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:17.312627 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-x2wlk" event={"ID":"09f28a75-b66e-4933-84e2-eb0941e5f6a3","Type":"ContainerStarted","Data":"9594da51ef046ae3dbb551a4dd4678982e413e6bc64c5d2d3ac0cb9cc9342176"} Nov 01 15:47:17.315335 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:17.315308 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" event={"ID":"8e87fe52-8f65-43d9-a04a-a86d6227230d","Type":"ContainerStarted","Data":"29ef7d2ac17295c174f8501f8de09286a057ae258765def48d91c6178c79429a"} Nov 01 15:47:17.331101 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:17.330095 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-x2wlk" podStartSLOduration=1.962056187 podStartE2EDuration="9.330080155s" totalImagesPullingTime="7.368023968s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 15:47:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:17.329967614 +0000 UTC m=+9.652133522" watchObservedRunningTime="2026-11-01 15:47:17.330080155 +0000 UTC m=+9.652246065" Nov 01 15:47:18.319573 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:18.319535 2576 generic.go:356] "Generic (PLEG): container finished" podID="4b8758dd-149c-4c37-9d32-a93fc953e548" containerID="c5ffd3456da5aa6e9a10e8aeae741dc451fb4b0f8ca3c3fec7d3fd15039ae740" exitCode=0 Nov 01 15:47:18.320080 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:18.319634 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f6klc" event={"ID":"4b8758dd-149c-4c37-9d32-a93fc953e548","Type":"ContainerDied","Data":"c5ffd3456da5aa6e9a10e8aeae741dc451fb4b0f8ca3c3fec7d3fd15039ae740"} Nov 01 15:47:18.322097 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:18.322060 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" event={"ID":"8e87fe52-8f65-43d9-a04a-a86d6227230d","Type":"ContainerStarted","Data":"310a239f4ee38575642a2132a917d1cfb3e12edaa479e1fab7c66e815ce6202b"} Nov 01 15:47:18.358395 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:18.358344 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" podStartSLOduration=2.261082926 podStartE2EDuration="10.358327068s" totalImagesPullingTime="8.097244142s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 15:47:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:18.358102065 +0000 UTC m=+10.680267974" watchObservedRunningTime="2026-11-01 15:47:18.358327068 +0000 UTC m=+10.680492985" Nov 01 15:47:19.162089 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:19.161892 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckz9h" Nov 01 15:47:19.162243 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:19.161892 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx4d8" Nov 01 15:47:19.162243 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:19.162190 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckz9h" podUID="c7838987-12dc-47bc-9b39-a87ffd11ddaf" Nov 01 15:47:19.162243 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:19.161906 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l26rw" Nov 01 15:47:19.162400 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:19.162305 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx4d8" podUID="12f347b0-ed8e-409d-bd18-261844bfeeed" Nov 01 15:47:19.162400 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:19.162374 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l26rw" podUID="34ffae54-fbc4-44f5-b785-a230835742c2" Nov 01 15:47:19.326360 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:19.326275 2576 generic.go:356] "Generic (PLEG): container finished" podID="4b8758dd-149c-4c37-9d32-a93fc953e548" containerID="f982b3d4ac58ef8e5d3b1e00c5dfded0ea728c741dfdd53ee697e7c201c31b4b" exitCode=0 Nov 01 15:47:19.326759 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:19.326372 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f6klc" event={"ID":"4b8758dd-149c-4c37-9d32-a93fc953e548","Type":"ContainerDied","Data":"f982b3d4ac58ef8e5d3b1e00c5dfded0ea728c741dfdd53ee697e7c201c31b4b"} Nov 01 15:47:19.329131 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:19.329101 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-86kzp" event={"ID":"b53051b0-243d-4e03-926a-1f83372e648f","Type":"ContainerStarted","Data":"7b306cf91ee5d7f7c13a1c627f0622f32d2bbc8333ba6d460e68622e0a885e0a"} Nov 01 15:47:19.330747 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:19.330724 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-92m6d" event={"ID":"60e635cd-c1f3-4d34-b795-2f6b16438f8f","Type":"ContainerStarted","Data":"3dadd2274e72bf60e859daa6565b58b8c19480d164bbf480d1597580a01297c2"} Nov 01 15:47:19.363481 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:19.363441 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-86kzp" podStartSLOduration=3.639901846 podStartE2EDuration="11.363431234s" totalImagesPullingTime="7.723529388s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 15:47:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:19.363114339 +0000 UTC m=+11.685280247" watchObservedRunningTime="2026-11-01 15:47:19.363431234 +0000 UTC m=+11.685597143" Nov 01 15:47:19.380573 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:19.380526 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-92m6d" podStartSLOduration=1.828378611 podStartE2EDuration="11.380510854s" totalImagesPullingTime="9.552132243s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 15:47:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:19.380402548 +0000 UTC m=+11.702568459" watchObservedRunningTime="2026-11-01 15:47:19.380510854 +0000 UTC m=+11.702676766" Nov 01 15:47:20.335928 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:20.335836 2576 generic.go:356] "Generic (PLEG): container finished" podID="4b8758dd-149c-4c37-9d32-a93fc953e548" containerID="3704f772e47fb53c46a08a5445a3865de8b608c61c1a9a1665951a7d79223023" exitCode=0 Nov 01 15:47:20.336549 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:20.335934 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f6klc" event={"ID":"4b8758dd-149c-4c37-9d32-a93fc953e548","Type":"ContainerDied","Data":"3704f772e47fb53c46a08a5445a3865de8b608c61c1a9a1665951a7d79223023"} Nov 01 15:47:21.161995 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:21.161959 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckz9h" Nov 01 15:47:21.162194 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:21.162108 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckz9h" podUID="c7838987-12dc-47bc-9b39-a87ffd11ddaf" Nov 01 15:47:21.162955 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:21.162647 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l26rw" Nov 01 15:47:21.162955 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:21.162755 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l26rw" podUID="34ffae54-fbc4-44f5-b785-a230835742c2" Nov 01 15:47:21.162955 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:21.162822 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx4d8" Nov 01 15:47:21.162955 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:21.162920 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx4d8" podUID="12f347b0-ed8e-409d-bd18-261844bfeeed" Nov 01 15:47:22.342834 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:22.342641 2576 generic.go:356] "Generic (PLEG): container finished" podID="4b8758dd-149c-4c37-9d32-a93fc953e548" containerID="15905a15a06fb738a13d5f2da47cddb9903ee5acc823dee7632fb496f08ee5ea" exitCode=0 Nov 01 15:47:22.343388 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:22.342725 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f6klc" event={"ID":"4b8758dd-149c-4c37-9d32-a93fc953e548","Type":"ContainerDied","Data":"15905a15a06fb738a13d5f2da47cddb9903ee5acc823dee7632fb496f08ee5ea"} Nov 01 15:47:22.344409 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:22.344385 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hkkmh" event={"ID":"667a7db5-cf73-424c-9d18-53bfbc5f65db","Type":"ContainerStarted","Data":"daef981bedfba746072afac2c55e79263510ff78bc01f337ad73f75e18711f0b"} Nov 01 15:47:22.380166 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:22.380109 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-hkkmh" podStartSLOduration=1.683559352 podStartE2EDuration="14.380094619s" totalImagesPullingTime="12.696535267s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 15:47:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:22.379800056 +0000 UTC m=+14.701965967" watchObservedRunningTime="2026-11-01 15:47:22.380094619 +0000 UTC m=+14.702260528" Nov 01 15:47:23.162036 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:23.161974 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckz9h" Nov 01 15:47:23.162215 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:23.162095 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx4d8" Nov 01 15:47:23.162215 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:23.162208 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l26rw" Nov 01 15:47:23.162346 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:23.162230 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckz9h" podUID="c7838987-12dc-47bc-9b39-a87ffd11ddaf" Nov 01 15:47:23.162346 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:23.162281 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l26rw" podUID="34ffae54-fbc4-44f5-b785-a230835742c2" Nov 01 15:47:23.162346 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:23.162335 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx4d8" podUID="12f347b0-ed8e-409d-bd18-261844bfeeed" Nov 01 15:47:23.351293 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:23.351258 2576 generic.go:356] "Generic (PLEG): container finished" podID="4b8758dd-149c-4c37-9d32-a93fc953e548" containerID="7890e88a268d8b9b6f9a79069478a524f89fe6104000d6fdb7031ba7f0650af0" exitCode=0 Nov 01 15:47:23.351702 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:23.351319 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f6klc" event={"ID":"4b8758dd-149c-4c37-9d32-a93fc953e548","Type":"ContainerDied","Data":"7890e88a268d8b9b6f9a79069478a524f89fe6104000d6fdb7031ba7f0650af0"} Nov 01 15:47:24.356241 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:24.356004 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtwhw_8254aafa-dd5b-4150-87e4-d165dab6bcb5/ovn-acl-logging/0.log" Nov 01 15:47:24.356585 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:24.356481 2576 generic.go:356] "Generic (PLEG): container finished" podID="8254aafa-dd5b-4150-87e4-d165dab6bcb5" containerID="d5b99f7a3b5406a8015f71ca144410db7f2500a34e0d1d80f0036060092d7a2a" exitCode=1 Nov 01 15:47:24.356585 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:24.356559 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" event={"ID":"8254aafa-dd5b-4150-87e4-d165dab6bcb5","Type":"ContainerStarted","Data":"cf945dc11ca9e75cf5688fec80e4593159402724f635b769738638dd158023c1"} Nov 01 15:47:24.356677 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:24.356665 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" event={"ID":"8254aafa-dd5b-4150-87e4-d165dab6bcb5","Type":"ContainerStarted","Data":"8219f410e6565e562da0929b3fc3a6f6a54b099d8504554060a328beeb3bce3e"} Nov 01 15:47:24.356715 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:24.356697 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" event={"ID":"8254aafa-dd5b-4150-87e4-d165dab6bcb5","Type":"ContainerStarted","Data":"d2c88b58c95a04aa618a299479429fabef582270d6a95bd57dfbd4e16da2d541"} Nov 01 15:47:24.356766 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:24.356725 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" event={"ID":"8254aafa-dd5b-4150-87e4-d165dab6bcb5","Type":"ContainerStarted","Data":"8ba7fb2562426fed3b8b4bb348b8d09bcbd3519e58507581c1165397fecf5e54"} Nov 01 15:47:24.356766 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:24.356759 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" event={"ID":"8254aafa-dd5b-4150-87e4-d165dab6bcb5","Type":"ContainerDied","Data":"d5b99f7a3b5406a8015f71ca144410db7f2500a34e0d1d80f0036060092d7a2a"} Nov 01 15:47:24.356862 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:24.356788 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" event={"ID":"8254aafa-dd5b-4150-87e4-d165dab6bcb5","Type":"ContainerStarted","Data":"4a1bdb4f448219946c9e7c204e9dec07c7c6365c0e177e6820706630dd6c082e"} Nov 01 15:47:24.359799 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:24.359776 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f6klc" event={"ID":"4b8758dd-149c-4c37-9d32-a93fc953e548","Type":"ContainerStarted","Data":"da775f0777b0f8aedfb94924ef420102aac0c3a681800f3cc551904a1a41e815"} Nov 01 15:47:24.382644 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:24.382607 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-f6klc" podStartSLOduration=6.59445897 podStartE2EDuration="16.382597655s" totalImagesPullingTime="9.788138685s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 15:47:08 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:24.380733116 +0000 UTC m=+16.702899026" watchObservedRunningTime="2026-11-01 15:47:24.382597655 +0000 UTC m=+16.704763563" Nov 01 15:47:24.759438 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:24.759366 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12f347b0-ed8e-409d-bd18-261844bfeeed-metrics-certs\") pod \"network-metrics-daemon-cx4d8\" (UID: \"12f347b0-ed8e-409d-bd18-261844bfeeed\") " pod="openshift-multus/network-metrics-daemon-cx4d8" Nov 01 15:47:24.759438 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:24.759412 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7838987-12dc-47bc-9b39-a87ffd11ddaf-original-pull-secret\") pod \"global-pull-secret-syncer-ckz9h\" (UID: \"c7838987-12dc-47bc-9b39-a87ffd11ddaf\") " pod="kube-system/global-pull-secret-syncer-ckz9h" Nov 01 15:47:24.759650 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:24.759503 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 15:47:24.759650 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:24.759517 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 15:47:24.759650 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:24.759551 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7838987-12dc-47bc-9b39-a87ffd11ddaf-original-pull-secret podName:c7838987-12dc-47bc-9b39-a87ffd11ddaf nodeName:}" failed. No retries permitted until 2026-11-01 15:47:40.759540192 +0000 UTC m=+33.081706092 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c7838987-12dc-47bc-9b39-a87ffd11ddaf-original-pull-secret") pod "global-pull-secret-syncer-ckz9h" (UID: "c7838987-12dc-47bc-9b39-a87ffd11ddaf") : object "kube-system"/"original-pull-secret" not registered Nov 01 15:47:24.759650 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:24.759574 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12f347b0-ed8e-409d-bd18-261844bfeeed-metrics-certs podName:12f347b0-ed8e-409d-bd18-261844bfeeed nodeName:}" failed. No retries permitted until 2026-11-01 15:47:40.759557426 +0000 UTC m=+33.081723313 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12f347b0-ed8e-409d-bd18-261844bfeeed-metrics-certs") pod "network-metrics-daemon-cx4d8" (UID: "12f347b0-ed8e-409d-bd18-261844bfeeed") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 15:47:24.860545 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:24.860519 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlwtm\" (UniqueName: \"kubernetes.io/projected/34ffae54-fbc4-44f5-b785-a230835742c2-kube-api-access-nlwtm\") pod \"network-check-target-l26rw\" (UID: \"34ffae54-fbc4-44f5-b785-a230835742c2\") " pod="openshift-network-diagnostics/network-check-target-l26rw" Nov 01 15:47:24.860657 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:24.860645 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 15:47:24.860708 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:24.860657 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 15:47:24.860708 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:24.860666 2576 projected.go:196] Error preparing data for projected volume kube-api-access-nlwtm for pod openshift-network-diagnostics/network-check-target-l26rw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 15:47:24.860708 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:24.860706 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34ffae54-fbc4-44f5-b785-a230835742c2-kube-api-access-nlwtm podName:34ffae54-fbc4-44f5-b785-a230835742c2 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:40.860695063 +0000 UTC m=+33.182860950 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nlwtm" (UniqueName: "kubernetes.io/projected/34ffae54-fbc4-44f5-b785-a230835742c2-kube-api-access-nlwtm") pod "network-check-target-l26rw" (UID: "34ffae54-fbc4-44f5-b785-a230835742c2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 15:47:25.161948 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:25.161866 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx4d8" Nov 01 15:47:25.162120 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:25.161866 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckz9h" Nov 01 15:47:25.162120 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:25.161971 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx4d8" podUID="12f347b0-ed8e-409d-bd18-261844bfeeed" Nov 01 15:47:25.162120 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:25.161870 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l26rw" Nov 01 15:47:25.162120 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:25.162096 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckz9h" podUID="c7838987-12dc-47bc-9b39-a87ffd11ddaf" Nov 01 15:47:25.162982 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:25.162162 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l26rw" podUID="34ffae54-fbc4-44f5-b785-a230835742c2" Nov 01 15:47:26.366666 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:26.366641 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtwhw_8254aafa-dd5b-4150-87e4-d165dab6bcb5/ovn-acl-logging/0.log" Nov 01 15:47:26.367130 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:26.367031 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" event={"ID":"8254aafa-dd5b-4150-87e4-d165dab6bcb5","Type":"ContainerStarted","Data":"3ed9e39a76852c33eeb803871f42d5befe43949fae04719633df9f6525478ced"} Nov 01 15:47:27.161994 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:27.161949 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx4d8" Nov 01 15:47:27.161994 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:27.161980 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckz9h" Nov 01 15:47:27.162249 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:27.161949 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l26rw" Nov 01 15:47:27.162249 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:27.162080 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx4d8" podUID="12f347b0-ed8e-409d-bd18-261844bfeeed" Nov 01 15:47:27.162249 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:27.162134 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l26rw" podUID="34ffae54-fbc4-44f5-b785-a230835742c2" Nov 01 15:47:27.162249 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:27.162190 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckz9h" podUID="c7838987-12dc-47bc-9b39-a87ffd11ddaf" Nov 01 15:47:28.270767 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:28.270736 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-d7slg" Nov 01 15:47:28.271334 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:28.270865 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 15:47:28.271334 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:28.271302 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-d7slg" Nov 01 15:47:28.374776 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:28.374752 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtwhw_8254aafa-dd5b-4150-87e4-d165dab6bcb5/ovn-acl-logging/0.log" Nov 01 15:47:28.375260 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:28.375223 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" event={"ID":"8254aafa-dd5b-4150-87e4-d165dab6bcb5","Type":"ContainerStarted","Data":"dfe50ea6d4d18df90d5f826987c0b92486616f3c032db469918ecdf934d57cc1"} Nov 01 15:47:28.375606 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:28.375579 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:28.375734 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:28.375692 2576 scope.go:122] "RemoveContainer" containerID="d5b99f7a3b5406a8015f71ca144410db7f2500a34e0d1d80f0036060092d7a2a" Nov 01 15:47:28.391566 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:28.391546 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:29.162229 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:29.162197 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckz9h" Nov 01 15:47:29.162381 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:29.162278 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx4d8" Nov 01 15:47:29.162381 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:29.162302 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l26rw" Nov 01 15:47:29.162455 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:29.162373 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckz9h" podUID="c7838987-12dc-47bc-9b39-a87ffd11ddaf" Nov 01 15:47:29.162455 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:29.162417 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l26rw" podUID="34ffae54-fbc4-44f5-b785-a230835742c2" Nov 01 15:47:29.162515 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:29.162491 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx4d8" podUID="12f347b0-ed8e-409d-bd18-261844bfeeed" Nov 01 15:47:29.380746 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:29.380720 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtwhw_8254aafa-dd5b-4150-87e4-d165dab6bcb5/ovn-acl-logging/0.log" Nov 01 15:47:29.381503 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:29.381070 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" event={"ID":"8254aafa-dd5b-4150-87e4-d165dab6bcb5","Type":"ContainerStarted","Data":"a238ec8229ed44ca0bcd27aa8ef01c1af4bd6979a3a9a23e83c01edc17f732be"} Nov 01 15:47:29.381503 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:29.381212 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:29.381503 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:29.381276 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:29.395859 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:29.395839 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:29.415170 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:29.415105 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" podStartSLOduration=7.219714983 podStartE2EDuration="21.415094023s" totalImagesPullingTime="14.19537904s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 15:47:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:29.414679009 +0000 UTC m=+21.736844929" watchObservedRunningTime="2026-11-01 15:47:29.415094023 +0000 UTC m=+21.737259931" Nov 01 15:47:29.949595 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:29.949396 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cx4d8"] Nov 01 15:47:29.949734 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:29.949678 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx4d8" Nov 01 15:47:29.949811 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:29.949791 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx4d8" podUID="12f347b0-ed8e-409d-bd18-261844bfeeed" Nov 01 15:47:29.951412 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:29.951385 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-l26rw"] Nov 01 15:47:29.951529 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:29.951432 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ckz9h"] Nov 01 15:47:29.951529 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:29.951484 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckz9h" Nov 01 15:47:29.951612 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:29.951549 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckz9h" podUID="c7838987-12dc-47bc-9b39-a87ffd11ddaf" Nov 01 15:47:29.951612 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:29.951586 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l26rw" Nov 01 15:47:29.951673 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:29.951630 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l26rw" podUID="34ffae54-fbc4-44f5-b785-a230835742c2" Nov 01 15:47:31.161678 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:31.161638 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx4d8" Nov 01 15:47:31.162131 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:31.161758 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx4d8" podUID="12f347b0-ed8e-409d-bd18-261844bfeeed" Nov 01 15:47:31.162131 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:31.161808 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckz9h" Nov 01 15:47:31.162131 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:31.161884 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckz9h" podUID="c7838987-12dc-47bc-9b39-a87ffd11ddaf" Nov 01 15:47:32.161498 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:32.161471 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l26rw" Nov 01 15:47:32.161727 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:32.161575 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l26rw" podUID="34ffae54-fbc4-44f5-b785-a230835742c2" Nov 01 15:47:33.162090 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:33.162007 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx4d8" Nov 01 15:47:33.162432 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:33.162116 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx4d8" podUID="12f347b0-ed8e-409d-bd18-261844bfeeed" Nov 01 15:47:33.162432 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:33.162169 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckz9h" Nov 01 15:47:33.162432 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:33.162253 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckz9h" podUID="c7838987-12dc-47bc-9b39-a87ffd11ddaf" Nov 01 15:47:34.161950 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:34.161918 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l26rw" Nov 01 15:47:34.162145 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:34.162039 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l26rw" podUID="34ffae54-fbc4-44f5-b785-a230835742c2" Nov 01 15:47:35.161614 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:35.161579 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckz9h" Nov 01 15:47:35.161772 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:35.161692 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckz9h" podUID="c7838987-12dc-47bc-9b39-a87ffd11ddaf" Nov 01 15:47:35.161772 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:35.161721 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx4d8" Nov 01 15:47:35.161839 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:35.161802 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx4d8" podUID="12f347b0-ed8e-409d-bd18-261844bfeeed" Nov 01 15:47:36.161865 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:36.161840 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l26rw" Nov 01 15:47:36.162207 ip-10-0-128-238 kubenswrapper[2576]: E1101 15:47:36.161944 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l26rw" podUID="34ffae54-fbc4-44f5-b785-a230835742c2" Nov 01 15:47:36.497135 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:36.497066 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-238.us-east-2.compute.internal" event="NodeReady" Nov 01 15:47:36.497263 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:36.497161 2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 01 15:47:37.161739 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:37.161708 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx4d8" Nov 01 15:47:37.161739 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:37.161733 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckz9h" Nov 01 15:47:37.167232 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:37.167208 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4p8cp\"" Nov 01 15:47:37.167624 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:37.167361 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-qm9xj\"" Nov 01 15:47:37.167624 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:37.167387 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 15:47:37.167624 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:37.167493 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 15:47:38.162914 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:38.162883 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l26rw" Nov 01 15:47:38.167092 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:38.167070 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 15:47:38.167092 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:38.167085 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 15:47:38.167281 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:38.167182 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-2bqxm\"" Nov 01 15:47:40.776926 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:40.776854 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12f347b0-ed8e-409d-bd18-261844bfeeed-metrics-certs\") pod \"network-metrics-daemon-cx4d8\" (UID: \"12f347b0-ed8e-409d-bd18-261844bfeeed\") " pod="openshift-multus/network-metrics-daemon-cx4d8" Nov 01 15:47:40.776926 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:40.776911 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7838987-12dc-47bc-9b39-a87ffd11ddaf-original-pull-secret\") pod \"global-pull-secret-syncer-ckz9h\" (UID: \"c7838987-12dc-47bc-9b39-a87ffd11ddaf\") " pod="kube-system/global-pull-secret-syncer-ckz9h" Nov 01 15:47:40.780207 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:40.780183 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7838987-12dc-47bc-9b39-a87ffd11ddaf-original-pull-secret\") pod \"global-pull-secret-syncer-ckz9h\" (UID: \"c7838987-12dc-47bc-9b39-a87ffd11ddaf\") " pod="kube-system/global-pull-secret-syncer-ckz9h" Nov 01 15:47:40.780318 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:40.780301 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12f347b0-ed8e-409d-bd18-261844bfeeed-metrics-certs\") pod \"network-metrics-daemon-cx4d8\" (UID: \"12f347b0-ed8e-409d-bd18-261844bfeeed\") " pod="openshift-multus/network-metrics-daemon-cx4d8" Nov 01 15:47:40.781170 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:40.781155 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx4d8" Nov 01 15:47:40.784316 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:40.784294 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckz9h" Nov 01 15:47:40.877898 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:40.877863 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlwtm\" (UniqueName: \"kubernetes.io/projected/34ffae54-fbc4-44f5-b785-a230835742c2-kube-api-access-nlwtm\") pod \"network-check-target-l26rw\" (UID: \"34ffae54-fbc4-44f5-b785-a230835742c2\") " pod="openshift-network-diagnostics/network-check-target-l26rw" Nov 01 15:47:40.880905 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:40.880880 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlwtm\" (UniqueName: \"kubernetes.io/projected/34ffae54-fbc4-44f5-b785-a230835742c2-kube-api-access-nlwtm\") pod \"network-check-target-l26rw\" (UID: \"34ffae54-fbc4-44f5-b785-a230835742c2\") " pod="openshift-network-diagnostics/network-check-target-l26rw" Nov 01 15:47:40.917759 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:40.917730 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l26rw" Nov 01 15:47:41.151192 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:41.151127 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cx4d8"] Nov 01 15:47:41.154039 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:41.154000 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ckz9h"] Nov 01 15:47:41.154456 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:41.154434 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12f347b0_ed8e_409d_bd18_261844bfeeed.slice/crio-a9677234a195d43fd94578b1fbc59beea601e64e2fb660e308314e87433dd19e WatchSource:0}: Error finding container a9677234a195d43fd94578b1fbc59beea601e64e2fb660e308314e87433dd19e: Status 404 returned error can't find the container with id a9677234a195d43fd94578b1fbc59beea601e64e2fb660e308314e87433dd19e Nov 01 15:47:41.156674 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:41.156645 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7838987_12dc_47bc_9b39_a87ffd11ddaf.slice/crio-7be9b9bfa2d4719d2e9b46c2a57a91f446ec1ebdcfaba01f41170b83d76efd60 WatchSource:0}: Error finding container 7be9b9bfa2d4719d2e9b46c2a57a91f446ec1ebdcfaba01f41170b83d76efd60: Status 404 returned error can't find the container with id 7be9b9bfa2d4719d2e9b46c2a57a91f446ec1ebdcfaba01f41170b83d76efd60 Nov 01 15:47:41.203992 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:41.203966 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-l26rw"] Nov 01 15:47:41.207090 ip-10-0-128-238 kubenswrapper[2576]: W1101 15:47:41.207062 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34ffae54_fbc4_44f5_b785_a230835742c2.slice/crio-2e953b6a75acf4f5780f4f6f578ec108ba07a1cddee158fa836f342bae2e7fce WatchSource:0}: Error finding container 2e953b6a75acf4f5780f4f6f578ec108ba07a1cddee158fa836f342bae2e7fce: Status 404 returned error can't find the container with id 2e953b6a75acf4f5780f4f6f578ec108ba07a1cddee158fa836f342bae2e7fce Nov 01 15:47:41.406502 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:41.406430 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ckz9h" event={"ID":"c7838987-12dc-47bc-9b39-a87ffd11ddaf","Type":"ContainerStarted","Data":"7be9b9bfa2d4719d2e9b46c2a57a91f446ec1ebdcfaba01f41170b83d76efd60"} Nov 01 15:47:41.407694 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:41.407674 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cx4d8" event={"ID":"12f347b0-ed8e-409d-bd18-261844bfeeed","Type":"ContainerStarted","Data":"a9677234a195d43fd94578b1fbc59beea601e64e2fb660e308314e87433dd19e"} Nov 01 15:47:41.408738 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:41.408717 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-l26rw" event={"ID":"34ffae54-fbc4-44f5-b785-a230835742c2","Type":"ContainerStarted","Data":"2e953b6a75acf4f5780f4f6f578ec108ba07a1cddee158fa836f342bae2e7fce"} Nov 01 15:47:42.474185 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:42.474111 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7slg_f13a984b-8706-43d5-859f-abbfcaf58abf/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:42.638175 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:42.638099 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7slg_f13a984b-8706-43d5-859f-abbfcaf58abf/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:42.760236 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:42.760153 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7slg_f13a984b-8706-43d5-859f-abbfcaf58abf/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:43.036647 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:43.036615 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7slg_f13a984b-8706-43d5-859f-abbfcaf58abf/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:43.235111 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:43.234959 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d7slg_f13a984b-8706-43d5-859f-abbfcaf58abf/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:43.417034 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:43.416934 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cx4d8" event={"ID":"12f347b0-ed8e-409d-bd18-261844bfeeed","Type":"ContainerStarted","Data":"9bf3ace6022874e0c981fe36998413c80086e3dd262e0f401f321361657290d0"} Nov 01 15:47:43.417034 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:43.416986 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cx4d8" event={"ID":"12f347b0-ed8e-409d-bd18-261844bfeeed","Type":"ContainerStarted","Data":"1892036430f55ca07bfb804a1c4a2f424f10948d9847c0aae32b2cf8de40d2de"} Nov 01 15:47:43.434671 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:43.434620 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-cx4d8" podStartSLOduration=33.666739215 podStartE2EDuration="35.434601615s" totalImagesPullingTime="1.7678624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:47:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:43.433198638 +0000 UTC m=+35.755364561" watchObservedRunningTime="2026-11-01 15:47:43.434601615 +0000 UTC m=+35.756767527" Nov 01 15:47:45.221287 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:45.221207 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hkkmh_667a7db5-cf73-424c-9d18-53bfbc5f65db/dns-node-resolver/0.log" Nov 01 15:47:45.423718 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:45.423682 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-l26rw" event={"ID":"34ffae54-fbc4-44f5-b785-a230835742c2","Type":"ContainerStarted","Data":"8ff18394804fcfcdae47c75ba3ba078c3b954f723b194150ccca30ed824944f0"} Nov 01 15:47:45.423904 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:45.423786 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-l26rw" Nov 01 15:47:46.020805 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:46.020777 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-44klx_56e8fbab-06d3-4421-9995-822247d94d90/node-ca/0.log" Nov 01 15:47:46.427969 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:46.427935 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ckz9h" event={"ID":"c7838987-12dc-47bc-9b39-a87ffd11ddaf","Type":"ContainerStarted","Data":"4be82dd6d59f70fd640ec5fe023b8eddbf9f4a6599698098276718fed49abe23"} Nov 01 15:47:46.600609 ip-10-0-128-238 kubenswrapper[2576]: I1101 15:47:46.600528 2576 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 15:47:46.600691 ip-10-0-128-238 systemd[1]: Stopping Kubernetes Kubelet... Nov 01 15:47:46.620492 ip-10-0-128-238 systemd[1]: kubelet.service: Deactivated successfully. Nov 01 15:47:46.620669 ip-10-0-128-238 systemd[1]: Stopped Kubernetes Kubelet. Nov 01 15:47:46.620720 ip-10-0-128-238 systemd[1]: kubelet.service: Consumed 2.100s CPU time. Nov 01 15:47:46.637634 ip-10-0-128-238 systemd[1]: Starting Kubernetes Kubelet... Nov 01 15:47:46.720813 ip-10-0-128-238 kubenswrapper[4363]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 15:47:46.720813 ip-10-0-128-238 kubenswrapper[4363]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 15:47:46.720813 ip-10-0-128-238 kubenswrapper[4363]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 15:47:46.720813 ip-10-0-128-238 kubenswrapper[4363]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 15:47:46.723048 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.722948 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:47:46.723048 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723048 4363 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:47:46.723048 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723052 4363 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:47:46.723150 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723057 4363 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:47:46.723150 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723060 4363 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:47:46.723150 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723063 4363 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:47:46.723150 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723066 4363 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:47:46.723150 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723069 4363 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:47:46.723150 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723072 4363 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:47:46.723150 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723075 4363 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:47:46.723150 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723077 4363 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:47:46.723150 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723080 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:47:46.723150 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723083 4363 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:47:46.723150 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723086 4363 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:47:46.723150 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723089 4363 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:47:46.723150 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723092 4363 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:47:46.723150 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723094 4363 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:47:46.723150 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723097 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:47:46.723150 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723100 4363 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:47:46.723150 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723102 4363 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:47:46.723150 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723105 4363 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:47:46.723150 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723107 4363 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:47:46.723150 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723110 4363 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:47:46.723844 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723112 4363 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:47:46.723844 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723115 4363 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:47:46.723844 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723118 4363 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:47:46.723844 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723120 4363 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:47:46.723844 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723123 4363 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:47:46.723844 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723125 4363 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:47:46.723844 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723128 4363 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:47:46.723844 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723131 4363 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:47:46.723844 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723133 4363 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:47:46.723844 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723136 4363 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:47:46.723844 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723139 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:47:46.723844 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723150 4363 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:47:46.723844 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723153 4363 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:47:46.723844 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723156 4363 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:47:46.723844 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723159 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:47:46.723844 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723162 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:47:46.723844 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723165 4363 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:47:46.723844 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723168 4363 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:47:46.723844 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723170 4363 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:47:46.723844 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723172 4363 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:47:46.724519 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723175 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:47:46.724519 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723178 4363 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:47:46.724519 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723180 4363 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:47:46.724519 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723183 4363 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:47:46.724519 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723185 4363 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:47:46.724519 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723188 4363 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:47:46.724519 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723190 4363 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:47:46.724519 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723193 4363 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:47:46.724519 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723195 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:47:46.724519 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723198 4363 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:47:46.724519 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723201 4363 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:47:46.724519 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723204 4363 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:47:46.724519 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723206 4363 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:47:46.724519 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723209 4363 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:47:46.724519 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723212 4363 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:47:46.724519 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723214 4363 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:47:46.724519 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723218 4363 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:47:46.724519 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723221 4363 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:47:46.724519 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723223 4363 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:47:46.724519 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723226 4363 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:47:46.725088 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723228 4363 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:47:46.725088 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723232 4363 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:47:46.725088 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723234 4363 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:47:46.725088 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723237 4363 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:47:46.725088 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723279 4363 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:47:46.725088 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723455 4363 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:47:46.725088 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723462 4363 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:47:46.725088 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723467 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:47:46.725088 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723472 4363 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:47:46.725088 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723476 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:47:46.725088 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723481 4363 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:47:46.725088 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723486 4363 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:47:46.725088 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723490 4363 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:47:46.725088 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723494 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:47:46.725088 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723499 4363 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:47:46.725088 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723503 4363 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:47:46.725088 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723508 4363 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:47:46.725088 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723519 4363 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:47:46.725088 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723523 4363 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:47:46.725549 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723527 4363 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:47:46.725549 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723532 4363 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:47:46.725549 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723539 4363 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:47:46.725549 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723545 4363 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:47:46.725549 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723550 4363 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:47:46.725549 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723555 4363 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:47:46.725549 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723559 4363 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:47:46.725549 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723563 4363 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:47:46.725549 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723568 4363 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:47:46.725549 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723573 4363 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:47:46.725549 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723583 4363 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:47:46.725549 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723587 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:47:46.725549 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723591 4363 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:47:46.725549 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723596 4363 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:47:46.725549 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723600 4363 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:47:46.725549 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723605 4363 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:47:46.725549 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723609 4363 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:47:46.725549 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723613 4363 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:47:46.725549 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723692 4363 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:47:46.726005 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723697 4363 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:47:46.726005 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723701 4363 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:47:46.726005 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723705 4363 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:47:46.726005 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723710 4363 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:47:46.726005 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.723724 4363 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:47:46.726005 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725182 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:47:46.726005 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725195 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:47:46.726005 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725198 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:47:46.726005 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725201 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:47:46.726005 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725205 4363 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:47:46.726005 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725208 4363 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:47:46.726005 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725212 4363 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:47:46.726005 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725217 4363 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:47:46.726005 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725220 4363 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:47:46.726005 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725223 4363 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:47:46.726005 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725227 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:47:46.726005 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725230 4363 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:47:46.726005 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725233 4363 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:47:46.726005 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725235 4363 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:47:46.726497 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725238 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:47:46.726497 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725241 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:47:46.726497 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725244 4363 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:47:46.726497 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725246 4363 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:47:46.726497 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725249 4363 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:47:46.726497 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725252 4363 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:47:46.726497 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725254 4363 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:47:46.726497 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725257 4363 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:47:46.726497 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725260 4363 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:47:46.726497 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725262 4363 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:47:46.726497 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725265 4363 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:47:46.726497 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725267 4363 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:47:46.726497 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725270 4363 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:47:46.726497 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725272 4363 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:47:46.726497 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725275 4363 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:47:46.726497 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725278 4363 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:47:46.726497 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725281 4363 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:47:46.726497 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725284 4363 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:47:46.726497 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725286 4363 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:47:46.726972 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725289 4363 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:47:46.726972 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725292 4363 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:47:46.726972 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725294 4363 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:47:46.726972 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725297 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:47:46.726972 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725299 4363 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:47:46.726972 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725302 4363 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:47:46.726972 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725304 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:47:46.726972 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725307 4363 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:47:46.726972 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725310 4363 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:47:46.726972 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725313 4363 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:47:46.726972 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725315 4363 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:47:46.726972 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725318 4363 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:47:46.726972 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725320 4363 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:47:46.726972 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725322 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:47:46.726972 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725325 4363 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:47:46.726972 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725327 4363 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:47:46.726972 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725330 4363 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:47:46.726972 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725333 4363 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:47:46.726972 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725336 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:47:46.727459 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725339 4363 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:47:46.727459 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725342 4363 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:47:46.727459 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725344 4363 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:47:46.727459 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725347 4363 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:47:46.727459 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725349 4363 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:47:46.727459 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725352 4363 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:47:46.727459 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725355 4363 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:47:46.727459 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725358 4363 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:47:46.727459 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725360 4363 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:47:46.727459 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725363 4363 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:47:46.727459 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725365 4363 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:47:46.727459 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725368 4363 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:47:46.727459 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725370 4363 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:47:46.727459 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725373 4363 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:47:46.727459 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725375 4363 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:47:46.727459 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725378 4363 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:47:46.727459 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725380 4363 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:47:46.727459 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725383 4363 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:47:46.727459 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725385 4363 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:47:46.727459 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725388 4363 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:47:46.727958 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725390 4363 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:47:46.727958 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725393 4363 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:47:46.727958 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725395 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:47:46.727958 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725397 4363 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:47:46.727958 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725400 4363 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:47:46.727958 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725402 4363 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:47:46.727958 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725405 4363 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:47:46.727958 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725408 4363 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:47:46.727958 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725410 4363 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:47:46.727958 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725413 4363 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:47:46.727958 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725415 4363 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:47:46.727958 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725418 4363 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:47:46.727958 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725421 4363 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:47:46.727958 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725423 4363 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:47:46.727958 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725426 4363 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:47:46.727958 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725428 4363 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:47:46.727958 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725431 4363 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:47:46.727958 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725434 4363 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:47:46.727958 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725436 4363 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:47:46.727958 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725439 4363 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:47:46.728462 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725443 4363 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:47:46.728462 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725446 4363 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:47:46.728462 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725449 4363 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:47:46.728462 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725452 4363 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:47:46.728462 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725455 4363 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:47:46.728462 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725457 4363 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:47:46.728462 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725461 4363 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:47:46.728462 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725463 4363 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:47:46.728462 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725466 4363 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:47:46.728462 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725468 4363 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:47:46.728462 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725471 4363 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:47:46.728462 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725473 4363 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:47:46.728462 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725476 4363 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:47:46.728462 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.725479 4363 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:47:46.728462 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726004 4363 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:47:46.728462 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726010 4363 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:47:46.728462 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726034 4363 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:47:46.728462 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726038 4363 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:47:46.728462 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726041 4363 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:47:46.728933 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726043 4363 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:47:46.728933 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726047 4363 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:47:46.728933 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726050 4363 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:47:46.728933 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726052 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:47:46.728933 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726056 4363 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:47:46.728933 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726059 4363 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:47:46.728933 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726061 4363 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:47:46.728933 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726064 4363 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:47:46.728933 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726067 4363 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:47:46.728933 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726070 4363 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:47:46.728933 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726073 4363 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:47:46.728933 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726076 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:47:46.728933 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726078 4363 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:47:46.728933 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726081 4363 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:47:46.728933 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726083 4363 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:47:46.728933 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726086 4363 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:47:46.728933 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726089 4363 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:47:46.728933 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726093 4363 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:47:46.728933 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726097 4363 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:47:46.729398 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726100 4363 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:47:46.729398 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726103 4363 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:47:46.729398 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726106 4363 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:47:46.729398 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726109 4363 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:47:46.729398 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726112 4363 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:47:46.729398 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726115 4363 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:47:46.729398 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726117 4363 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:47:46.729398 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726120 4363 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:47:46.729398 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726123 4363 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:47:46.729398 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726127 4363 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:47:46.729398 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726131 4363 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:47:46.729398 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726134 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:47:46.729398 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726137 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:47:46.729398 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726140 4363 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:47:46.729398 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726142 4363 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:47:46.729398 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726145 4363 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:47:46.729398 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726147 4363 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:47:46.729398 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726151 4363 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:47:46.729398 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726153 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:47:46.729398 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726156 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:47:46.729883 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726159 4363 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:47:46.729883 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726162 4363 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:47:46.729883 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726165 4363 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:47:46.729883 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726167 4363 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:47:46.729883 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726170 4363 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:47:46.729883 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726172 4363 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:47:46.729883 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726175 4363 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:47:46.729883 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726177 4363 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:47:46.729883 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726180 4363 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:47:46.729883 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726183 4363 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:47:46.729883 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726185 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:47:46.729883 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726188 4363 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:47:46.729883 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726190 4363 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:47:46.729883 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726193 4363 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:47:46.729883 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726195 4363 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:47:46.729883 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726198 4363 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:47:46.729883 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726201 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:47:46.729883 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726204 4363 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:47:46.729883 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726207 4363 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:47:46.729883 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726209 4363 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:47:46.730464 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726212 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:47:46.730464 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726215 4363 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:47:46.730464 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726217 4363 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:47:46.730464 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726220 4363 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:47:46.730464 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726222 4363 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:47:46.730464 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726225 4363 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:47:46.730464 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726228 4363 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:47:46.730464 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726231 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:47:46.730464 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726233 4363 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:47:46.730464 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726236 4363 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:47:46.730464 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726238 4363 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:47:46.730464 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726241 4363 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:47:46.730464 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726244 4363 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:47:46.730464 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726246 4363 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:47:46.730464 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726249 4363 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:47:46.730464 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726251 4363 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:47:46.730464 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726254 4363 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:47:46.730464 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726257 4363 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:47:46.730464 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726260 4363 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:47:46.730923 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726262 4363 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:47:46.730923 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726264 4363 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:47:46.730923 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726267 4363 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:47:46.730923 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726269 4363 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:47:46.730923 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726272 4363 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:47:46.730923 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726274 4363 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:47:46.730923 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726277 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:47:46.730923 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726280 4363 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:47:46.730923 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726285 4363 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:47:46.730923 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726288 4363 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:47:46.730923 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726291 4363 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:47:46.730923 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726293 4363 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:47:46.730923 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726296 4363 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:47:46.730923 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726299 4363 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:47:46.730923 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726301 4363 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:47:46.730923 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726304 4363 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:47:46.730923 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726306 4363 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:47:46.730923 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726309 4363 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:47:46.730923 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726311 4363 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:47:46.731397 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726314 4363 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:47:46.731397 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726317 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:47:46.731397 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726319 4363 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:47:46.731397 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.726322 4363 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:47:46.731397 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726427 4363 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 15:47:46.731397 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726446 4363 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 15:47:46.731397 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726454 4363 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 15:47:46.731397 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726459 4363 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 15:47:46.731397 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726463 4363 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 15:47:46.731397 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726466 4363 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 15:47:46.731397 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726471 4363 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 15:47:46.731397 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726475 4363 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 15:47:46.731397 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726479 4363 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 15:47:46.731397 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726482 4363 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 15:47:46.731397 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726485 4363 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 15:47:46.731397 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726489 4363 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 15:47:46.731397 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726492 4363 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 15:47:46.731397 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726495 4363 flags.go:64] FLAG: --cgroup-root="" Nov 01 15:47:46.731397 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726499 4363 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 15:47:46.731397 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726503 4363 flags.go:64] FLAG: --client-ca-file="" Nov 01 15:47:46.731397 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726506 4363 flags.go:64] FLAG: --cloud-provider="external" Nov 01 15:47:46.731397 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726510 4363 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 15:47:46.731397 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726514 4363 flags.go:64] FLAG: --cluster-domain="" Nov 01 15:47:46.731397 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726517 4363 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 15:47:46.731968 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726520 4363 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 15:47:46.731968 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726523 4363 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 15:47:46.731968 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726527 4363 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 15:47:46.731968 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726531 4363 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 15:47:46.731968 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726534 4363 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 15:47:46.731968 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726537 4363 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 15:47:46.731968 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726540 4363 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 15:47:46.731968 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726543 4363 flags.go:64] FLAG: --contention-profiling="false" Nov 01 15:47:46.731968 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726546 4363 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 15:47:46.731968 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726550 4363 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 15:47:46.731968 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726554 4363 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 15:47:46.731968 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726557 4363 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 15:47:46.731968 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726561 4363 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 15:47:46.731968 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726564 4363 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 15:47:46.731968 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726567 4363 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 15:47:46.731968 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726570 4363 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 15:47:46.731968 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726573 4363 flags.go:64] FLAG: --enable-server="true" Nov 01 15:47:46.731968 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726576 4363 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 15:47:46.731968 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726583 4363 flags.go:64] FLAG: --event-burst="100" Nov 01 15:47:46.731968 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726586 4363 flags.go:64] FLAG: --event-qps="50" Nov 01 15:47:46.731968 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726589 4363 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 15:47:46.731968 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726609 4363 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 15:47:46.731968 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726614 4363 flags.go:64] FLAG: --eviction-hard="" Nov 01 15:47:46.731968 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726618 4363 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 15:47:46.732722 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726624 4363 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 15:47:46.732722 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726629 4363 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 15:47:46.732722 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726632 4363 flags.go:64] FLAG: --eviction-soft="" Nov 01 15:47:46.732722 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726635 4363 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 15:47:46.732722 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726638 4363 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 15:47:46.732722 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726642 4363 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 15:47:46.732722 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726645 4363 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 15:47:46.732722 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726647 4363 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 15:47:46.732722 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726650 4363 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 15:47:46.732722 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726653 4363 flags.go:64] FLAG: --feature-gates="" Nov 01 15:47:46.732722 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726658 4363 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 15:47:46.732722 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726661 4363 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 15:47:46.732722 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726664 4363 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 15:47:46.732722 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726667 4363 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 15:47:46.732722 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726670 4363 flags.go:64] FLAG: --healthz-port="10248" Nov 01 15:47:46.732722 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726673 4363 flags.go:64] FLAG: --help="false" Nov 01 15:47:46.732722 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726676 4363 flags.go:64] FLAG: --hostname-override="ip-10-0-128-238.us-east-2.compute.internal" Nov 01 15:47:46.732722 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726680 4363 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 15:47:46.732722 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726683 4363 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 15:47:46.732722 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726687 4363 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 15:47:46.732722 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726690 4363 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 15:47:46.732722 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726693 4363 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 15:47:46.732722 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726696 4363 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 15:47:46.733340 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726699 4363 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 15:47:46.733340 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726702 4363 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 15:47:46.733340 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726705 4363 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 15:47:46.733340 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726708 4363 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 15:47:46.733340 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726711 4363 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 15:47:46.733340 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726714 4363 flags.go:64] FLAG: --kube-reserved="" Nov 01 15:47:46.733340 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726717 4363 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 15:47:46.733340 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726720 4363 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 15:47:46.733340 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726723 4363 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 15:47:46.733340 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726728 4363 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 15:47:46.733340 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726731 4363 flags.go:64] FLAG: --lock-file="" Nov 01 15:47:46.733340 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726734 4363 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 15:47:46.733340 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726737 4363 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 15:47:46.733340 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726740 4363 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 15:47:46.733340 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726746 4363 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 15:47:46.733340 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726749 4363 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 15:47:46.733340 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726752 4363 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 15:47:46.733340 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726755 4363 flags.go:64] FLAG: --logging-format="text" Nov 01 15:47:46.733340 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726758 4363 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 15:47:46.733340 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726761 4363 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 15:47:46.733340 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726764 4363 flags.go:64] FLAG: --manifest-url="" Nov 01 15:47:46.733340 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726767 4363 flags.go:64] FLAG: --manifest-url-header="" Nov 01 15:47:46.733340 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726772 4363 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 15:47:46.733340 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726775 4363 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 15:47:46.733340 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726783 4363 flags.go:64] FLAG: --max-pods="110" Nov 01 15:47:46.733934 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726786 4363 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 15:47:46.733934 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726789 4363 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 15:47:46.733934 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726793 4363 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 15:47:46.733934 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726796 4363 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 15:47:46.733934 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726799 4363 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 15:47:46.733934 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726802 4363 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 15:47:46.733934 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726805 4363 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 15:47:46.733934 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726814 4363 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 15:47:46.733934 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726816 4363 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 15:47:46.733934 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726820 4363 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 15:47:46.733934 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726823 4363 flags.go:64] FLAG: --pod-cidr="" Nov 01 15:47:46.733934 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726826 4363 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 15:47:46.733934 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726829 4363 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 15:47:46.733934 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726832 4363 flags.go:64] FLAG: --pods-per-core="0" Nov 01 15:47:46.733934 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726835 4363 flags.go:64] FLAG: --port="10250" Nov 01 15:47:46.733934 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726838 4363 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 15:47:46.733934 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726842 4363 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-07a90eca50fbde4de" Nov 01 15:47:46.733934 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726845 4363 flags.go:64] FLAG: --qos-reserved="" Nov 01 15:47:46.733934 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726848 4363 flags.go:64] FLAG: --read-only-port="10255" Nov 01 15:47:46.733934 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726851 4363 flags.go:64] FLAG: --register-node="true" Nov 01 15:47:46.733934 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726854 4363 flags.go:64] FLAG: --register-with-taints="" Nov 01 15:47:46.733934 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726858 4363 flags.go:64] FLAG: --registry-burst="10" Nov 01 15:47:46.733934 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726861 4363 flags.go:64] FLAG: --registry-qps="5" Nov 01 15:47:46.733934 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726863 4363 flags.go:64] FLAG: --reserved-cpus="" Nov 01 15:47:46.733934 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726866 4363 flags.go:64] FLAG: --reserved-memory="" Nov 01 15:47:46.733934 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726870 4363 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 15:47:46.734674 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726873 4363 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 15:47:46.734674 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726876 4363 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 15:47:46.734674 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726879 4363 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 15:47:46.734674 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726882 4363 flags.go:64] FLAG: --runonce="false" Nov 01 15:47:46.734674 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726885 4363 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 15:47:46.734674 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726888 4363 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 15:47:46.734674 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726891 4363 flags.go:64] FLAG: --seccomp-default="false" Nov 01 15:47:46.734674 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726894 4363 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 15:47:46.734674 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726897 4363 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 15:47:46.734674 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726900 4363 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 15:47:46.734674 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726904 4363 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 15:47:46.734674 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726907 4363 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 15:47:46.734674 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726912 4363 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 15:47:46.734674 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726915 4363 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 15:47:46.734674 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726918 4363 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 15:47:46.734674 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726921 4363 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 15:47:46.734674 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726924 4363 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 15:47:46.734674 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726927 4363 flags.go:64] FLAG: --system-cgroups="" Nov 01 15:47:46.734674 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726930 4363 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 15:47:46.734674 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726935 4363 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 15:47:46.734674 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726938 4363 flags.go:64] FLAG: --tls-cert-file="" Nov 01 15:47:46.734674 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726941 4363 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 15:47:46.734674 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726947 4363 flags.go:64] FLAG: --tls-min-version="" Nov 01 15:47:46.734674 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726950 4363 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 15:47:46.734674 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726952 4363 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 15:47:46.735329 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726955 4363 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 15:47:46.735329 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726958 4363 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 15:47:46.735329 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726962 4363 flags.go:64] FLAG: --v="2" Nov 01 15:47:46.735329 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726966 4363 flags.go:64] FLAG: --version="false" Nov 01 15:47:46.735329 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726970 4363 flags.go:64] FLAG: --vmodule="" Nov 01 15:47:46.735329 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726974 4363 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 15:47:46.735329 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.726977 4363 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 15:47:46.735329 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727130 4363 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:47:46.735329 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727134 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:47:46.735329 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727137 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:47:46.735329 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727140 4363 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:47:46.735329 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727143 4363 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:47:46.735329 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727146 4363 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:47:46.735329 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727149 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:47:46.735329 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727152 4363 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:47:46.735329 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727154 4363 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:47:46.735329 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727157 4363 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:47:46.735329 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727160 4363 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:47:46.735329 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727163 4363 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:47:46.735329 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727167 4363 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:47:46.735329 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727170 4363 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:47:46.735329 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727173 4363 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:47:46.735863 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727175 4363 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:47:46.735863 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727178 4363 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:47:46.735863 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727180 4363 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:47:46.735863 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727183 4363 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:47:46.735863 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727186 4363 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:47:46.735863 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727188 4363 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:47:46.735863 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727191 4363 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:47:46.735863 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727194 4363 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:47:46.735863 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727196 4363 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:47:46.735863 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727199 4363 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:47:46.735863 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727202 4363 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:47:46.735863 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727204 4363 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:47:46.735863 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727217 4363 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:47:46.735863 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727220 4363 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:47:46.735863 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727223 4363 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:47:46.735863 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727226 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:47:46.735863 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727228 4363 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:47:46.735863 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727231 4363 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:47:46.735863 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727233 4363 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:47:46.746603 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727236 4363 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:47:46.746603 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727239 4363 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:47:46.746603 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727241 4363 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:47:46.746603 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727244 4363 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:47:46.746603 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727246 4363 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:47:46.746603 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727249 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:47:46.746603 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727251 4363 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:47:46.746603 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727254 4363 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:47:46.746603 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727256 4363 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:47:46.746603 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727259 4363 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:47:46.746603 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727265 4363 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:47:46.746603 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727270 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:47:46.746603 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727274 4363 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:47:46.746603 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727276 4363 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:47:46.746603 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727279 4363 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:47:46.746603 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727282 4363 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:47:46.746603 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727285 4363 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:47:46.746603 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727288 4363 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:47:46.746603 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727291 4363 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:47:46.747116 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727293 4363 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:47:46.747116 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727295 4363 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:47:46.747116 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727298 4363 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:47:46.747116 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727301 4363 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:47:46.747116 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727304 4363 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:47:46.747116 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727307 4363 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:47:46.747116 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727309 4363 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:47:46.747116 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727312 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:47:46.747116 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727315 4363 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:47:46.747116 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727317 4363 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:47:46.747116 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727320 4363 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:47:46.747116 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727323 4363 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:47:46.747116 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727327 4363 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:47:46.747116 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727329 4363 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:47:46.747116 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727332 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:47:46.747116 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727334 4363 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:47:46.747116 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727337 4363 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:47:46.747116 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727339 4363 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:47:46.747116 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727342 4363 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:47:46.747582 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727344 4363 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:47:46.747582 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727347 4363 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:47:46.747582 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727349 4363 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:47:46.747582 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727352 4363 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:47:46.747582 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727356 4363 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:47:46.747582 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727359 4363 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:47:46.747582 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727361 4363 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:47:46.747582 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727364 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:47:46.747582 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727366 4363 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:47:46.747582 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727369 4363 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:47:46.747582 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727372 4363 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:47:46.747582 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727375 4363 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:47:46.747582 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727377 4363 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:47:46.747582 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727380 4363 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:47:46.747582 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727382 4363 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:47:46.747582 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727385 4363 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:47:46.747582 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727387 4363 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:47:46.747582 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727390 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:47:46.747582 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727392 4363 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:47:46.747582 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727395 4363 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:47:46.748135 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727397 4363 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:47:46.748135 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727400 4363 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:47:46.748135 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727402 4363 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:47:46.748135 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727405 4363 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:47:46.748135 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727407 4363 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:47:46.748135 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727410 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:47:46.748135 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727412 4363 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:47:46.748135 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727415 4363 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:47:46.748135 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727417 4363 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:47:46.748135 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727420 4363 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:47:46.748135 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727423 4363 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:47:46.748135 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727425 4363 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:47:46.748135 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727428 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:47:46.748135 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727430 4363 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:47:46.748135 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.727458 4363 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 15:47:46.748135 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727499 4363 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:47:46.748525 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727505 4363 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:47:46.748525 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727508 4363 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:47:46.748525 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727512 4363 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:47:46.748525 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727515 4363 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:47:46.748525 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727517 4363 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:47:46.748525 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727520 4363 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:47:46.748525 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727522 4363 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:47:46.748525 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727526 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:47:46.748525 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727529 4363 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:47:46.748525 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727531 4363 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:47:46.748525 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727534 4363 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:47:46.748525 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727537 4363 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:47:46.748525 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727539 4363 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:47:46.748525 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727542 4363 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:47:46.748525 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727544 4363 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:47:46.748525 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727547 4363 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:47:46.748525 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727549 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:47:46.748525 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727552 4363 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:47:46.748525 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727554 4363 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:47:46.748985 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727557 4363 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:47:46.748985 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727560 4363 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:47:46.748985 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727564 4363 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:47:46.748985 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727567 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:47:46.748985 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727570 4363 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:47:46.748985 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727573 4363 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:47:46.748985 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727576 4363 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:47:46.748985 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727578 4363 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:47:46.748985 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727581 4363 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:47:46.748985 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727583 4363 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:47:46.748985 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727586 4363 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:47:46.748985 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727590 4363 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:47:46.748985 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727593 4363 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:47:46.748985 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727596 4363 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:47:46.748985 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727599 4363 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:47:46.748985 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727602 4363 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:47:46.748985 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727605 4363 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:47:46.748985 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727608 4363 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:47:46.748985 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727611 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:47:46.749470 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727613 4363 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:47:46.749470 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727616 4363 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:47:46.749470 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727618 4363 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:47:46.749470 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727622 4363 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:47:46.749470 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727625 4363 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:47:46.749470 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727628 4363 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:47:46.749470 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727631 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:47:46.749470 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727634 4363 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:47:46.749470 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727636 4363 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:47:46.749470 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727639 4363 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:47:46.749470 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727641 4363 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:47:46.749470 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727644 4363 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:47:46.749470 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727646 4363 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:47:46.749470 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727649 4363 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:47:46.749470 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727652 4363 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:47:46.749470 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727654 4363 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:47:46.749470 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727657 4363 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:47:46.749470 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727659 4363 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:47:46.749470 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727662 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:47:46.749470 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727664 4363 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:47:46.749940 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727667 4363 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:47:46.749940 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727670 4363 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:47:46.749940 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727673 4363 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:47:46.749940 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727675 4363 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:47:46.749940 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727680 4363 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:47:46.749940 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727683 4363 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:47:46.749940 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727686 4363 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:47:46.749940 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727689 4363 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:47:46.749940 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727691 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:47:46.749940 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727694 4363 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:47:46.749940 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727697 4363 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:47:46.749940 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727700 4363 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:47:46.749940 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727702 4363 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:47:46.749940 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727705 4363 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:47:46.749940 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727708 4363 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:47:46.749940 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727710 4363 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:47:46.749940 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727713 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:47:46.749940 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727716 4363 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:47:46.749940 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727718 4363 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:47:46.749940 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727721 4363 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:47:46.750477 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727723 4363 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:47:46.750477 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727726 4363 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:47:46.750477 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727729 4363 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:47:46.750477 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727731 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:47:46.750477 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727734 4363 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:47:46.750477 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727736 4363 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:47:46.750477 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727739 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:47:46.750477 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727742 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:47:46.750477 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727744 4363 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:47:46.750477 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727747 4363 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:47:46.750477 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727749 4363 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:47:46.750477 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727752 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:47:46.750477 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727755 4363 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:47:46.750477 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727758 4363 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:47:46.750477 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727760 4363 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:47:46.750477 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727763 4363 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:47:46.750477 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727767 4363 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:47:46.750477 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727769 4363 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:47:46.750477 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727773 4363 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:47:46.750477 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727775 4363 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:47:46.750963 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727778 4363 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:47:46.750963 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727781 4363 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:47:46.750963 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727783 4363 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:47:46.750963 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727786 4363 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:47:46.750963 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727789 4363 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:47:46.750963 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727791 4363 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:47:46.750963 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.727794 4363 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:47:46.750963 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.731560 4363 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 15:47:46.750963 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.731576 4363 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 15:47:46.750963 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.731594 4363 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 15:47:46.750963 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.731599 4363 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 15:47:46.750963 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731658 4363 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:47:46.750963 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731663 4363 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:47:46.750963 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731666 4363 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:47:46.750963 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731669 4363 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:47:46.750963 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731672 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:47:46.750963 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731674 4363 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:47:46.750963 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731677 4363 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:47:46.750963 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731680 4363 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:47:46.750963 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731682 4363 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:47:46.751548 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731685 4363 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:47:46.751548 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731688 4363 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:47:46.751548 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731691 4363 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:47:46.751548 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731693 4363 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:47:46.751548 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731696 4363 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:47:46.751548 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731699 4363 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:47:46.751548 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731701 4363 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:47:46.751548 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731704 4363 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:47:46.751548 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731707 4363 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:47:46.751548 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731709 4363 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:47:46.751548 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731712 4363 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:47:46.751548 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731714 4363 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:47:46.751548 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731717 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:47:46.751548 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731719 4363 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:47:46.751548 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731722 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:47:46.751548 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731724 4363 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:47:46.751548 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731727 4363 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:47:46.751548 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731730 4363 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:47:46.751548 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731732 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:47:46.751548 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731736 4363 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:47:46.752063 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731740 4363 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:47:46.752063 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731743 4363 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:47:46.752063 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731745 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:47:46.752063 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731748 4363 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:47:46.752063 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731751 4363 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:47:46.752063 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731753 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:47:46.752063 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731756 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:47:46.752063 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731758 4363 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:47:46.752063 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731761 4363 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:47:46.752063 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731763 4363 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:47:46.752063 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731766 4363 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:47:46.752063 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731768 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:47:46.752063 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731771 4363 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:47:46.752063 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731773 4363 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:47:46.752063 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731776 4363 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:47:46.752063 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731778 4363 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:47:46.752063 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731781 4363 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:47:46.752063 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731783 4363 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:47:46.752063 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731786 4363 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:47:46.752620 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731789 4363 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:47:46.752620 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731792 4363 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:47:46.752620 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731795 4363 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:47:46.752620 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731798 4363 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:47:46.752620 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731801 4363 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:47:46.752620 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731804 4363 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:47:46.752620 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731806 4363 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:47:46.752620 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731809 4363 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:47:46.752620 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731812 4363 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:47:46.752620 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731814 4363 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:47:46.752620 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731817 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:47:46.752620 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731820 4363 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:47:46.752620 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731822 4363 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:47:46.752620 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731824 4363 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:47:46.752620 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731828 4363 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:47:46.752620 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731830 4363 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:47:46.752620 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731834 4363 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:47:46.752620 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731838 4363 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:47:46.752620 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731841 4363 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:47:46.752620 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731844 4363 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:47:46.753197 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731846 4363 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:47:46.753197 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731849 4363 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:47:46.753197 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731851 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:47:46.753197 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731854 4363 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:47:46.753197 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731857 4363 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:47:46.753197 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731859 4363 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:47:46.753197 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731862 4363 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:47:46.753197 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731865 4363 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:47:46.753197 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731867 4363 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:47:46.753197 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731870 4363 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:47:46.753197 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731872 4363 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:47:46.753197 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731875 4363 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:47:46.753197 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731877 4363 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:47:46.753197 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731881 4363 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:47:46.753197 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731885 4363 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:47:46.753197 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731889 4363 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:47:46.753197 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731892 4363 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:47:46.753197 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731895 4363 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:47:46.753197 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731897 4363 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:47:46.753660 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731900 4363 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:47:46.753660 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731903 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:47:46.753660 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731905 4363 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:47:46.753660 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731908 4363 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:47:46.753660 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731911 4363 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:47:46.753660 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731913 4363 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:47:46.753660 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731915 4363 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:47:46.753660 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731918 4363 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:47:46.753660 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731921 4363 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:47:46.753660 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731924 4363 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:47:46.753660 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731926 4363 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:47:46.753660 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731929 4363 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:47:46.753660 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731931 4363 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:47:46.753660 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731934 4363 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:47:46.753660 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731936 4363 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:47:46.753660 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731939 4363 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:47:46.753660 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731941 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:47:46.753660 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731944 4363 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:47:46.753660 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.731946 4363 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:47:46.754139 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.731972 4363 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 15:47:46.754139 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732166 4363 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:47:46.754139 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732171 4363 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:47:46.754139 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732175 4363 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:47:46.754139 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732178 4363 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:47:46.754139 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732181 4363 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:47:46.754139 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732184 4363 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:47:46.754139 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732187 4363 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:47:46.754139 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732190 4363 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:47:46.754139 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732192 4363 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:47:46.754139 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732195 4363 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:47:46.754139 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732198 4363 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:47:46.754139 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732201 4363 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:47:46.754139 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732203 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:47:46.754139 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732206 4363 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:47:46.754139 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732209 4363 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:47:46.754543 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732211 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:47:46.754543 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732214 4363 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:47:46.754543 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732216 4363 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:47:46.754543 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732218 4363 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:47:46.754543 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732221 4363 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:47:46.754543 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732224 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:47:46.754543 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732233 4363 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:47:46.754543 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732236 4363 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:47:46.754543 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732238 4363 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:47:46.754543 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732241 4363 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:47:46.754543 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732244 4363 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:47:46.754543 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732246 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:47:46.754543 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732249 4363 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:47:46.754543 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732253 4363 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:47:46.754543 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732256 4363 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:47:46.754543 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732259 4363 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:47:46.754543 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732262 4363 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:47:46.754543 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732264 4363 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:47:46.754543 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732267 4363 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:47:46.755417 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732269 4363 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:47:46.755417 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732272 4363 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:47:46.755417 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732275 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:47:46.755417 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732277 4363 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:47:46.755417 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732282 4363 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:47:46.755417 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732285 4363 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:47:46.755417 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732287 4363 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:47:46.755417 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732290 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:47:46.755417 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732293 4363 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:47:46.755417 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732295 4363 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:47:46.755417 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732298 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:47:46.755417 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732301 4363 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:47:46.755417 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732304 4363 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:47:46.755417 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732306 4363 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:47:46.755417 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732309 4363 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:47:46.755417 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732311 4363 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:47:46.755417 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732314 4363 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:47:46.755417 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732316 4363 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:47:46.755417 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732319 4363 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:47:46.755908 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732321 4363 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:47:46.755908 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732329 4363 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:47:46.755908 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732332 4363 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:47:46.755908 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732335 4363 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:47:46.755908 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732338 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:47:46.755908 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732340 4363 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:47:46.755908 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732343 4363 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:47:46.755908 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732346 4363 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:47:46.755908 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732348 4363 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:47:46.755908 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732351 4363 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:47:46.755908 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732354 4363 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:47:46.755908 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732356 4363 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:47:46.755908 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732359 4363 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:47:46.755908 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732361 4363 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:47:46.755908 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732364 4363 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:47:46.755908 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732366 4363 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:47:46.755908 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732369 4363 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:47:46.755908 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732373 4363 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:47:46.755908 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732376 4363 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:47:46.755908 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732378 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:47:46.756445 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732381 4363 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:47:46.756445 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732384 4363 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:47:46.756445 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732386 4363 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:47:46.756445 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732389 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:47:46.756445 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732392 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:47:46.756445 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732394 4363 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:47:46.756445 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732397 4363 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:47:46.756445 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732399 4363 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:47:46.756445 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732402 4363 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:47:46.756445 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732405 4363 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:47:46.756445 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732407 4363 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:47:46.756445 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732410 4363 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:47:46.756445 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732412 4363 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:47:46.756445 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732415 4363 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:47:46.756445 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732423 4363 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:47:46.756445 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732426 4363 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:47:46.756445 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732428 4363 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:47:46.756445 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732431 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:47:46.756445 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732433 4363 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:47:46.756445 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732436 4363 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:47:46.756940 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732439 4363 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:47:46.756940 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732442 4363 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:47:46.756940 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732445 4363 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:47:46.756940 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732449 4363 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:47:46.756940 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732452 4363 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:47:46.756940 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732456 4363 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:47:46.756940 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732458 4363 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:47:46.756940 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732461 4363 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:47:46.756940 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732464 4363 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:47:46.756940 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732468 4363 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:47:46.756940 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732471 4363 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:47:46.756940 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732473 4363 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:47:46.756940 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732476 4363 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:47:46.756940 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.732500 4363 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 15:47:46.756940 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732541 4363 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:47:46.757357 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732545 4363 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:47:46.757357 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732548 4363 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:47:46.757357 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732551 4363 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:47:46.757357 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732554 4363 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:47:46.757357 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732557 4363 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:47:46.757357 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732559 4363 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:47:46.757357 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732562 4363 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:47:46.757357 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732565 4363 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:47:46.757357 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732574 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:47:46.757357 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732577 4363 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:47:46.757357 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732580 4363 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:47:46.757357 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732583 4363 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:47:46.757357 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732591 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:47:46.757357 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732594 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:47:46.757357 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732597 4363 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:47:46.757357 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732599 4363 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:47:46.757357 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732602 4363 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:47:46.757357 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732604 4363 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:47:46.757357 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732607 4363 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:47:46.757818 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732610 4363 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:47:46.757818 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732612 4363 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:47:46.757818 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732615 4363 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:47:46.757818 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732617 4363 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:47:46.757818 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732620 4363 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:47:46.757818 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732624 4363 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:47:46.757818 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732627 4363 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:47:46.757818 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732629 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:47:46.757818 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732634 4363 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:47:46.757818 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732637 4363 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:47:46.757818 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732640 4363 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:47:46.757818 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732643 4363 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:47:46.757818 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732647 4363 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:47:46.757818 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732650 4363 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:47:46.757818 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732653 4363 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:47:46.757818 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732656 4363 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:47:46.757818 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732659 4363 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:47:46.757818 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732663 4363 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:47:46.757818 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732665 4363 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:47:46.758460 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732669 4363 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:47:46.758460 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732673 4363 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:47:46.758460 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732677 4363 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:47:46.758460 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732681 4363 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:47:46.758460 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732685 4363 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:47:46.758460 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732688 4363 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:47:46.758460 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732692 4363 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:47:46.758460 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732706 4363 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:47:46.758460 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732709 4363 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:47:46.758460 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732712 4363 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:47:46.758460 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732714 4363 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:47:46.758460 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732717 4363 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:47:46.758460 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732719 4363 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:47:46.758460 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732722 4363 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:47:46.758460 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732726 4363 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:47:46.758460 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732729 4363 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:47:46.758460 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732731 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:47:46.758460 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732734 4363 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:47:46.758460 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732738 4363 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:47:46.758460 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732741 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:47:46.759420 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732743 4363 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:47:46.759420 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732763 4363 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:47:46.759420 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732768 4363 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:47:46.759420 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732771 4363 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:47:46.759420 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732774 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:47:46.759420 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732777 4363 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:47:46.759420 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732779 4363 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:47:46.759420 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732782 4363 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:47:46.759420 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732785 4363 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:47:46.759420 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732788 4363 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:47:46.759420 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732791 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:47:46.759420 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732793 4363 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:47:46.759420 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732797 4363 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:47:46.759420 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732801 4363 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:47:46.759420 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732804 4363 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:47:46.759420 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732807 4363 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:47:46.759420 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732809 4363 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:47:46.759420 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732812 4363 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:47:46.759420 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732815 4363 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:47:46.758649 ip-10-0-128-238 systemd[1]: Started Kubernetes Kubelet. Nov 01 15:47:46.760386 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732817 4363 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:47:46.760386 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732829 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:47:46.760386 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732832 4363 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:47:46.760386 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732834 4363 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:47:46.760386 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732837 4363 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:47:46.760386 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732840 4363 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:47:46.760386 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732842 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:47:46.760386 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732845 4363 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:47:46.760386 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732847 4363 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:47:46.760386 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732850 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:47:46.760386 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732853 4363 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:47:46.760386 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732857 4363 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:47:46.760386 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732859 4363 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:47:46.760386 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732862 4363 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:47:46.760386 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732872 4363 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:47:46.760386 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732875 4363 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:47:46.760386 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732877 4363 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:47:46.760386 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732880 4363 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:47:46.760386 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732882 4363 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:47:46.761219 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732885 4363 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:47:46.761219 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732889 4363 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:47:46.761219 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732891 4363 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:47:46.761219 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732894 4363 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:47:46.761219 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732896 4363 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:47:46.761219 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732899 4363 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:47:46.761219 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732902 4363 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:47:46.761219 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732904 4363 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:47:46.761219 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:46.732907 4363 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:47:46.761219 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.732968 4363 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 15:47:46.761219 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.734006 4363 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 01 15:47:46.761219 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.734084 4363 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 01 15:47:46.761219 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.734274 4363 server.go:1014] "Starting client certificate rotation" Nov 01 15:47:46.761219 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.734430 4363 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 15:47:46.761219 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.734524 4363 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 15:42:08 +0000 UTC" deadline="2028-03-31 17:53:45.296853916 +0000 UTC" Nov 01 15:47:46.761219 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.734584 4363 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12386h5m58.562272083s" Nov 01 15:47:46.761937 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.734856 4363 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 15:47:46.761937 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.736036 4363 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 15:47:46.761937 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.738494 4363 log.go:25] "Validated CRI v1 runtime API" Nov 01 15:47:46.761937 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.740160 4363 log.go:25] "Validated CRI v1 image API" Nov 01 15:47:46.761937 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.740935 4363 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 15:47:46.761937 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.743487 4363 fs.go:135] Filesystem UUIDs: map[49d632c8-b5c9-4292-98d3-a47e3e65bfa5:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9df08b24-6cad-4ca2-b0d4-7392efcf0dec:/dev/nvme0n1p4] Nov 01 15:47:46.762253 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.743509 4363 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1fd78a54e606c6f9d9e263ddb1349684db7653078d9f5b9524e2debf12cc08f3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1fd78a54e606c6f9d9e263ddb1349684db7653078d9f5b9524e2debf12cc08f3/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2e953b6a75acf4f5780f4f6f578ec108ba07a1cddee158fa836f342bae2e7fce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2e953b6a75acf4f5780f4f6f578ec108ba07a1cddee158fa836f342bae2e7fce/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/49011229e0e6d68ef5b69103031362491e74528b2942fd19f7e57d9897009e2a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/49011229e0e6d68ef5b69103031362491e74528b2942fd19f7e57d9897009e2a/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4baf3b81c3058ccf6560267398abfaea5af29aeb6a7b9295fee685f4e2bb4fc1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4baf3b81c3058ccf6560267398abfaea5af29aeb6a7b9295fee685f4e2bb4fc1/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/74534d21e666655f844f8a4dce38fb0a3ed04f94c7c214a87da6d6b336832651/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/74534d21e666655f844f8a4dce38fb0a3ed04f94c7c214a87da6d6b336832651/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7be9b9bfa2d4719d2e9b46c2a57a91f446ec1ebdcfaba01f41170b83d76efd60/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7be9b9bfa2d4719d2e9b46c2a57a91f446ec1ebdcfaba01f41170b83d76efd60/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/803e85f1b86f85efe617b68cd13e21b22870860367c89256d954973adff1927e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/803e85f1b86f85efe617b68cd13e21b22870860367c89256d954973adff1927e/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9b717001af95764d3d50972a296153b9f1b650251ab71ecb27e1fdab544fca0b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9b717001af95764d3d50972a296153b9f1b650251ab71ecb27e1fdab544fca0b/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9d5263a747b1d06072586979cb12c5bd28b85c8eb029fe696c87a956fbdf2787/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9d5263a747b1d06072586979cb12c5bd28b85c8eb029fe696c87a956fbdf2787/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a732a83c796496797985edc42d736f58652177e7d9e4f6e3177a4c1a9f1d84e7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a732a83c796496797985edc42d736f58652177e7d9e4f6e3177a4c1a9f1d84e7/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a9677234a195d43fd94578b1fbc59beea601e64e2fb660e308314e87433dd19e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a9677234a195d43fd94578b1fbc59beea601e64e2fb660e308314e87433dd19e/userdata/shm major:0 minor:201 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b8db1b3a385d3d6a0dbce8ac88f785a826315f7f35eb317d4177bafa842525a6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b8db1b3a385d3d6a0dbce8ac88f785a826315f7f35eb317d4177bafa842525a6/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eed1a6e1117b93c7d32aa445a6b08f47f555ec343f4d3007de78f66134168519/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eed1a6e1117b93c7d32aa445a6b08f47f555ec343f4d3007de78f66134168519/userdata/shm major:0 minor:64 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09f28a75-b66e-4933-84e2-eb0941e5f6a3/volumes/kubernetes.io~projected/kube-api-access-gdpt9:{mountpoint:/var/lib/kubelet/pods/09f28a75-b66e-4933-84e2-eb0941e5f6a3/volumes/kubernetes.io~projected/kube-api-access-gdpt9 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12f347b0-ed8e-409d-bd18-261844bfeeed/volumes/kubernetes.io~projected/kube-api-access-njf2l:{mountpoint:/var/lib/kubelet/pods/12f347b0-ed8e-409d-bd18-261844bfeeed/volumes/kubernetes.io~projected/kube-api-access-njf2l major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12f347b0-ed8e-409d-bd18-261844bfeeed/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/12f347b0-ed8e-409d-bd18-261844bfeeed/volumes/kubernetes.io~secret/metrics-certs major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34ffae54-fbc4-44f5-b785-a230835742c2/volumes/kubernetes.io~projected/kube-api-access-nlwtm:{mountpoint:/var/lib/kubelet/pods/34ffae54-fbc4-44f5-b785-a230835742c2/volumes/kubernetes.io~projected/kube-api-access-nlwtm major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b8758dd-149c-4c37-9d32-a93fc953e548/volumes/kubernetes.io~projected/kube-api-access-xl7rt:{mountpoint:/var/lib/kubelet/pods/4b8758dd-149c-4c37-9d32-a93fc953e548/volumes/kubernetes.io~projected/kube-api-access-xl7rt major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56e8fbab-06d3-4421-9995-822247d94d90/volumes/kubernetes.io~projected/kube-api-access-4kmfx:{mountpoint:/var/lib/kubelet/pods/56e8fbab-06d3-4421-9995-822247d94d90/volumes/kubernetes.io~projected/kube-api-access-4kmfx major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60e635cd-c1f3-4d34-b795-2f6b16438f8f/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/60e635cd-c1f3-4d34-b795-2f6b16438f8f/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60e635cd-c1f3-4d34-b795-2f6b16438f8f/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/60e635cd-c1f3-4d34-b795-2f6b16438f8f/volumes/kubernetes.io~empty-dir/tmp major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60e635cd-c1f3-4d34-b795-2f6b16438f8f/volumes/kubernetes.io~projected/kube-api-access-h9r89:{mountpoint:/var/lib/kubelet/pods/60e635cd-c1f3-4d34-b795-2f6b16438f8f/volumes/kubernetes.io~projected/kube-api-access-h9r89 major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/667a7db5-cf73-424c-9d18-53bfbc5f65db/volumes/kubernetes.io~projected/kube-api-access-t4s2s:{mountpoint:/var/lib/kubelet/pods/667a7db5-cf73-424c-9d18-53bfbc5f65db/volumes/kubernetes.io~projected/kube-api-access-t4s2s major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8254aafa-dd5b-4150-87e4-d165dab6bcb5/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/8254aafa-dd5b-4150-87e4-d165dab6bcb5/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8254aafa-dd5b-4150-87e4-d165dab6bcb5/volumes/kubernetes.io~projected/kube-api-access-glz5x:{mountpoint:/var/lib/kubelet/pods/8254aafa-dd5b-4150-87e4-d165dab6bcb5/volumes/kubernetes.io~projected/kube-api-access-glz5x major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8254aafa-dd5b-4150-87e4-d165dab6bcb5/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/8254aafa-dd5b-4150-87e4-d165dab6bcb5/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e87fe52-8f65-43d9-a04a-a86d6227230d/volumes/kubernetes.io~projected/kube-api-access-wkkf9:{mountpoint:/var/lib/kubelet/pods/8e87fe52-8f65-43d9-a04a-a86d6227230d/volumes/kubernetes.io~projected/kube-api-access-wkkf9 major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b53051b0-243d-4e03-926a-1f83372e648f/volumes/kubernetes.io~projected/kube-api-access-pjb4z:{mountpoint:/var/lib/kubelet/pods/b53051b0-243d-4e03-926a-1f83372e648f/volumes/kubernetes.io~projected/kube-api-access-pjb4z major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7838987-12dc-47bc-9b39-a87ffd11ddaf/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/c7838987-12dc-47bc-9b39-a87ffd11ddaf/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f13a984b-8706-43d5-859f-abbfcaf58abf/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/f13a984b-8706-43d5-859f-abbfcaf58abf/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/71178aca5b1e5700a59bda1ec9bea92df4369ef46f1e85dca168fe89f233510f/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/4fa0a50026b34b1d9832e59e9f371eb173a18abf510514fe5d7e4ea1b8e1f4e9/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/e8af335452d4129a6b7b60d9ed8a01510374b7ed29c95f43df14e7f1d825e24c/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/17dd0a463f3f3a44b1efbda3dce27e2dd175260cb94b485ed7876517d6e81bf8/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/1f47796452d48bae97638f2eba09efcdb234aedde9bdf04312d2264311356c20/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/386819acf135c39e8d0872a6b0568d77f880ec423914db81745bb17f26f3556f/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/b494f3cf09f602870e3fe670b84e87a1e6c58147e410fc4f2d7cda3769506534/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/c378b606fea07f994339975e8be65d1e3fed5788680554f11e70ef5fd126a07b/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/2bd28f0bee2c3ee18de02204d3a489e0257396630c8b7d21ac38de566610d4e3/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/d9d7668a957859ca449840564fbdeda58d6fa86dd60d3241d0d2ae317fee4d9d/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/a51c1cd139d264a1bb70b812b347beb55ba4a90987a4bb0f377065173a0d1dcc/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/f41ecbf261a7d6c9390f142475905a66676cf6b04d9fdca2978a92f0429fef6d/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/5e9a705f20fda8050f399e6a5f5d1b5f8872dc70ca269a5523be691f720f17e9/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/d1179315b06089da98f2d7b6e5a4388dc6d2db51cd16083ee12ccb3e1398afc6/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/ca8e0dee3eda4b65ea0a3d65e37ac78bb4cbef30dab1ad0750abc958200e8503/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/0b0ac7d310105e5f99035e12ec98c2a3194dd7f9f3653f2189a4b190beb13376/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/deaccdc75af629655ab749a6c51df08a9831b86f08bea255a361c949af4caf8e/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/cec1f8b09dd88308ebf16279062575b960a8f7d22308682604a283a44cd52edc/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/f1ccadf9c0afad6657afee01e0579d60e16ed396619ebbfd6118527907288061/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/32d1ec1b71115f6e90dea121a656ae5bdd3460d7b4f003161aa555910f1b7de6/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/0a8d8accb63d9895000830457bca932cf8fb890db6c9781392d1510e4c93fe4f/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-208:{mountpoint:/var/lib/containers/storage/overlay/f84a1bbd64c403fb231288a70792fd4979cd579109b2d1c2c5a25aa97ec28c12/merged major:0 minor:208 fsType:overlay blockSize:0} overlay_0-210:{mountpoint:/var/lib/containers/storage/overlay/4e12d12e1220fffb408b35219b13f4f95c3074e567343086a096df73d726b526/merged major:0 minor:210 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/ff19c35be8f6e0dd012cf1a55d8c841886fd1fed2385dda881e2d850a51f4897/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/0390ab71ac32a0829916c2b96cd41095b69dbd8e6a1bf295b0ae8190cbe1b841/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/f581588d826d8090a333c387e79a810c77c1733b2ea25927e2a7c26194421c83/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/f5a7bddf7d107866d18f09c0410a563b915899f2651dbcee6c10d178aa1aedae/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-227:{mountpoint:/var/lib/containers/storage/overlay/7706096fa11bf79e433a9e587130a505c0c6c519b9eaa4e06ab68cf96457f770/merged major:0 minor:227 fsType:overlay blockSize:0} overlay_0-62:{mountpoint:/var/lib/containers/storage/overlay/7712460183fdc9b67e56180379c25d57bf86c6d14c86c08a6ed3c4b2dcf9889a/merged major:0 minor:62 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/a1372265f7bac8177c38ba89fd1e61f0055f84436bf4eefe4cd56569ae6496ae/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/d551879c90fcaebe418f80c025926b9355a8c89eb9cc811c2a711ffc1e4ba3e9/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/4cf8bd9b6d09a8683bce47099ccfc53a8cf7c148cada25030c8c3d826726be42/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/99d037f17e2fb9d823a205752163d25743be972795536eddf51665496e84714f/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-81:{mountpoint:/var/lib/containers/storage/overlay/cc237d414a2bc0b5c5bd880d221dd976b2bd3e3f5d4c476c0746addfb5842b03/merged major:0 minor:81 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/3f3285119a093d2f63f20a4c6d8d341d905ffc42e33af6fe207334410b24dcae/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/4f88c7ea1169a771c9dca1eaca9b2c6f3985f6733e769f88f03ba6a92c90dbfb/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/218d75f260863573f27ed05fc441d5baecfe167fccfe69c3d824dce309be6509/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/2d10d69807aae9cc5f272cfa61be9d27a3fdb1978d054105001448ae72d019b0/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 01 15:47:46.762253 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.749823 4363 manager.go:217] Machine: {Timestamp:2026-11-01 15:47:46.749362241 +0000 UTC m=+0.081099892 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101681 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28d0b073fd7323338be4b86c915ee2 SystemUUID:ec28d0b0-73fd-7323-338b-e4b86c915ee2 BootID:35dfd4b2-c399-4bc2-8cda-065fcccc4abc Filesystems:[{Device:/run/containers/storage/overlay-containers/a9677234a195d43fd94578b1fbc59beea601e64e2fb660e308314e87433dd19e/userdata/shm DeviceMajor:0 DeviceMinor:201 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1fd78a54e606c6f9d9e263ddb1349684db7653078d9f5b9524e2debf12cc08f3/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/34ffae54-fbc4-44f5-b785-a230835742c2/volumes/kubernetes.io~projected/kube-api-access-nlwtm DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-208 DeviceMajor:0 DeviceMinor:208 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/12f347b0-ed8e-409d-bd18-261844bfeeed/volumes/kubernetes.io~projected/kube-api-access-njf2l DeviceMajor:0 DeviceMinor:53 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/09f28a75-b66e-4933-84e2-eb0941e5f6a3/volumes/kubernetes.io~projected/kube-api-access-gdpt9 DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/667a7db5-cf73-424c-9d18-53bfbc5f65db/volumes/kubernetes.io~projected/kube-api-access-t4s2s DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/803e85f1b86f85efe617b68cd13e21b22870860367c89256d954973adff1927e/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b53051b0-243d-4e03-926a-1f83372e648f/volumes/kubernetes.io~projected/kube-api-access-pjb4z DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a732a83c796496797985edc42d736f58652177e7d9e4f6e3177a4c1a9f1d84e7/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9d5263a747b1d06072586979cb12c5bd28b85c8eb029fe696c87a956fbdf2787/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/49011229e0e6d68ef5b69103031362491e74528b2942fd19f7e57d9897009e2a/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b8db1b3a385d3d6a0dbce8ac88f785a826315f7f35eb317d4177bafa842525a6/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8254aafa-dd5b-4150-87e4-d165dab6bcb5/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2e953b6a75acf4f5780f4f6f578ec108ba07a1cddee158fa836f342bae2e7fce/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/60e635cd-c1f3-4d34-b795-2f6b16438f8f/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:47 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8e87fe52-8f65-43d9-a04a-a86d6227230d/volumes/kubernetes.io~projected/kube-api-access-wkkf9 DeviceMajor:0 DeviceMinor:51 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/60e635cd-c1f3-4d34-b795-2f6b16438f8f/volumes/kubernetes.io~projected/kube-api-access-h9r89 DeviceMajor:0 DeviceMinor:52 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8254aafa-dd5b-4150-87e4-d165dab6bcb5/volumes/kubernetes.io~projected/kube-api-access-glz5x DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-210 DeviceMajor:0 DeviceMinor:210 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f13a984b-8706-43d5-859f-abbfcaf58abf/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/12f347b0-ed8e-409d-bd18-261844bfeeed/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-62 DeviceMajor:0 DeviceMinor:62 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c7838987-12dc-47bc-9b39-a87ffd11ddaf/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-227 DeviceMajor:0 DeviceMinor:227 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/60e635cd-c1f3-4d34-b795-2f6b16438f8f/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:48 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8254aafa-dd5b-4150-87e4-d165dab6bcb5/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:54 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/74534d21e666655f844f8a4dce38fb0a3ed04f94c7c214a87da6d6b336832651/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-81 DeviceMajor:0 DeviceMinor:81 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4b8758dd-149c-4c37-9d32-a93fc953e548/volumes/kubernetes.io~projected/kube-api-access-xl7rt DeviceMajor:0 DeviceMinor:49 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/56e8fbab-06d3-4421-9995-822247d94d90/volumes/kubernetes.io~projected/kube-api-access-4kmfx DeviceMajor:0 DeviceMinor:50 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eed1a6e1117b93c7d32aa445a6b08f47f555ec343f4d3007de78f66134168519/userdata/shm DeviceMajor:0 DeviceMinor:64 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9b717001af95764d3d50972a296153b9f1b650251ab71ecb27e1fdab544fca0b/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7be9b9bfa2d4719d2e9b46c2a57a91f446ec1ebdcfaba01f41170b83d76efd60/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4baf3b81c3058ccf6560267398abfaea5af29aeb6a7b9295fee685f4e2bb4fc1/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:2e953b6a75acf4f MacAddress:c6:a0:29:1a:0d:25 Speed:10000 Mtu:8901} {Name:7be9b9bfa2d4719 MacAddress:6a:71:46:3c:c9:4c Speed:10000 Mtu:8901} {Name:a9677234a195d43 MacAddress:2a:c1:54:55:22:0b Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:e4:d4:00:dd Speed:0 Mtu:9001} {Name:br-int MacAddress:72:c3:18:89:ab:f2 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:e4:d4:00:dd Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:2a:b7:e9:05:29:f6 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ca:22:96:65:d6:5c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 15:47:46.762253 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.750060 4363 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 15:47:46.762253 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.750104 4363 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 15:47:46.762253 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.750401 4363 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 15:47:46.762781 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.750418 4363 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-238.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 15:47:46.762781 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.750556 4363 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 15:47:46.762781 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.750564 4363 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 15:47:46.762781 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.750577 4363 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 15:47:46.762781 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.750590 4363 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 15:47:46.762781 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.750682 4363 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 15:47:46.762781 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.750833 4363 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 15:47:46.762781 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.750957 4363 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 15:47:46.762781 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.750994 4363 kubelet.go:493] "Attempting to sync node with API server" Nov 01 15:47:46.762781 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.751006 4363 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 15:47:46.762781 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.751036 4363 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 15:47:46.762781 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.751047 4363 kubelet.go:405] "Adding apiserver pod source" Nov 01 15:47:46.762781 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.751056 4363 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 15:47:46.762781 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.751870 4363 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 15:47:46.762781 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.751883 4363 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 15:47:46.762781 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.752368 4363 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 15:47:46.762781 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.754188 4363 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 15:47:46.762781 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.754197 4363 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 15:47:46.762781 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.754289 4363 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 01 15:47:46.762781 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.754475 4363 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 15:47:46.762781 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.754501 4363 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 15:47:46.762781 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.754588 4363 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 15:47:46.762781 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.754601 4363 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 15:47:46.762781 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.754606 4363 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 15:47:46.762781 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.754613 4363 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 15:47:46.762781 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.754620 4363 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 15:47:46.762781 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.754626 4363 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 15:47:46.762781 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.754633 4363 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 15:47:46.762781 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.754639 4363 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 15:47:46.762781 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.754646 4363 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 15:47:46.762781 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.754651 4363 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 15:47:46.762781 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.754668 4363 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 15:47:46.762781 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.754678 4363 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 15:47:46.762781 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.754704 4363 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 15:47:46.762781 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.754710 4363 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 15:47:46.762781 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.758121 4363 server.go:1263] "Started kubelet" Nov 01 15:47:46.762781 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.758200 4363 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 15:47:46.762781 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.758331 4363 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 15:47:46.762781 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.758392 4363 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 15:47:46.762781 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.762662 4363 server.go:317] "Adding debug handlers to kubelet server" Nov 01 15:47:46.764741 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.763633 4363 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 15:47:46.766191 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.766172 4363 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 15:47:46.766191 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.766189 4363 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 15:47:46.766340 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.766232 4363 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 15:42:09 +0000 UTC" deadline="2028-06-08 22:13:44.326798688 +0000 UTC" Nov 01 15:47:46.766340 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.766253 4363 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14046h25m57.560550078s" Nov 01 15:47:46.766340 ip-10-0-128-238 kubenswrapper[4363]: E1101 15:47:46.766288 4363 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-238.us-east-2.compute.internal\" not found" Nov 01 15:47:46.766676 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.766656 4363 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 15:47:46.767094 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.767074 4363 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 15:47:46.767094 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.767094 4363 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 15:47:46.767970 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.767939 4363 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60e635cd-c1f3-4d34-b795-2f6b16438f8f" volumeName="kubernetes.io/empty-dir/60e635cd-c1f3-4d34-b795-2f6b16438f8f-tmp" seLinuxMountContext="" Nov 01 15:47:46.768085 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.767970 4363 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="667a7db5-cf73-424c-9d18-53bfbc5f65db" volumeName="kubernetes.io/projected/667a7db5-cf73-424c-9d18-53bfbc5f65db-kube-api-access-t4s2s" seLinuxMountContext="" Nov 01 15:47:46.768085 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.767983 4363 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8254aafa-dd5b-4150-87e4-d165dab6bcb5" volumeName="kubernetes.io/configmap/8254aafa-dd5b-4150-87e4-d165dab6bcb5-ovnkube-script-lib" seLinuxMountContext="" Nov 01 15:47:46.768085 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.767995 4363 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7838987-12dc-47bc-9b39-a87ffd11ddaf" volumeName="kubernetes.io/secret/c7838987-12dc-47bc-9b39-a87ffd11ddaf-original-pull-secret" seLinuxMountContext="" Nov 01 15:47:46.768085 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.768007 4363 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f13a984b-8706-43d5-859f-abbfcaf58abf" volumeName="kubernetes.io/secret/f13a984b-8706-43d5-859f-abbfcaf58abf-agent-certs" seLinuxMountContext="" Nov 01 15:47:46.768085 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.768033 4363 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09f28a75-b66e-4933-84e2-eb0941e5f6a3" volumeName="kubernetes.io/configmap/09f28a75-b66e-4933-84e2-eb0941e5f6a3-cni-binary-copy" seLinuxMountContext="" Nov 01 15:47:46.768085 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.768045 4363 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12f347b0-ed8e-409d-bd18-261844bfeeed" volumeName="kubernetes.io/projected/12f347b0-ed8e-409d-bd18-261844bfeeed-kube-api-access-njf2l" seLinuxMountContext="" Nov 01 15:47:46.768085 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.768057 4363 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12f347b0-ed8e-409d-bd18-261844bfeeed" volumeName="kubernetes.io/secret/12f347b0-ed8e-409d-bd18-261844bfeeed-metrics-certs" seLinuxMountContext="" Nov 01 15:47:46.768085 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.768070 4363 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b8758dd-149c-4c37-9d32-a93fc953e548" volumeName="kubernetes.io/configmap/4b8758dd-149c-4c37-9d32-a93fc953e548-cni-binary-copy" seLinuxMountContext="" Nov 01 15:47:46.768085 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.768081 4363 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56e8fbab-06d3-4421-9995-822247d94d90" volumeName="kubernetes.io/projected/56e8fbab-06d3-4421-9995-822247d94d90-kube-api-access-4kmfx" seLinuxMountContext="" Nov 01 15:47:46.768464 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.768093 4363 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8254aafa-dd5b-4150-87e4-d165dab6bcb5" volumeName="kubernetes.io/configmap/8254aafa-dd5b-4150-87e4-d165dab6bcb5-ovnkube-config" seLinuxMountContext="" Nov 01 15:47:46.768464 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.768104 4363 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8254aafa-dd5b-4150-87e4-d165dab6bcb5" volumeName="kubernetes.io/projected/8254aafa-dd5b-4150-87e4-d165dab6bcb5-kube-api-access-glz5x" seLinuxMountContext="" Nov 01 15:47:46.768464 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.768131 4363 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8254aafa-dd5b-4150-87e4-d165dab6bcb5" volumeName="kubernetes.io/secret/8254aafa-dd5b-4150-87e4-d165dab6bcb5-ovn-node-metrics-cert" seLinuxMountContext="" Nov 01 15:47:46.768464 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.768143 4363 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34ffae54-fbc4-44f5-b785-a230835742c2" volumeName="kubernetes.io/projected/34ffae54-fbc4-44f5-b785-a230835742c2-kube-api-access-nlwtm" seLinuxMountContext="" Nov 01 15:47:46.768464 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.768158 4363 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b8758dd-149c-4c37-9d32-a93fc953e548" volumeName="kubernetes.io/configmap/4b8758dd-149c-4c37-9d32-a93fc953e548-cni-sysctl-allowlist" seLinuxMountContext="" Nov 01 15:47:46.768464 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.768170 4363 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b8758dd-149c-4c37-9d32-a93fc953e548" volumeName="kubernetes.io/projected/4b8758dd-149c-4c37-9d32-a93fc953e548-kube-api-access-xl7rt" seLinuxMountContext="" Nov 01 15:47:46.768464 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.768182 4363 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60e635cd-c1f3-4d34-b795-2f6b16438f8f" volumeName="kubernetes.io/empty-dir/60e635cd-c1f3-4d34-b795-2f6b16438f8f-etc-tuned" seLinuxMountContext="" Nov 01 15:47:46.768464 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.768193 4363 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8254aafa-dd5b-4150-87e4-d165dab6bcb5" volumeName="kubernetes.io/configmap/8254aafa-dd5b-4150-87e4-d165dab6bcb5-env-overrides" seLinuxMountContext="" Nov 01 15:47:46.768464 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.768206 4363 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e87fe52-8f65-43d9-a04a-a86d6227230d" volumeName="kubernetes.io/projected/8e87fe52-8f65-43d9-a04a-a86d6227230d-kube-api-access-wkkf9" seLinuxMountContext="" Nov 01 15:47:46.768464 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.768218 4363 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b53051b0-243d-4e03-926a-1f83372e648f" volumeName="kubernetes.io/configmap/b53051b0-243d-4e03-926a-1f83372e648f-iptables-alerter-script" seLinuxMountContext="" Nov 01 15:47:46.768464 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.768229 4363 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b53051b0-243d-4e03-926a-1f83372e648f" volumeName="kubernetes.io/projected/b53051b0-243d-4e03-926a-1f83372e648f-kube-api-access-pjb4z" seLinuxMountContext="" Nov 01 15:47:46.768464 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.768240 4363 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09f28a75-b66e-4933-84e2-eb0941e5f6a3" volumeName="kubernetes.io/projected/09f28a75-b66e-4933-84e2-eb0941e5f6a3-kube-api-access-gdpt9" seLinuxMountContext="" Nov 01 15:47:46.768464 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.768251 4363 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60e635cd-c1f3-4d34-b795-2f6b16438f8f" volumeName="kubernetes.io/projected/60e635cd-c1f3-4d34-b795-2f6b16438f8f-kube-api-access-h9r89" seLinuxMountContext="" Nov 01 15:47:46.768464 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.768262 4363 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="667a7db5-cf73-424c-9d18-53bfbc5f65db" volumeName="kubernetes.io/empty-dir/667a7db5-cf73-424c-9d18-53bfbc5f65db-tmp-dir" seLinuxMountContext="" Nov 01 15:47:46.768464 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.768275 4363 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f13a984b-8706-43d5-859f-abbfcaf58abf" volumeName="kubernetes.io/configmap/f13a984b-8706-43d5-859f-abbfcaf58abf-konnectivity-ca" seLinuxMountContext="" Nov 01 15:47:46.768464 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.768288 4363 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09f28a75-b66e-4933-84e2-eb0941e5f6a3" volumeName="kubernetes.io/configmap/09f28a75-b66e-4933-84e2-eb0941e5f6a3-multus-daemon-config" seLinuxMountContext="" Nov 01 15:47:46.768464 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.768303 4363 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b8758dd-149c-4c37-9d32-a93fc953e548" volumeName="kubernetes.io/configmap/4b8758dd-149c-4c37-9d32-a93fc953e548-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 01 15:47:46.768464 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.768316 4363 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56e8fbab-06d3-4421-9995-822247d94d90" volumeName="kubernetes.io/configmap/56e8fbab-06d3-4421-9995-822247d94d90-serviceca" seLinuxMountContext="" Nov 01 15:47:46.768464 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.768330 4363 reconstruct.go:97] "Volume reconstruction finished" Nov 01 15:47:46.768464 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.768337 4363 reconciler.go:29] "Reconciler: start to sync state" Nov 01 15:47:46.769411 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.769285 4363 factory.go:55] Registering systemd factory Nov 01 15:47:46.769411 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.769304 4363 factory.go:223] Registration of the systemd container factory successfully Nov 01 15:47:46.769625 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.769604 4363 factory.go:153] Registering CRI-O factory Nov 01 15:47:46.769625 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.769624 4363 factory.go:223] Registration of the crio container factory successfully Nov 01 15:47:46.769746 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.769683 4363 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 15:47:46.769746 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.769702 4363 factory.go:103] Registering Raw factory Nov 01 15:47:46.769746 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.769716 4363 manager.go:1199] Started watching for new ooms in manager Nov 01 15:47:46.770347 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.770123 4363 manager.go:319] Starting recovery of all containers Nov 01 15:47:46.771728 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.771708 4363 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 15:47:46.771842 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.771729 4363 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 15:47:46.771842 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.771807 4363 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 15:47:46.771965 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.771891 4363 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 01 15:47:46.775990 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.775970 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtwhw_8254aafa-dd5b-4150-87e4-d165dab6bcb5/ovn-acl-logging/0.log" Nov 01 15:47:46.786762 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.786742 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtwhw_8254aafa-dd5b-4150-87e4-d165dab6bcb5/ovn-acl-logging/0.log" Nov 01 15:47:46.787583 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.787555 4363 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 15:47:46.789127 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.789100 4363 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 15:47:46.789127 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.789129 4363 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 15:47:46.789306 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.789149 4363 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 15:47:46.789306 ip-10-0-128-238 kubenswrapper[4363]: E1101 15:47:46.789189 4363 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 01 15:47:46.792231 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.792211 4363 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 15:47:46.796095 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.796078 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtwhw_8254aafa-dd5b-4150-87e4-d165dab6bcb5/ovn-acl-logging/0.log" Nov 01 15:47:46.796441 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.796417 4363 generic.go:356] "Generic (PLEG): container finished" podID="8254aafa-dd5b-4150-87e4-d165dab6bcb5" containerID="d5b99f7a3b5406a8015f71ca144410db7f2500a34e0d1d80f0036060092d7a2a" exitCode=1 Nov 01 15:47:46.801958 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.801935 4363 generic.go:356] "Generic (PLEG): container finished" podID="4b8758dd-149c-4c37-9d32-a93fc953e548" containerID="7890e88a268d8b9b6f9a79069478a524f89fe6104000d6fdb7031ba7f0650af0" exitCode=0 Nov 01 15:47:46.802077 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.801960 4363 generic.go:356] "Generic (PLEG): container finished" podID="4b8758dd-149c-4c37-9d32-a93fc953e548" containerID="15905a15a06fb738a13d5f2da47cddb9903ee5acc823dee7632fb496f08ee5ea" exitCode=0 Nov 01 15:47:46.802077 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.801973 4363 generic.go:356] "Generic (PLEG): container finished" podID="4b8758dd-149c-4c37-9d32-a93fc953e548" containerID="3704f772e47fb53c46a08a5445a3865de8b608c61c1a9a1665951a7d79223023" exitCode=0 Nov 01 15:47:46.802077 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.801983 4363 generic.go:356] "Generic (PLEG): container finished" podID="4b8758dd-149c-4c37-9d32-a93fc953e548" containerID="f982b3d4ac58ef8e5d3b1e00c5dfded0ea728c741dfdd53ee697e7c201c31b4b" exitCode=0 Nov 01 15:47:46.802077 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.801991 4363 generic.go:356] "Generic (PLEG): container finished" podID="4b8758dd-149c-4c37-9d32-a93fc953e548" containerID="c5ffd3456da5aa6e9a10e8aeae741dc451fb4b0f8ca3c3fec7d3fd15039ae740" exitCode=0 Nov 01 15:47:46.802077 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.802003 4363 generic.go:356] "Generic (PLEG): container finished" podID="4b8758dd-149c-4c37-9d32-a93fc953e548" containerID="8643efd57aa767d3e191eeeadd2a91fdaa4c4dafeff7c800d2e8371fdcc4e490" exitCode=0 Nov 01 15:47:46.817949 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.817922 4363 generic.go:356] "Generic (PLEG): container finished" podID="5d3a91f23ccfbb9c1af1cd6d04b30fd6" containerID="c13981f4537784a0dbec6ec40fe85598cf111a0142799131942fd7bce219988a" exitCode=0 Nov 01 15:47:46.832577 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.832553 4363 manager.go:324] Recovery completed Nov 01 15:47:46.845438 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.845422 4363 cpu_manager.go:226] "Starting" policy="none" Nov 01 15:47:46.845438 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.845434 4363 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 15:47:46.845584 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.845449 4363 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 15:47:46.845584 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.845558 4363 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 01 15:47:46.845584 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.845567 4363 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 01 15:47:46.845584 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.845585 4363 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 01 15:47:46.845736 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.845590 4363 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 01 15:47:46.845736 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.845595 4363 policy_none.go:50] "Start" Nov 01 15:47:46.845736 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.845603 4363 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 15:47:46.845736 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.845611 4363 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 15:47:46.845736 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.845726 4363 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 01 15:47:46.845736 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.845733 4363 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 01 15:47:46.845736 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.845738 4363 policy_none.go:44] "Start" Nov 01 15:47:46.849753 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.849723 4363 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 15:47:46.852318 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.852282 4363 manager.go:341] "Starting Device Plugin manager" Nov 01 15:47:46.852776 ip-10-0-128-238 kubenswrapper[4363]: E1101 15:47:46.852760 4363 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 15:47:46.852776 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.852775 4363 server.go:87] "Starting device plugin registration server" Nov 01 15:47:46.853032 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.853005 4363 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 15:47:46.853088 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.853030 4363 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 15:47:46.853134 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.853097 4363 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 15:47:46.853231 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.853198 4363 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 15:47:46.853294 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.853245 4363 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 15:47:46.853294 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.853252 4363 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 15:47:46.853378 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.853311 4363 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T15:47:46.853228483Z","UUID":"2f0fc6c3-6b3e-4664-b3aa-e058794c2edf","Handler":null,"Name":"","Endpoint":""} Nov 01 15:47:46.854507 ip-10-0-128-238 kubenswrapper[4363]: E1101 15:47:46.854475 4363 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 15:47:46.854644 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.854626 4363 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 15:47:46.854716 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.854652 4363 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 15:47:46.889931 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.889878 4363 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-238.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-238.us-east-2.compute.internal"] Nov 01 15:47:46.890368 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.890339 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-238.us-east-2.compute.internal" event={"ID":"ae861e52d07eb0e1937b2b22576e5cfd","Type":"ContainerStarted","Data":"a8353b93a829549def42b86442c888f2e0d6c41d1372611e7b8188895cb4a281"} Nov 01 15:47:46.890447 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.890375 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-238.us-east-2.compute.internal" event={"ID":"ae861e52d07eb0e1937b2b22576e5cfd","Type":"ContainerStarted","Data":"eed1a6e1117b93c7d32aa445a6b08f47f555ec343f4d3007de78f66134168519"} Nov 01 15:47:46.890447 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.890388 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-238.us-east-2.compute.internal" event={"ID":"5d3a91f23ccfbb9c1af1cd6d04b30fd6","Type":"ContainerStarted","Data":"3ef08cfc3e9507e6623121bf67835e082b661d8525bcb00fbafabf7e73f68ce9"} Nov 01 15:47:46.890447 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.890400 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-238.us-east-2.compute.internal" event={"ID":"5d3a91f23ccfbb9c1af1cd6d04b30fd6","Type":"ContainerDied","Data":"c13981f4537784a0dbec6ec40fe85598cf111a0142799131942fd7bce219988a"} Nov 01 15:47:46.890447 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.890413 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-238.us-east-2.compute.internal" event={"ID":"5d3a91f23ccfbb9c1af1cd6d04b30fd6","Type":"ContainerStarted","Data":"9d5263a747b1d06072586979cb12c5bd28b85c8eb029fe696c87a956fbdf2787"} Nov 01 15:47:46.890643 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.890525 4363 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-238.us-east-2.compute.internal" Nov 01 15:47:46.890643 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.890552 4363 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-238.us-east-2.compute.internal" Nov 01 15:47:46.899134 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.899118 4363 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 15:47:46.899172 ip-10-0-128-238 kubenswrapper[4363]: E1101 15:47:46.899163 4363 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-238.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-238.us-east-2.compute.internal" Nov 01 15:47:46.899674 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.899660 4363 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 15:47:46.899707 ip-10-0-128-238 kubenswrapper[4363]: E1101 15:47:46.899696 4363 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-238.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-238.us-east-2.compute.internal" Nov 01 15:47:46.953806 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.953783 4363 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 15:47:46.954978 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.954962 4363 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-238.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 15:47:46.955077 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.954997 4363 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-238.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 15:47:46.955077 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.955011 4363 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-238.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 15:47:46.955077 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.955067 4363 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-238.us-east-2.compute.internal" Nov 01 15:47:46.965887 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.965868 4363 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-238.us-east-2.compute.internal" Nov 01 15:47:46.965939 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.965918 4363 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-238.us-east-2.compute.internal" Nov 01 15:47:46.972372 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.972354 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5d3a91f23ccfbb9c1af1cd6d04b30fd6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-238.us-east-2.compute.internal\" (UID: \"5d3a91f23ccfbb9c1af1cd6d04b30fd6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-238.us-east-2.compute.internal" Nov 01 15:47:46.972448 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.972384 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d3a91f23ccfbb9c1af1cd6d04b30fd6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-238.us-east-2.compute.internal\" (UID: \"5d3a91f23ccfbb9c1af1cd6d04b30fd6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-238.us-east-2.compute.internal" Nov 01 15:47:46.972448 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:46.972409 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ae861e52d07eb0e1937b2b22576e5cfd-config\") pod \"kube-apiserver-proxy-ip-10-0-128-238.us-east-2.compute.internal\" (UID: \"ae861e52d07eb0e1937b2b22576e5cfd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-238.us-east-2.compute.internal" Nov 01 15:47:47.073249 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.073227 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5d3a91f23ccfbb9c1af1cd6d04b30fd6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-238.us-east-2.compute.internal\" (UID: \"5d3a91f23ccfbb9c1af1cd6d04b30fd6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-238.us-east-2.compute.internal" Nov 01 15:47:47.073365 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.073261 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d3a91f23ccfbb9c1af1cd6d04b30fd6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-238.us-east-2.compute.internal\" (UID: \"5d3a91f23ccfbb9c1af1cd6d04b30fd6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-238.us-east-2.compute.internal" Nov 01 15:47:47.073365 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.073288 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ae861e52d07eb0e1937b2b22576e5cfd-config\") pod \"kube-apiserver-proxy-ip-10-0-128-238.us-east-2.compute.internal\" (UID: \"ae861e52d07eb0e1937b2b22576e5cfd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-238.us-east-2.compute.internal" Nov 01 15:47:47.073365 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.073341 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ae861e52d07eb0e1937b2b22576e5cfd-config\") pod \"kube-apiserver-proxy-ip-10-0-128-238.us-east-2.compute.internal\" (UID: \"ae861e52d07eb0e1937b2b22576e5cfd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-238.us-east-2.compute.internal" Nov 01 15:47:47.073365 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.073347 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d3a91f23ccfbb9c1af1cd6d04b30fd6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-238.us-east-2.compute.internal\" (UID: \"5d3a91f23ccfbb9c1af1cd6d04b30fd6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-238.us-east-2.compute.internal" Nov 01 15:47:47.073508 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.073347 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5d3a91f23ccfbb9c1af1cd6d04b30fd6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-238.us-east-2.compute.internal\" (UID: \"5d3a91f23ccfbb9c1af1cd6d04b30fd6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-238.us-east-2.compute.internal" Nov 01 15:47:47.751204 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.751173 4363 apiserver.go:52] "Watching apiserver" Nov 01 15:47:47.756068 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.756046 4363 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 15:47:47.756398 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.756383 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-238.us-east-2.compute.internal","openshift-network-operator/iptables-alerter-86kzp","openshift-ovn-kubernetes/ovnkube-node-jtwhw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7","openshift-image-registry/node-ca-44klx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-238.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-f6klc","openshift-multus/multus-x2wlk","openshift-cluster-node-tuning-operator/tuned-92m6d","openshift-dns/node-resolver-hkkmh","openshift-multus/network-metrics-daemon-cx4d8","openshift-network-diagnostics/network-check-target-l26rw","kube-system/global-pull-secret-syncer-ckz9h","kube-system/konnectivity-agent-d7slg"] Nov 01 15:47:47.763177 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.763151 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 15:47:47.763671 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.763647 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 15:47:47.763769 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.763649 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 15:47:47.763769 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.763738 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 15:47:47.763892 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.763772 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 15:47:47.763892 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.763841 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 15:47:47.763892 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.763862 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 15:47:47.763892 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.763883 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 15:47:47.764084 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.763921 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6x5r5\"" Nov 01 15:47:47.764084 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.763921 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pjc9m\"" Nov 01 15:47:47.764084 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.763989 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-58l9j\"" Nov 01 15:47:47.764084 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.764002 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 15:47:47.764084 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.764004 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 15:47:47.764312 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.764110 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5vhjr\"" Nov 01 15:47:47.764312 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.764127 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 15:47:47.764312 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.764203 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 15:47:47.764312 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.764302 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 15:47:47.764479 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.764323 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 15:47:47.764479 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.764399 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 15:47:47.764479 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.764423 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 15:47:47.764479 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.764451 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 15:47:47.764479 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.764460 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 15:47:47.764479 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.764431 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 15:47:47.764682 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.764492 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9hxd8\"" Nov 01 15:47:47.764682 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.764510 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 15:47:47.764682 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.764511 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 15:47:47.764765 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.764703 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 15:47:47.765170 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.765155 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bwtj5\"" Nov 01 15:47:47.765242 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.765178 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 15:47:47.765242 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.765213 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jsz89\"" Nov 01 15:47:47.765242 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.765229 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 15:47:47.765359 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.765246 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 15:47:47.765359 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.765267 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-ttxgn\"" Nov 01 15:47:47.765422 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.765391 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4p8cp\"" Nov 01 15:47:47.765453 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.765424 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 15:47:47.765517 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.765505 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-28hsn\"" Nov 01 15:47:47.765573 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.765558 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-2bqxm\"" Nov 01 15:47:47.765814 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.765801 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-qm9xj\"" Nov 01 15:47:47.765897 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.765884 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 15:47:47.765897 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.765893 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 15:47:47.765983 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.765899 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 15:47:47.765983 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.765962 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 15:47:47.766068 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.766048 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 15:47:47.767693 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.767680 4363 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 15:47:47.776140 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.776121 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8e87fe52-8f65-43d9-a04a-a86d6227230d-registration-dir\") pod \"aws-ebs-csi-driver-node-cgvz7\" (UID: \"8e87fe52-8f65-43d9-a04a-a86d6227230d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" Nov 01 15:47:47.776226 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.776146 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8e87fe52-8f65-43d9-a04a-a86d6227230d-sys-fs\") pod \"aws-ebs-csi-driver-node-cgvz7\" (UID: \"8e87fe52-8f65-43d9-a04a-a86d6227230d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" Nov 01 15:47:47.776226 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.776162 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b53051b0-243d-4e03-926a-1f83372e648f-host-slash\") pod \"iptables-alerter-86kzp\" (UID: \"b53051b0-243d-4e03-926a-1f83372e648f\") " pod="openshift-network-operator/iptables-alerter-86kzp" Nov 01 15:47:47.776226 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.776179 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7838987-12dc-47bc-9b39-a87ffd11ddaf-original-pull-secret\") pod \"global-pull-secret-syncer-ckz9h\" (UID: \"c7838987-12dc-47bc-9b39-a87ffd11ddaf\") " pod="kube-system/global-pull-secret-syncer-ckz9h" Nov 01 15:47:47.776226 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.776194 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-run-ovn\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.776226 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.776210 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.776442 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.776258 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8254aafa-dd5b-4150-87e4-d165dab6bcb5-ovnkube-script-lib\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.776442 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.776420 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8e87fe52-8f65-43d9-a04a-a86d6227230d-etc-selinux\") pod \"aws-ebs-csi-driver-node-cgvz7\" (UID: \"8e87fe52-8f65-43d9-a04a-a86d6227230d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" Nov 01 15:47:47.776531 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.776451 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-multus-cni-dir\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.776531 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.776503 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjb4z\" (UniqueName: \"kubernetes.io/projected/b53051b0-243d-4e03-926a-1f83372e648f-kube-api-access-pjb4z\") pod \"iptables-alerter-86kzp\" (UID: \"b53051b0-243d-4e03-926a-1f83372e648f\") " pod="openshift-network-operator/iptables-alerter-86kzp" Nov 01 15:47:47.776531 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.776525 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-etc-sysctl-conf\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:47.776677 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.776545 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c7838987-12dc-47bc-9b39-a87ffd11ddaf-dbus\") pod \"global-pull-secret-syncer-ckz9h\" (UID: \"c7838987-12dc-47bc-9b39-a87ffd11ddaf\") " pod="kube-system/global-pull-secret-syncer-ckz9h" Nov 01 15:47:47.776677 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.776578 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-host-run-netns\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.776677 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.776594 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7838987-12dc-47bc-9b39-a87ffd11ddaf-original-pull-secret\") pod \"global-pull-secret-syncer-ckz9h\" (UID: \"c7838987-12dc-47bc-9b39-a87ffd11ddaf\") " pod="kube-system/global-pull-secret-syncer-ckz9h" Nov 01 15:47:47.776677 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.776632 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8254aafa-dd5b-4150-87e4-d165dab6bcb5-ovnkube-script-lib\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.776677 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.776656 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-node-log\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.776905 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.776691 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-host-run-ovn-kubernetes\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.776905 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.776725 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b53051b0-243d-4e03-926a-1f83372e648f-iptables-alerter-script\") pod \"iptables-alerter-86kzp\" (UID: \"b53051b0-243d-4e03-926a-1f83372e648f\") " pod="openshift-network-operator/iptables-alerter-86kzp" Nov 01 15:47:47.776905 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.776760 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-run\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:47.776905 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.776802 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-var-lib-kubelet\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:47.776905 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.776832 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/60e635cd-c1f3-4d34-b795-2f6b16438f8f-etc-tuned\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:47.776905 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.776850 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-host-var-lib-cni-multus\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.776905 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.776874 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b8758dd-149c-4c37-9d32-a93fc953e548-system-cni-dir\") pod \"multus-additional-cni-plugins-f6klc\" (UID: \"4b8758dd-149c-4c37-9d32-a93fc953e548\") " pod="openshift-multus/multus-additional-cni-plugins-f6klc" Nov 01 15:47:47.777209 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.776919 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/60e635cd-c1f3-4d34-b795-2f6b16438f8f-etc-tuned\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:47.777209 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.776950 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b53051b0-243d-4e03-926a-1f83372e648f-iptables-alerter-script\") pod \"iptables-alerter-86kzp\" (UID: \"b53051b0-243d-4e03-926a-1f83372e648f\") " pod="openshift-network-operator/iptables-alerter-86kzp" Nov 01 15:47:47.777209 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.776993 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-etc-kubernetes\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:47.777209 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.777046 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9r89\" (UniqueName: \"kubernetes.io/projected/60e635cd-c1f3-4d34-b795-2f6b16438f8f-kube-api-access-h9r89\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:47.777209 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.777096 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4b8758dd-149c-4c37-9d32-a93fc953e548-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f6klc\" (UID: \"4b8758dd-149c-4c37-9d32-a93fc953e548\") " pod="openshift-multus/multus-additional-cni-plugins-f6klc" Nov 01 15:47:47.777209 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.777114 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-etc-sysconfig\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:47.777209 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.777134 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-etc-systemd\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:47.777209 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.777149 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8254aafa-dd5b-4150-87e4-d165dab6bcb5-env-overrides\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.777209 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.777166 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8e87fe52-8f65-43d9-a04a-a86d6227230d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cgvz7\" (UID: \"8e87fe52-8f65-43d9-a04a-a86d6227230d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" Nov 01 15:47:47.777209 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.777200 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/56e8fbab-06d3-4421-9995-822247d94d90-serviceca\") pod \"node-ca-44klx\" (UID: \"56e8fbab-06d3-4421-9995-822247d94d90\") " pod="openshift-image-registry/node-ca-44klx" Nov 01 15:47:47.777617 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.777273 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-host-run-netns\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.777617 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.777305 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/09f28a75-b66e-4933-84e2-eb0941e5f6a3-multus-daemon-config\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.777617 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.777333 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdpt9\" (UniqueName: \"kubernetes.io/projected/09f28a75-b66e-4933-84e2-eb0941e5f6a3-kube-api-access-gdpt9\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.777617 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.777372 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8254aafa-dd5b-4150-87e4-d165dab6bcb5-env-overrides\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.777617 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.777422 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4s2s\" (UniqueName: \"kubernetes.io/projected/667a7db5-cf73-424c-9d18-53bfbc5f65db-kube-api-access-t4s2s\") pod \"node-resolver-hkkmh\" (UID: \"667a7db5-cf73-424c-9d18-53bfbc5f65db\") " pod="openshift-dns/node-resolver-hkkmh" Nov 01 15:47:47.777617 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.777422 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4b8758dd-149c-4c37-9d32-a93fc953e548-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f6klc\" (UID: \"4b8758dd-149c-4c37-9d32-a93fc953e548\") " pod="openshift-multus/multus-additional-cni-plugins-f6klc" Nov 01 15:47:47.777617 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.777516 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/09f28a75-b66e-4933-84e2-eb0941e5f6a3-multus-daemon-config\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.777617 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.777512 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b8758dd-149c-4c37-9d32-a93fc953e548-cnibin\") pod \"multus-additional-cni-plugins-f6klc\" (UID: \"4b8758dd-149c-4c37-9d32-a93fc953e548\") " pod="openshift-multus/multus-additional-cni-plugins-f6klc" Nov 01 15:47:47.777617 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.777529 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/56e8fbab-06d3-4421-9995-822247d94d90-serviceca\") pod \"node-ca-44klx\" (UID: \"56e8fbab-06d3-4421-9995-822247d94d90\") " pod="openshift-image-registry/node-ca-44klx" Nov 01 15:47:47.777970 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.777624 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4b8758dd-149c-4c37-9d32-a93fc953e548-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f6klc\" (UID: \"4b8758dd-149c-4c37-9d32-a93fc953e548\") " pod="openshift-multus/multus-additional-cni-plugins-f6klc" Nov 01 15:47:47.777970 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.777653 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-host-var-lib-kubelet\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.777970 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.777676 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-multus-conf-dir\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.777970 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.777725 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f13a984b-8706-43d5-859f-abbfcaf58abf-konnectivity-ca\") pod \"konnectivity-agent-d7slg\" (UID: \"f13a984b-8706-43d5-859f-abbfcaf58abf\") " pod="kube-system/konnectivity-agent-d7slg" Nov 01 15:47:47.777970 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.777757 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-host-kubelet\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.777970 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.777782 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-systemd-units\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.777970 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.777795 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4b8758dd-149c-4c37-9d32-a93fc953e548-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f6klc\" (UID: \"4b8758dd-149c-4c37-9d32-a93fc953e548\") " pod="openshift-multus/multus-additional-cni-plugins-f6klc" Nov 01 15:47:47.777970 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.777805 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-host-slash\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.777970 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.777853 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-var-lib-openvswitch\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.777970 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.777889 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-host-var-lib-cni-bin\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.777970 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.777941 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8e87fe52-8f65-43d9-a04a-a86d6227230d-device-dir\") pod \"aws-ebs-csi-driver-node-cgvz7\" (UID: \"8e87fe52-8f65-43d9-a04a-a86d6227230d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" Nov 01 15:47:47.777970 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.777960 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f13a984b-8706-43d5-859f-abbfcaf58abf-konnectivity-ca\") pod \"konnectivity-agent-d7slg\" (UID: \"f13a984b-8706-43d5-859f-abbfcaf58abf\") " pod="kube-system/konnectivity-agent-d7slg" Nov 01 15:47:47.777970 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.777969 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/09f28a75-b66e-4933-84e2-eb0941e5f6a3-cni-binary-copy\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.778572 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.777996 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl7rt\" (UniqueName: \"kubernetes.io/projected/4b8758dd-149c-4c37-9d32-a93fc953e548-kube-api-access-xl7rt\") pod \"multus-additional-cni-plugins-f6klc\" (UID: \"4b8758dd-149c-4c37-9d32-a93fc953e548\") " pod="openshift-multus/multus-additional-cni-plugins-f6klc" Nov 01 15:47:47.778572 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.778075 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-run-systemd\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.778572 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.778101 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-etc-openvswitch\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.778572 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.778126 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8254aafa-dd5b-4150-87e4-d165dab6bcb5-ovn-node-metrics-cert\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.778572 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.778150 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-etc-modprobe-d\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:47.778572 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.778163 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/56e8fbab-06d3-4421-9995-822247d94d90-host\") pod \"node-ca-44klx\" (UID: \"56e8fbab-06d3-4421-9995-822247d94d90\") " pod="openshift-image-registry/node-ca-44klx" Nov 01 15:47:47.778572 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.778185 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/09f28a75-b66e-4933-84e2-eb0941e5f6a3-cni-binary-copy\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.778572 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.778199 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-multus-socket-dir-parent\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.778572 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.778256 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b8758dd-149c-4c37-9d32-a93fc953e548-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f6klc\" (UID: \"4b8758dd-149c-4c37-9d32-a93fc953e548\") " pod="openshift-multus/multus-additional-cni-plugins-f6klc" Nov 01 15:47:47.778572 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.778299 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-etc-sysctl-d\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:47.778572 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.778314 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-host-run-k8s-cni-cncf-io\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.778572 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.778330 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/667a7db5-cf73-424c-9d18-53bfbc5f65db-tmp-dir\") pod \"node-resolver-hkkmh\" (UID: \"667a7db5-cf73-424c-9d18-53bfbc5f65db\") " pod="openshift-dns/node-resolver-hkkmh" Nov 01 15:47:47.778572 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.778344 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12f347b0-ed8e-409d-bd18-261844bfeeed-metrics-certs\") pod \"network-metrics-daemon-cx4d8\" (UID: \"12f347b0-ed8e-409d-bd18-261844bfeeed\") " pod="openshift-multus/network-metrics-daemon-cx4d8" Nov 01 15:47:47.778572 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.778358 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-sys\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:47.778572 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.778373 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kmfx\" (UniqueName: \"kubernetes.io/projected/56e8fbab-06d3-4421-9995-822247d94d90-kube-api-access-4kmfx\") pod \"node-ca-44klx\" (UID: \"56e8fbab-06d3-4421-9995-822247d94d90\") " pod="openshift-image-registry/node-ca-44klx" Nov 01 15:47:47.778572 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.778401 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njf2l\" (UniqueName: \"kubernetes.io/projected/12f347b0-ed8e-409d-bd18-261844bfeeed-kube-api-access-njf2l\") pod \"network-metrics-daemon-cx4d8\" (UID: \"12f347b0-ed8e-409d-bd18-261844bfeeed\") " pod="openshift-multus/network-metrics-daemon-cx4d8" Nov 01 15:47:47.778572 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.778425 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-lib-modules\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:47.778572 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.778496 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/667a7db5-cf73-424c-9d18-53bfbc5f65db-tmp-dir\") pod \"node-resolver-hkkmh\" (UID: \"667a7db5-cf73-424c-9d18-53bfbc5f65db\") " pod="openshift-dns/node-resolver-hkkmh" Nov 01 15:47:47.779301 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.778592 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8254aafa-dd5b-4150-87e4-d165dab6bcb5-ovn-node-metrics-cert\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.779301 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.778596 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-run-openvswitch\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.779301 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.778632 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12f347b0-ed8e-409d-bd18-261844bfeeed-metrics-certs\") pod \"network-metrics-daemon-cx4d8\" (UID: \"12f347b0-ed8e-409d-bd18-261844bfeeed\") " pod="openshift-multus/network-metrics-daemon-cx4d8" Nov 01 15:47:47.779301 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.778635 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-log-socket\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.779301 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.778665 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b8758dd-149c-4c37-9d32-a93fc953e548-cni-binary-copy\") pod \"multus-additional-cni-plugins-f6klc\" (UID: \"4b8758dd-149c-4c37-9d32-a93fc953e548\") " pod="openshift-multus/multus-additional-cni-plugins-f6klc" Nov 01 15:47:47.779301 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.778719 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f13a984b-8706-43d5-859f-abbfcaf58abf-agent-certs\") pod \"konnectivity-agent-d7slg\" (UID: \"f13a984b-8706-43d5-859f-abbfcaf58abf\") " pod="kube-system/konnectivity-agent-d7slg" Nov 01 15:47:47.779301 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.778763 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8254aafa-dd5b-4150-87e4-d165dab6bcb5-ovnkube-config\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.779301 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.778789 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkkf9\" (UniqueName: \"kubernetes.io/projected/8e87fe52-8f65-43d9-a04a-a86d6227230d-kube-api-access-wkkf9\") pod \"aws-ebs-csi-driver-node-cgvz7\" (UID: \"8e87fe52-8f65-43d9-a04a-a86d6227230d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" Nov 01 15:47:47.779301 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.778848 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b8758dd-149c-4c37-9d32-a93fc953e548-cni-binary-copy\") pod \"multus-additional-cni-plugins-f6klc\" (UID: \"4b8758dd-149c-4c37-9d32-a93fc953e548\") " pod="openshift-multus/multus-additional-cni-plugins-f6klc" Nov 01 15:47:47.779301 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.778874 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-system-cni-dir\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.779301 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.778897 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-hostroot\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.779301 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.778911 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-host-run-multus-certs\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.779301 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.778931 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlwtm\" (UniqueName: \"kubernetes.io/projected/34ffae54-fbc4-44f5-b785-a230835742c2-kube-api-access-nlwtm\") pod \"network-check-target-l26rw\" (UID: \"34ffae54-fbc4-44f5-b785-a230835742c2\") " pod="openshift-network-diagnostics/network-check-target-l26rw" Nov 01 15:47:47.779301 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.778987 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-host-cni-bin\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.779301 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.778995 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8254aafa-dd5b-4150-87e4-d165dab6bcb5-ovnkube-config\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.779301 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.779047 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f13a984b-8706-43d5-859f-abbfcaf58abf-agent-certs\") pod \"konnectivity-agent-d7slg\" (UID: \"f13a984b-8706-43d5-859f-abbfcaf58abf\") " pod="kube-system/konnectivity-agent-d7slg" Nov 01 15:47:47.779301 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.779126 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glz5x\" (UniqueName: \"kubernetes.io/projected/8254aafa-dd5b-4150-87e4-d165dab6bcb5-kube-api-access-glz5x\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.779301 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.779165 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8e87fe52-8f65-43d9-a04a-a86d6227230d-socket-dir\") pod \"aws-ebs-csi-driver-node-cgvz7\" (UID: \"8e87fe52-8f65-43d9-a04a-a86d6227230d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" Nov 01 15:47:47.779301 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.779271 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-cnibin\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.779301 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.779297 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-etc-kubernetes\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.779923 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.779348 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/667a7db5-cf73-424c-9d18-53bfbc5f65db-hosts-file\") pod \"node-resolver-hkkmh\" (UID: \"667a7db5-cf73-424c-9d18-53bfbc5f65db\") " pod="openshift-dns/node-resolver-hkkmh" Nov 01 15:47:47.779923 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.779387 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-host\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:47.779923 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.779414 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60e635cd-c1f3-4d34-b795-2f6b16438f8f-tmp\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:47.779923 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.779518 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c7838987-12dc-47bc-9b39-a87ffd11ddaf-kubelet-config\") pod \"global-pull-secret-syncer-ckz9h\" (UID: \"c7838987-12dc-47bc-9b39-a87ffd11ddaf\") " pod="kube-system/global-pull-secret-syncer-ckz9h" Nov 01 15:47:47.779923 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.779523 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60e635cd-c1f3-4d34-b795-2f6b16438f8f-tmp\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:47.779923 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.779539 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-host-cni-netd\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.782491 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.782474 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9r89\" (UniqueName: \"kubernetes.io/projected/60e635cd-c1f3-4d34-b795-2f6b16438f8f-kube-api-access-h9r89\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:47.785480 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.785464 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjb4z\" (UniqueName: \"kubernetes.io/projected/b53051b0-243d-4e03-926a-1f83372e648f-kube-api-access-pjb4z\") pod \"iptables-alerter-86kzp\" (UID: \"b53051b0-243d-4e03-926a-1f83372e648f\") " pod="openshift-network-operator/iptables-alerter-86kzp" Nov 01 15:47:47.785533 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.785518 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdpt9\" (UniqueName: \"kubernetes.io/projected/09f28a75-b66e-4933-84e2-eb0941e5f6a3-kube-api-access-gdpt9\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.786317 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.786302 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4s2s\" (UniqueName: \"kubernetes.io/projected/667a7db5-cf73-424c-9d18-53bfbc5f65db-kube-api-access-t4s2s\") pod \"node-resolver-hkkmh\" (UID: \"667a7db5-cf73-424c-9d18-53bfbc5f65db\") " pod="openshift-dns/node-resolver-hkkmh" Nov 01 15:47:47.786578 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.786563 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl7rt\" (UniqueName: \"kubernetes.io/projected/4b8758dd-149c-4c37-9d32-a93fc953e548-kube-api-access-xl7rt\") pod \"multus-additional-cni-plugins-f6klc\" (UID: \"4b8758dd-149c-4c37-9d32-a93fc953e548\") " pod="openshift-multus/multus-additional-cni-plugins-f6klc" Nov 01 15:47:47.786830 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.786813 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kmfx\" (UniqueName: \"kubernetes.io/projected/56e8fbab-06d3-4421-9995-822247d94d90-kube-api-access-4kmfx\") pod \"node-ca-44klx\" (UID: \"56e8fbab-06d3-4421-9995-822247d94d90\") " pod="openshift-image-registry/node-ca-44klx" Nov 01 15:47:47.787252 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.787234 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glz5x\" (UniqueName: \"kubernetes.io/projected/8254aafa-dd5b-4150-87e4-d165dab6bcb5-kube-api-access-glz5x\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.787336 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.787289 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njf2l\" (UniqueName: \"kubernetes.io/projected/12f347b0-ed8e-409d-bd18-261844bfeeed-kube-api-access-njf2l\") pod \"network-metrics-daemon-cx4d8\" (UID: \"12f347b0-ed8e-409d-bd18-261844bfeeed\") " pod="openshift-multus/network-metrics-daemon-cx4d8" Nov 01 15:47:47.787488 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.787466 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlwtm\" (UniqueName: \"kubernetes.io/projected/34ffae54-fbc4-44f5-b785-a230835742c2-kube-api-access-nlwtm\") pod \"network-check-target-l26rw\" (UID: \"34ffae54-fbc4-44f5-b785-a230835742c2\") " pod="openshift-network-diagnostics/network-check-target-l26rw" Nov 01 15:47:47.787653 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.787640 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkkf9\" (UniqueName: \"kubernetes.io/projected/8e87fe52-8f65-43d9-a04a-a86d6227230d-kube-api-access-wkkf9\") pod \"aws-ebs-csi-driver-node-cgvz7\" (UID: \"8e87fe52-8f65-43d9-a04a-a86d6227230d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" Nov 01 15:47:47.823542 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.823524 4363 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-238.us-east-2.compute.internal" Nov 01 15:47:47.823653 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.823638 4363 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-238.us-east-2.compute.internal" Nov 01 15:47:47.834818 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.834797 4363 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 15:47:47.834908 ip-10-0-128-238 kubenswrapper[4363]: E1101 15:47:47.834853 4363 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-238.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-238.us-east-2.compute.internal" Nov 01 15:47:47.835142 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.835127 4363 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 15:47:47.835187 ip-10-0-128-238 kubenswrapper[4363]: E1101 15:47:47.835159 4363 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-238.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-238.us-east-2.compute.internal" Nov 01 15:47:47.880330 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880311 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-host-run-k8s-cni-cncf-io\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.880415 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880336 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-sys\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:47.880415 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880358 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-lib-modules\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:47.880415 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880383 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-run-openvswitch\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.880415 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880407 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-log-socket\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.880556 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880422 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-host-run-k8s-cni-cncf-io\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.880556 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880434 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-system-cni-dir\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.880556 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880437 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-sys\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:47.880556 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880446 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-run-openvswitch\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.880556 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880458 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-hostroot\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.880556 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880475 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-log-socket\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.880556 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880484 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-host-run-multus-certs\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.880556 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880495 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-system-cni-dir\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.880556 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880497 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-lib-modules\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:47.880556 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880513 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-hostroot\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.880556 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880508 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-host-cni-bin\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.880556 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880531 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-host-run-multus-certs\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.880556 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880548 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8e87fe52-8f65-43d9-a04a-a86d6227230d-socket-dir\") pod \"aws-ebs-csi-driver-node-cgvz7\" (UID: \"8e87fe52-8f65-43d9-a04a-a86d6227230d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" Nov 01 15:47:47.881150 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880577 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-cnibin\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.881150 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880592 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-etc-kubernetes\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.881150 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880604 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-host-cni-bin\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.881150 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880621 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-etc-kubernetes\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.881150 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880629 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-cnibin\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.881150 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880635 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8e87fe52-8f65-43d9-a04a-a86d6227230d-socket-dir\") pod \"aws-ebs-csi-driver-node-cgvz7\" (UID: \"8e87fe52-8f65-43d9-a04a-a86d6227230d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" Nov 01 15:47:47.881150 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880640 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/667a7db5-cf73-424c-9d18-53bfbc5f65db-hosts-file\") pod \"node-resolver-hkkmh\" (UID: \"667a7db5-cf73-424c-9d18-53bfbc5f65db\") " pod="openshift-dns/node-resolver-hkkmh" Nov 01 15:47:47.881150 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880668 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/667a7db5-cf73-424c-9d18-53bfbc5f65db-hosts-file\") pod \"node-resolver-hkkmh\" (UID: \"667a7db5-cf73-424c-9d18-53bfbc5f65db\") " pod="openshift-dns/node-resolver-hkkmh" Nov 01 15:47:47.881150 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880688 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-host\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:47.881150 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880716 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c7838987-12dc-47bc-9b39-a87ffd11ddaf-kubelet-config\") pod \"global-pull-secret-syncer-ckz9h\" (UID: \"c7838987-12dc-47bc-9b39-a87ffd11ddaf\") " pod="kube-system/global-pull-secret-syncer-ckz9h" Nov 01 15:47:47.881150 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880738 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-host-cni-netd\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.881150 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880745 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-host\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:47.881150 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880764 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8e87fe52-8f65-43d9-a04a-a86d6227230d-registration-dir\") pod \"aws-ebs-csi-driver-node-cgvz7\" (UID: \"8e87fe52-8f65-43d9-a04a-a86d6227230d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" Nov 01 15:47:47.881150 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880783 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c7838987-12dc-47bc-9b39-a87ffd11ddaf-kubelet-config\") pod \"global-pull-secret-syncer-ckz9h\" (UID: \"c7838987-12dc-47bc-9b39-a87ffd11ddaf\") " pod="kube-system/global-pull-secret-syncer-ckz9h" Nov 01 15:47:47.881150 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880789 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-host-cni-netd\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.881150 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880815 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8e87fe52-8f65-43d9-a04a-a86d6227230d-sys-fs\") pod \"aws-ebs-csi-driver-node-cgvz7\" (UID: \"8e87fe52-8f65-43d9-a04a-a86d6227230d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" Nov 01 15:47:47.881150 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880833 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b53051b0-243d-4e03-926a-1f83372e648f-host-slash\") pod \"iptables-alerter-86kzp\" (UID: \"b53051b0-243d-4e03-926a-1f83372e648f\") " pod="openshift-network-operator/iptables-alerter-86kzp" Nov 01 15:47:47.881150 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880838 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8e87fe52-8f65-43d9-a04a-a86d6227230d-registration-dir\") pod \"aws-ebs-csi-driver-node-cgvz7\" (UID: \"8e87fe52-8f65-43d9-a04a-a86d6227230d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" Nov 01 15:47:47.881150 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880847 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-run-ovn\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.881150 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880874 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-run-ovn\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.881150 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880877 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.881150 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880905 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b53051b0-243d-4e03-926a-1f83372e648f-host-slash\") pod \"iptables-alerter-86kzp\" (UID: \"b53051b0-243d-4e03-926a-1f83372e648f\") " pod="openshift-network-operator/iptables-alerter-86kzp" Nov 01 15:47:47.881150 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880913 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.881150 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880908 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8e87fe52-8f65-43d9-a04a-a86d6227230d-etc-selinux\") pod \"aws-ebs-csi-driver-node-cgvz7\" (UID: \"8e87fe52-8f65-43d9-a04a-a86d6227230d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" Nov 01 15:47:47.881150 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880938 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8e87fe52-8f65-43d9-a04a-a86d6227230d-etc-selinux\") pod \"aws-ebs-csi-driver-node-cgvz7\" (UID: \"8e87fe52-8f65-43d9-a04a-a86d6227230d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" Nov 01 15:47:47.881150 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880905 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8e87fe52-8f65-43d9-a04a-a86d6227230d-sys-fs\") pod \"aws-ebs-csi-driver-node-cgvz7\" (UID: \"8e87fe52-8f65-43d9-a04a-a86d6227230d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" Nov 01 15:47:47.881150 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880947 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-multus-cni-dir\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.881150 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880973 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-etc-sysctl-conf\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:47.881150 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.880995 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-multus-cni-dir\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.881150 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881004 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c7838987-12dc-47bc-9b39-a87ffd11ddaf-dbus\") pod \"global-pull-secret-syncer-ckz9h\" (UID: \"c7838987-12dc-47bc-9b39-a87ffd11ddaf\") " pod="kube-system/global-pull-secret-syncer-ckz9h" Nov 01 15:47:47.881150 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881036 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-host-run-netns\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.881150 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881053 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-node-log\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.881150 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881063 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c7838987-12dc-47bc-9b39-a87ffd11ddaf-dbus\") pod \"global-pull-secret-syncer-ckz9h\" (UID: \"c7838987-12dc-47bc-9b39-a87ffd11ddaf\") " pod="kube-system/global-pull-secret-syncer-ckz9h" Nov 01 15:47:47.881150 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881071 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-etc-sysctl-conf\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:47.881150 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881079 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-host-run-ovn-kubernetes\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.881150 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881094 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-host-run-netns\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.881150 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881103 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-run\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:47.881150 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881107 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-node-log\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.881150 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881117 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-host-run-ovn-kubernetes\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.881150 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881135 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-var-lib-kubelet\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:47.881150 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881143 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-run\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:47.881150 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881156 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-host-var-lib-cni-multus\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.881150 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881173 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b8758dd-149c-4c37-9d32-a93fc953e548-system-cni-dir\") pod \"multus-additional-cni-plugins-f6klc\" (UID: \"4b8758dd-149c-4c37-9d32-a93fc953e548\") " pod="openshift-multus/multus-additional-cni-plugins-f6klc" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881190 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-etc-kubernetes\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881197 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-var-lib-kubelet\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881224 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-host-var-lib-cni-multus\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881213 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-etc-sysconfig\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881269 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-etc-systemd\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881242 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b8758dd-149c-4c37-9d32-a93fc953e548-system-cni-dir\") pod \"multus-additional-cni-plugins-f6klc\" (UID: \"4b8758dd-149c-4c37-9d32-a93fc953e548\") " pod="openshift-multus/multus-additional-cni-plugins-f6klc" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881291 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-etc-sysconfig\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881325 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-etc-systemd\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881235 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-etc-kubernetes\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881655 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8e87fe52-8f65-43d9-a04a-a86d6227230d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cgvz7\" (UID: \"8e87fe52-8f65-43d9-a04a-a86d6227230d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881678 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-host-run-netns\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881696 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b8758dd-149c-4c37-9d32-a93fc953e548-cnibin\") pod \"multus-additional-cni-plugins-f6klc\" (UID: \"4b8758dd-149c-4c37-9d32-a93fc953e548\") " pod="openshift-multus/multus-additional-cni-plugins-f6klc" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881702 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8e87fe52-8f65-43d9-a04a-a86d6227230d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cgvz7\" (UID: \"8e87fe52-8f65-43d9-a04a-a86d6227230d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881711 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-host-var-lib-kubelet\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881731 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-multus-conf-dir\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881766 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b8758dd-149c-4c37-9d32-a93fc953e548-cnibin\") pod \"multus-additional-cni-plugins-f6klc\" (UID: \"4b8758dd-149c-4c37-9d32-a93fc953e548\") " pod="openshift-multus/multus-additional-cni-plugins-f6klc" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881771 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-multus-conf-dir\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881766 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-host-kubelet\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881797 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-host-kubelet\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881806 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-systemd-units\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881784 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-host-var-lib-kubelet\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881773 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-host-run-netns\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881833 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-host-slash\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881855 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-systemd-units\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881858 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-var-lib-openvswitch\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881858 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-host-slash\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881890 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-host-var-lib-cni-bin\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881897 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-var-lib-openvswitch\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881915 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8e87fe52-8f65-43d9-a04a-a86d6227230d-device-dir\") pod \"aws-ebs-csi-driver-node-cgvz7\" (UID: \"8e87fe52-8f65-43d9-a04a-a86d6227230d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881935 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-run-systemd\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881940 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-host-var-lib-cni-bin\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881950 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-etc-openvswitch\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881972 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8e87fe52-8f65-43d9-a04a-a86d6227230d-device-dir\") pod \"aws-ebs-csi-driver-node-cgvz7\" (UID: \"8e87fe52-8f65-43d9-a04a-a86d6227230d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgvz7" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881970 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-etc-modprobe-d\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881980 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-run-systemd\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.881994 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/56e8fbab-06d3-4421-9995-822247d94d90-host\") pod \"node-ca-44klx\" (UID: \"56e8fbab-06d3-4421-9995-822247d94d90\") " pod="openshift-image-registry/node-ca-44klx" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.882009 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8254aafa-dd5b-4150-87e4-d165dab6bcb5-etc-openvswitch\") pod \"ovnkube-node-jtwhw\" (UID: \"8254aafa-dd5b-4150-87e4-d165dab6bcb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.882012 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-multus-socket-dir-parent\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.882032 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-etc-modprobe-d\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.882063 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/09f28a75-b66e-4933-84e2-eb0941e5f6a3-multus-socket-dir-parent\") pod \"multus-x2wlk\" (UID: \"09f28a75-b66e-4933-84e2-eb0941e5f6a3\") " pod="openshift-multus/multus-x2wlk" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.882066 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/56e8fbab-06d3-4421-9995-822247d94d90-host\") pod \"node-ca-44klx\" (UID: \"56e8fbab-06d3-4421-9995-822247d94d90\") " pod="openshift-image-registry/node-ca-44klx" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.882054 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b8758dd-149c-4c37-9d32-a93fc953e548-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f6klc\" (UID: \"4b8758dd-149c-4c37-9d32-a93fc953e548\") " pod="openshift-multus/multus-additional-cni-plugins-f6klc" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.882106 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-etc-sysctl-d\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.882150 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b8758dd-149c-4c37-9d32-a93fc953e548-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f6klc\" (UID: \"4b8758dd-149c-4c37-9d32-a93fc953e548\") " pod="openshift-multus/multus-additional-cni-plugins-f6klc" Nov 01 15:47:47.882586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:47.882189 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/60e635cd-c1f3-4d34-b795-2f6b16438f8f-etc-sysctl-d\") pod \"tuned-92m6d\" (UID: \"60e635cd-c1f3-4d34-b795-2f6b16438f8f\") " pod="openshift-cluster-node-tuning-operator/tuned-92m6d" Nov 01 15:47:48.671037 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:48.670945 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:56.100067 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.100038 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:47:56.100552 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.100225 4363 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 15:47:56.220270 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.220236 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-l26rw" Nov 01 15:47:56.764297 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.763597 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vw5gf"] Nov 01 15:47:56.767789 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.767765 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lsfq4"] Nov 01 15:47:56.767910 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.767880 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vw5gf" Nov 01 15:47:56.769573 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.769552 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-d9spc"] Nov 01 15:47:56.769710 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.769673 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lsfq4" Nov 01 15:47:56.770993 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.770978 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d9spc" Nov 01 15:47:56.774616 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.774593 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 15:47:56.774730 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.774655 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 15:47:56.775376 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.775354 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 15:47:56.775451 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.775361 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mtfck\"" Nov 01 15:47:56.775515 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.775461 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 15:47:56.777380 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.777359 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 01 15:47:56.777485 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.777435 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-g2z52\"" Nov 01 15:47:56.777542 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.777507 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 15:47:56.777633 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.777618 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 15:47:56.778988 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.778965 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wl7pm\"" Nov 01 15:47:56.780340 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.779645 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vw5gf"] Nov 01 15:47:56.782861 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.781781 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lsfq4"] Nov 01 15:47:56.793476 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.793458 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-sjjtf"] Nov 01 15:47:56.794784 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.794769 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sjjtf" Nov 01 15:47:56.796192 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.796171 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d9spc"] Nov 01 15:47:56.806177 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.806157 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 15:47:56.806177 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.806157 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 15:47:56.806818 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.806801 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sjjtf"] Nov 01 15:47:56.825601 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.825577 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c51a9296-d37f-4161-936d-c1de0a2c2c82-config-volume\") pod \"dns-default-d9spc\" (UID: \"c51a9296-d37f-4161-936d-c1de0a2c2c82\") " pod="openshift-dns/dns-default-d9spc" Nov 01 15:47:56.825700 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.825608 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/32a5cc8c-047c-4616-a273-2bd712bef330-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-lsfq4\" (UID: \"32a5cc8c-047c-4616-a273-2bd712bef330\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lsfq4" Nov 01 15:47:56.825700 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.825628 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c51a9296-d37f-4161-936d-c1de0a2c2c82-metrics-tls\") pod \"dns-default-d9spc\" (UID: \"c51a9296-d37f-4161-936d-c1de0a2c2c82\") " pod="openshift-dns/dns-default-d9spc" Nov 01 15:47:56.825700 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.825642 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c51a9296-d37f-4161-936d-c1de0a2c2c82-tmp-dir\") pod \"dns-default-d9spc\" (UID: \"c51a9296-d37f-4161-936d-c1de0a2c2c82\") " pod="openshift-dns/dns-default-d9spc" Nov 01 15:47:56.825700 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.825660 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d0820c10-7071-4f70-82e6-fdbf2c0d6c76-crio-socket\") pod \"insights-runtime-extractor-vw5gf\" (UID: \"d0820c10-7071-4f70-82e6-fdbf2c0d6c76\") " pod="openshift-insights/insights-runtime-extractor-vw5gf" Nov 01 15:47:56.825887 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.825749 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2d417d8c-560b-4038-9971-76592795cc92-cert\") pod \"ingress-canary-sjjtf\" (UID: \"2d417d8c-560b-4038-9971-76592795cc92\") " pod="openshift-ingress-canary/ingress-canary-sjjtf" Nov 01 15:47:56.825887 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.825789 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d0820c10-7071-4f70-82e6-fdbf2c0d6c76-data-volume\") pod \"insights-runtime-extractor-vw5gf\" (UID: \"d0820c10-7071-4f70-82e6-fdbf2c0d6c76\") " pod="openshift-insights/insights-runtime-extractor-vw5gf" Nov 01 15:47:56.825887 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.825821 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0820c10-7071-4f70-82e6-fdbf2c0d6c76-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vw5gf\" (UID: \"d0820c10-7071-4f70-82e6-fdbf2c0d6c76\") " pod="openshift-insights/insights-runtime-extractor-vw5gf" Nov 01 15:47:56.825983 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.825921 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x7h8\" (UniqueName: \"kubernetes.io/projected/2d417d8c-560b-4038-9971-76592795cc92-kube-api-access-7x7h8\") pod \"ingress-canary-sjjtf\" (UID: \"2d417d8c-560b-4038-9971-76592795cc92\") " pod="openshift-ingress-canary/ingress-canary-sjjtf" Nov 01 15:47:56.826038 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.826004 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8k66\" (UniqueName: \"kubernetes.io/projected/c51a9296-d37f-4161-936d-c1de0a2c2c82-kube-api-access-h8k66\") pod \"dns-default-d9spc\" (UID: \"c51a9296-d37f-4161-936d-c1de0a2c2c82\") " pod="openshift-dns/dns-default-d9spc" Nov 01 15:47:56.826135 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.826043 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d0820c10-7071-4f70-82e6-fdbf2c0d6c76-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vw5gf\" (UID: \"d0820c10-7071-4f70-82e6-fdbf2c0d6c76\") " pod="openshift-insights/insights-runtime-extractor-vw5gf" Nov 01 15:47:56.826135 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.826069 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmd5f\" (UniqueName: \"kubernetes.io/projected/d0820c10-7071-4f70-82e6-fdbf2c0d6c76-kube-api-access-nmd5f\") pod \"insights-runtime-extractor-vw5gf\" (UID: \"d0820c10-7071-4f70-82e6-fdbf2c0d6c76\") " pod="openshift-insights/insights-runtime-extractor-vw5gf" Nov 01 15:47:56.866921 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.866895 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-85755df759-sswt8"] Nov 01 15:47:56.868629 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.868614 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85755df759-sswt8" Nov 01 15:47:56.875003 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.874983 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 01 15:47:56.875610 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.875591 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 01 15:47:56.875933 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.875879 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-srrqq\"" Nov 01 15:47:56.875933 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.875901 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 01 15:47:56.875933 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.875913 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 15:47:56.875933 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.875915 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-slghr\"" Nov 01 15:47:56.876124 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.875974 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 01 15:47:56.882352 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.882328 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85755df759-sswt8"] Nov 01 15:47:56.926514 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.926485 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ca3f5966-c817-45e8-b1f9-d5a8815ae2e1-registry-certificates\") pod \"image-registry-85755df759-sswt8\" (UID: \"ca3f5966-c817-45e8-b1f9-d5a8815ae2e1\") " pod="openshift-image-registry/image-registry-85755df759-sswt8" Nov 01 15:47:56.926514 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.926517 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ca3f5966-c817-45e8-b1f9-d5a8815ae2e1-bound-sa-token\") pod \"image-registry-85755df759-sswt8\" (UID: \"ca3f5966-c817-45e8-b1f9-d5a8815ae2e1\") " pod="openshift-image-registry/image-registry-85755df759-sswt8" Nov 01 15:47:56.926676 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.926535 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ca3f5966-c817-45e8-b1f9-d5a8815ae2e1-image-registry-private-configuration\") pod \"image-registry-85755df759-sswt8\" (UID: \"ca3f5966-c817-45e8-b1f9-d5a8815ae2e1\") " pod="openshift-image-registry/image-registry-85755df759-sswt8" Nov 01 15:47:56.926676 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.926552 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ca3f5966-c817-45e8-b1f9-d5a8815ae2e1-ca-trust-extracted\") pod \"image-registry-85755df759-sswt8\" (UID: \"ca3f5966-c817-45e8-b1f9-d5a8815ae2e1\") " pod="openshift-image-registry/image-registry-85755df759-sswt8" Nov 01 15:47:56.926676 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.926662 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c51a9296-d37f-4161-936d-c1de0a2c2c82-config-volume\") pod \"dns-default-d9spc\" (UID: \"c51a9296-d37f-4161-936d-c1de0a2c2c82\") " pod="openshift-dns/dns-default-d9spc" Nov 01 15:47:56.926776 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.926693 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ca3f5966-c817-45e8-b1f9-d5a8815ae2e1-registry-tls\") pod \"image-registry-85755df759-sswt8\" (UID: \"ca3f5966-c817-45e8-b1f9-d5a8815ae2e1\") " pod="openshift-image-registry/image-registry-85755df759-sswt8" Nov 01 15:47:56.926776 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.926715 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/32a5cc8c-047c-4616-a273-2bd712bef330-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-lsfq4\" (UID: \"32a5cc8c-047c-4616-a273-2bd712bef330\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lsfq4" Nov 01 15:47:56.926776 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.926754 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c51a9296-d37f-4161-936d-c1de0a2c2c82-metrics-tls\") pod \"dns-default-d9spc\" (UID: \"c51a9296-d37f-4161-936d-c1de0a2c2c82\") " pod="openshift-dns/dns-default-d9spc" Nov 01 15:47:56.926776 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.926771 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c51a9296-d37f-4161-936d-c1de0a2c2c82-tmp-dir\") pod \"dns-default-d9spc\" (UID: \"c51a9296-d37f-4161-936d-c1de0a2c2c82\") " pod="openshift-dns/dns-default-d9spc" Nov 01 15:47:56.926918 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.926787 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ca3f5966-c817-45e8-b1f9-d5a8815ae2e1-installation-pull-secrets\") pod \"image-registry-85755df759-sswt8\" (UID: \"ca3f5966-c817-45e8-b1f9-d5a8815ae2e1\") " pod="openshift-image-registry/image-registry-85755df759-sswt8" Nov 01 15:47:56.926918 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.926807 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d0820c10-7071-4f70-82e6-fdbf2c0d6c76-crio-socket\") pod \"insights-runtime-extractor-vw5gf\" (UID: \"d0820c10-7071-4f70-82e6-fdbf2c0d6c76\") " pod="openshift-insights/insights-runtime-extractor-vw5gf" Nov 01 15:47:56.926918 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.926831 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ca3f5966-c817-45e8-b1f9-d5a8815ae2e1-trusted-ca\") pod \"image-registry-85755df759-sswt8\" (UID: \"ca3f5966-c817-45e8-b1f9-d5a8815ae2e1\") " pod="openshift-image-registry/image-registry-85755df759-sswt8" Nov 01 15:47:56.926918 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.926880 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2d417d8c-560b-4038-9971-76592795cc92-cert\") pod \"ingress-canary-sjjtf\" (UID: \"2d417d8c-560b-4038-9971-76592795cc92\") " pod="openshift-ingress-canary/ingress-canary-sjjtf" Nov 01 15:47:56.926918 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.926909 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d0820c10-7071-4f70-82e6-fdbf2c0d6c76-crio-socket\") pod \"insights-runtime-extractor-vw5gf\" (UID: \"d0820c10-7071-4f70-82e6-fdbf2c0d6c76\") " pod="openshift-insights/insights-runtime-extractor-vw5gf" Nov 01 15:47:56.927117 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.926947 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d0820c10-7071-4f70-82e6-fdbf2c0d6c76-data-volume\") pod \"insights-runtime-extractor-vw5gf\" (UID: \"d0820c10-7071-4f70-82e6-fdbf2c0d6c76\") " pod="openshift-insights/insights-runtime-extractor-vw5gf" Nov 01 15:47:56.927117 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.927062 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0820c10-7071-4f70-82e6-fdbf2c0d6c76-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vw5gf\" (UID: \"d0820c10-7071-4f70-82e6-fdbf2c0d6c76\") " pod="openshift-insights/insights-runtime-extractor-vw5gf" Nov 01 15:47:56.927117 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.927090 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58ws8\" (UniqueName: \"kubernetes.io/projected/ca3f5966-c817-45e8-b1f9-d5a8815ae2e1-kube-api-access-58ws8\") pod \"image-registry-85755df759-sswt8\" (UID: \"ca3f5966-c817-45e8-b1f9-d5a8815ae2e1\") " pod="openshift-image-registry/image-registry-85755df759-sswt8" Nov 01 15:47:56.927244 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.927156 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x7h8\" (UniqueName: \"kubernetes.io/projected/2d417d8c-560b-4038-9971-76592795cc92-kube-api-access-7x7h8\") pod \"ingress-canary-sjjtf\" (UID: \"2d417d8c-560b-4038-9971-76592795cc92\") " pod="openshift-ingress-canary/ingress-canary-sjjtf" Nov 01 15:47:56.927244 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.927162 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c51a9296-d37f-4161-936d-c1de0a2c2c82-tmp-dir\") pod \"dns-default-d9spc\" (UID: \"c51a9296-d37f-4161-936d-c1de0a2c2c82\") " pod="openshift-dns/dns-default-d9spc" Nov 01 15:47:56.927244 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.927199 4363 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 15:47:56.927384 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.927246 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8k66\" (UniqueName: \"kubernetes.io/projected/c51a9296-d37f-4161-936d-c1de0a2c2c82-kube-api-access-h8k66\") pod \"dns-default-d9spc\" (UID: \"c51a9296-d37f-4161-936d-c1de0a2c2c82\") " pod="openshift-dns/dns-default-d9spc" Nov 01 15:47:56.927384 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.927276 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d0820c10-7071-4f70-82e6-fdbf2c0d6c76-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vw5gf\" (UID: \"d0820c10-7071-4f70-82e6-fdbf2c0d6c76\") " pod="openshift-insights/insights-runtime-extractor-vw5gf" Nov 01 15:47:56.927384 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.927301 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmd5f\" (UniqueName: \"kubernetes.io/projected/d0820c10-7071-4f70-82e6-fdbf2c0d6c76-kube-api-access-nmd5f\") pod \"insights-runtime-extractor-vw5gf\" (UID: \"d0820c10-7071-4f70-82e6-fdbf2c0d6c76\") " pod="openshift-insights/insights-runtime-extractor-vw5gf" Nov 01 15:47:56.927540 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.927410 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d0820c10-7071-4f70-82e6-fdbf2c0d6c76-data-volume\") pod \"insights-runtime-extractor-vw5gf\" (UID: \"d0820c10-7071-4f70-82e6-fdbf2c0d6c76\") " pod="openshift-insights/insights-runtime-extractor-vw5gf" Nov 01 15:47:56.927540 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.927528 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c51a9296-d37f-4161-936d-c1de0a2c2c82-config-volume\") pod \"dns-default-d9spc\" (UID: \"c51a9296-d37f-4161-936d-c1de0a2c2c82\") " pod="openshift-dns/dns-default-d9spc" Nov 01 15:47:56.928876 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.928851 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d0820c10-7071-4f70-82e6-fdbf2c0d6c76-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vw5gf\" (UID: \"d0820c10-7071-4f70-82e6-fdbf2c0d6c76\") " pod="openshift-insights/insights-runtime-extractor-vw5gf" Nov 01 15:47:56.929533 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.929503 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c51a9296-d37f-4161-936d-c1de0a2c2c82-metrics-tls\") pod \"dns-default-d9spc\" (UID: \"c51a9296-d37f-4161-936d-c1de0a2c2c82\") " pod="openshift-dns/dns-default-d9spc" Nov 01 15:47:56.929677 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.929660 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2d417d8c-560b-4038-9971-76592795cc92-cert\") pod \"ingress-canary-sjjtf\" (UID: \"2d417d8c-560b-4038-9971-76592795cc92\") " pod="openshift-ingress-canary/ingress-canary-sjjtf" Nov 01 15:47:56.929815 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.929797 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0820c10-7071-4f70-82e6-fdbf2c0d6c76-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vw5gf\" (UID: \"d0820c10-7071-4f70-82e6-fdbf2c0d6c76\") " pod="openshift-insights/insights-runtime-extractor-vw5gf" Nov 01 15:47:56.930091 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.930072 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/32a5cc8c-047c-4616-a273-2bd712bef330-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-lsfq4\" (UID: \"32a5cc8c-047c-4616-a273-2bd712bef330\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lsfq4" Nov 01 15:47:56.935932 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.935908 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x7h8\" (UniqueName: \"kubernetes.io/projected/2d417d8c-560b-4038-9971-76592795cc92-kube-api-access-7x7h8\") pod \"ingress-canary-sjjtf\" (UID: \"2d417d8c-560b-4038-9971-76592795cc92\") " pod="openshift-ingress-canary/ingress-canary-sjjtf" Nov 01 15:47:56.936104 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.935980 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8k66\" (UniqueName: \"kubernetes.io/projected/c51a9296-d37f-4161-936d-c1de0a2c2c82-kube-api-access-h8k66\") pod \"dns-default-d9spc\" (UID: \"c51a9296-d37f-4161-936d-c1de0a2c2c82\") " pod="openshift-dns/dns-default-d9spc" Nov 01 15:47:56.936273 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:56.936250 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmd5f\" (UniqueName: \"kubernetes.io/projected/d0820c10-7071-4f70-82e6-fdbf2c0d6c76-kube-api-access-nmd5f\") pod \"insights-runtime-extractor-vw5gf\" (UID: \"d0820c10-7071-4f70-82e6-fdbf2c0d6c76\") " pod="openshift-insights/insights-runtime-extractor-vw5gf" Nov 01 15:47:57.028310 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:57.028234 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58ws8\" (UniqueName: \"kubernetes.io/projected/ca3f5966-c817-45e8-b1f9-d5a8815ae2e1-kube-api-access-58ws8\") pod \"image-registry-85755df759-sswt8\" (UID: \"ca3f5966-c817-45e8-b1f9-d5a8815ae2e1\") " pod="openshift-image-registry/image-registry-85755df759-sswt8" Nov 01 15:47:57.028310 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:57.028287 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ca3f5966-c817-45e8-b1f9-d5a8815ae2e1-registry-certificates\") pod \"image-registry-85755df759-sswt8\" (UID: \"ca3f5966-c817-45e8-b1f9-d5a8815ae2e1\") " pod="openshift-image-registry/image-registry-85755df759-sswt8" Nov 01 15:47:57.028498 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:57.028309 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ca3f5966-c817-45e8-b1f9-d5a8815ae2e1-bound-sa-token\") pod \"image-registry-85755df759-sswt8\" (UID: \"ca3f5966-c817-45e8-b1f9-d5a8815ae2e1\") " pod="openshift-image-registry/image-registry-85755df759-sswt8" Nov 01 15:47:57.028498 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:57.028334 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ca3f5966-c817-45e8-b1f9-d5a8815ae2e1-image-registry-private-configuration\") pod \"image-registry-85755df759-sswt8\" (UID: \"ca3f5966-c817-45e8-b1f9-d5a8815ae2e1\") " pod="openshift-image-registry/image-registry-85755df759-sswt8" Nov 01 15:47:57.028498 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:57.028355 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ca3f5966-c817-45e8-b1f9-d5a8815ae2e1-ca-trust-extracted\") pod \"image-registry-85755df759-sswt8\" (UID: \"ca3f5966-c817-45e8-b1f9-d5a8815ae2e1\") " pod="openshift-image-registry/image-registry-85755df759-sswt8" Nov 01 15:47:57.028498 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:57.028386 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ca3f5966-c817-45e8-b1f9-d5a8815ae2e1-registry-tls\") pod \"image-registry-85755df759-sswt8\" (UID: \"ca3f5966-c817-45e8-b1f9-d5a8815ae2e1\") " pod="openshift-image-registry/image-registry-85755df759-sswt8" Nov 01 15:47:57.028498 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:57.028421 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ca3f5966-c817-45e8-b1f9-d5a8815ae2e1-installation-pull-secrets\") pod \"image-registry-85755df759-sswt8\" (UID: \"ca3f5966-c817-45e8-b1f9-d5a8815ae2e1\") " pod="openshift-image-registry/image-registry-85755df759-sswt8" Nov 01 15:47:57.028498 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:57.028449 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ca3f5966-c817-45e8-b1f9-d5a8815ae2e1-trusted-ca\") pod \"image-registry-85755df759-sswt8\" (UID: \"ca3f5966-c817-45e8-b1f9-d5a8815ae2e1\") " pod="openshift-image-registry/image-registry-85755df759-sswt8" Nov 01 15:47:57.028857 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:57.028835 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ca3f5966-c817-45e8-b1f9-d5a8815ae2e1-ca-trust-extracted\") pod \"image-registry-85755df759-sswt8\" (UID: \"ca3f5966-c817-45e8-b1f9-d5a8815ae2e1\") " pod="openshift-image-registry/image-registry-85755df759-sswt8" Nov 01 15:47:57.029256 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:57.029236 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ca3f5966-c817-45e8-b1f9-d5a8815ae2e1-registry-certificates\") pod \"image-registry-85755df759-sswt8\" (UID: \"ca3f5966-c817-45e8-b1f9-d5a8815ae2e1\") " pod="openshift-image-registry/image-registry-85755df759-sswt8" Nov 01 15:47:57.029349 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:57.029297 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ca3f5966-c817-45e8-b1f9-d5a8815ae2e1-trusted-ca\") pod \"image-registry-85755df759-sswt8\" (UID: \"ca3f5966-c817-45e8-b1f9-d5a8815ae2e1\") " pod="openshift-image-registry/image-registry-85755df759-sswt8" Nov 01 15:47:57.030769 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:57.030750 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ca3f5966-c817-45e8-b1f9-d5a8815ae2e1-registry-tls\") pod \"image-registry-85755df759-sswt8\" (UID: \"ca3f5966-c817-45e8-b1f9-d5a8815ae2e1\") " pod="openshift-image-registry/image-registry-85755df759-sswt8" Nov 01 15:47:57.031091 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:57.031073 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ca3f5966-c817-45e8-b1f9-d5a8815ae2e1-image-registry-private-configuration\") pod \"image-registry-85755df759-sswt8\" (UID: \"ca3f5966-c817-45e8-b1f9-d5a8815ae2e1\") " pod="openshift-image-registry/image-registry-85755df759-sswt8" Nov 01 15:47:57.031213 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:57.031194 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ca3f5966-c817-45e8-b1f9-d5a8815ae2e1-installation-pull-secrets\") pod \"image-registry-85755df759-sswt8\" (UID: \"ca3f5966-c817-45e8-b1f9-d5a8815ae2e1\") " pod="openshift-image-registry/image-registry-85755df759-sswt8" Nov 01 15:47:57.036031 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:57.035997 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58ws8\" (UniqueName: \"kubernetes.io/projected/ca3f5966-c817-45e8-b1f9-d5a8815ae2e1-kube-api-access-58ws8\") pod \"image-registry-85755df759-sswt8\" (UID: \"ca3f5966-c817-45e8-b1f9-d5a8815ae2e1\") " pod="openshift-image-registry/image-registry-85755df759-sswt8" Nov 01 15:47:57.036175 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:57.036156 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ca3f5966-c817-45e8-b1f9-d5a8815ae2e1-bound-sa-token\") pod \"image-registry-85755df759-sswt8\" (UID: \"ca3f5966-c817-45e8-b1f9-d5a8815ae2e1\") " pod="openshift-image-registry/image-registry-85755df759-sswt8" Nov 01 15:47:57.088428 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:57.088403 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vw5gf" Nov 01 15:47:57.090555 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:57.090532 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lsfq4" Nov 01 15:47:57.095571 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:57.095550 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d9spc" Nov 01 15:47:57.104507 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:57.104488 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sjjtf" Nov 01 15:47:57.178370 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:57.178347 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85755df759-sswt8" Nov 01 15:47:57.371078 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:57.371045 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lsfq4"] Nov 01 15:47:57.373829 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:57.373801 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vw5gf"] Nov 01 15:47:57.374938 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:57.374898 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32a5cc8c_047c_4616_a273_2bd712bef330.slice/crio-a40c3cfee4b4f70a071689a8a9464c644d61930e5c2b25e15c05967293e5bbd4 WatchSource:0}: Error finding container a40c3cfee4b4f70a071689a8a9464c644d61930e5c2b25e15c05967293e5bbd4: Status 404 returned error can't find the container with id a40c3cfee4b4f70a071689a8a9464c644d61930e5c2b25e15c05967293e5bbd4 Nov 01 15:47:57.377139 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:57.377116 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0820c10_7071_4f70_82e6_fdbf2c0d6c76.slice/crio-66c701b0ee86af90eebbcf126e28033c66344862a9898ef5a05f5d4711838e1a WatchSource:0}: Error finding container 66c701b0ee86af90eebbcf126e28033c66344862a9898ef5a05f5d4711838e1a: Status 404 returned error can't find the container with id 66c701b0ee86af90eebbcf126e28033c66344862a9898ef5a05f5d4711838e1a Nov 01 15:47:57.377227 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:57.377190 4363 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 15:47:57.379520 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:57.379498 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d9spc"] Nov 01 15:47:57.382084 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:57.382062 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc51a9296_d37f_4161_936d_c1de0a2c2c82.slice/crio-1a8a43d14d4073a14dd9cee4fd334605cefb17b1930695b6d78c3fb7f72d9010 WatchSource:0}: Error finding container 1a8a43d14d4073a14dd9cee4fd334605cefb17b1930695b6d78c3fb7f72d9010: Status 404 returned error can't find the container with id 1a8a43d14d4073a14dd9cee4fd334605cefb17b1930695b6d78c3fb7f72d9010 Nov 01 15:47:57.389606 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:57.389573 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sjjtf"] Nov 01 15:47:57.392765 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:57.392747 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d417d8c_560b_4038_9971_76592795cc92.slice/crio-9820e8ea94d7bed232fa8882a6b8e6c2c73feda6786e64c2d6d924074bb51f61 WatchSource:0}: Error finding container 9820e8ea94d7bed232fa8882a6b8e6c2c73feda6786e64c2d6d924074bb51f61: Status 404 returned error can't find the container with id 9820e8ea94d7bed232fa8882a6b8e6c2c73feda6786e64c2d6d924074bb51f61 Nov 01 15:47:57.452368 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:57.452268 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85755df759-sswt8"] Nov 01 15:47:57.457235 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:47:57.457206 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca3f5966_c817_45e8_b1f9_d5a8815ae2e1.slice/crio-8c971d2785ffa5a77b696fd99a9ff6f1d5fa18cdbafc5413415cf9e1e7eecee6 WatchSource:0}: Error finding container 8c971d2785ffa5a77b696fd99a9ff6f1d5fa18cdbafc5413415cf9e1e7eecee6: Status 404 returned error can't find the container with id 8c971d2785ffa5a77b696fd99a9ff6f1d5fa18cdbafc5413415cf9e1e7eecee6 Nov 01 15:47:57.847915 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:57.847873 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sjjtf" event={"ID":"2d417d8c-560b-4038-9971-76592795cc92","Type":"ContainerStarted","Data":"9820e8ea94d7bed232fa8882a6b8e6c2c73feda6786e64c2d6d924074bb51f61"} Nov 01 15:47:57.849078 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:57.849049 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d9spc" event={"ID":"c51a9296-d37f-4161-936d-c1de0a2c2c82","Type":"ContainerStarted","Data":"1a8a43d14d4073a14dd9cee4fd334605cefb17b1930695b6d78c3fb7f72d9010"} Nov 01 15:47:57.850962 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:57.850932 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lsfq4" event={"ID":"32a5cc8c-047c-4616-a273-2bd712bef330","Type":"ContainerStarted","Data":"a40c3cfee4b4f70a071689a8a9464c644d61930e5c2b25e15c05967293e5bbd4"} Nov 01 15:47:57.852786 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:57.852753 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vw5gf" event={"ID":"d0820c10-7071-4f70-82e6-fdbf2c0d6c76","Type":"ContainerStarted","Data":"27bec899a2613e6813ba87ff4090416c38a1f1deb93fcca6ccb7ee66a116b8fc"} Nov 01 15:47:57.852889 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:57.852808 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vw5gf" event={"ID":"d0820c10-7071-4f70-82e6-fdbf2c0d6c76","Type":"ContainerStarted","Data":"66c701b0ee86af90eebbcf126e28033c66344862a9898ef5a05f5d4711838e1a"} Nov 01 15:47:57.854465 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:57.854440 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85755df759-sswt8" event={"ID":"ca3f5966-c817-45e8-b1f9-d5a8815ae2e1","Type":"ContainerStarted","Data":"d72d05cbe97c8ea17b801a2e54d51d93f02c9e98c75d873844f317a905cbcc64"} Nov 01 15:47:57.854561 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:57.854483 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85755df759-sswt8" event={"ID":"ca3f5966-c817-45e8-b1f9-d5a8815ae2e1","Type":"ContainerStarted","Data":"8c971d2785ffa5a77b696fd99a9ff6f1d5fa18cdbafc5413415cf9e1e7eecee6"} Nov 01 15:47:57.854561 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:57.854534 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-85755df759-sswt8" Nov 01 15:47:57.871336 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:57.871297 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-85755df759-sswt8" podStartSLOduration=1.871286672 podStartE2EDuration="1.871286672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:47:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:57.871113388 +0000 UTC m=+11.202851038" watchObservedRunningTime="2026-11-01 15:47:57.871286672 +0000 UTC m=+11.203024312" Nov 01 15:47:58.862712 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:58.862680 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lsfq4" event={"ID":"32a5cc8c-047c-4616-a273-2bd712bef330","Type":"ContainerStarted","Data":"43c5a3f39dd9cfff8ce2b73e13c322d9f45d6b73da575f7ea6a336bb8fb430d0"} Nov 01 15:47:58.864133 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:58.864106 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lsfq4" Nov 01 15:47:58.868204 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:58.868180 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vw5gf" event={"ID":"d0820c10-7071-4f70-82e6-fdbf2c0d6c76","Type":"ContainerStarted","Data":"37078074d0474e454944f5e2d6ab6b631b596137c31b6ad2162164a1dd8b553d"} Nov 01 15:47:58.869831 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:58.869809 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lsfq4" Nov 01 15:47:58.877197 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:58.877088 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lsfq4" podStartSLOduration=1.684288702 podStartE2EDuration="2.877068948s" totalImagesPullingTime="1.192780246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:47:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:58.876099383 +0000 UTC m=+12.207837033" watchObservedRunningTime="2026-11-01 15:47:58.877068948 +0000 UTC m=+12.208806599" Nov 01 15:47:59.874132 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:59.874099 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d9spc" event={"ID":"c51a9296-d37f-4161-936d-c1de0a2c2c82","Type":"ContainerStarted","Data":"d1b5a5ddb05c3423c0d200aacd68982894260dc430c40d663abedfb1c0c44a4f"} Nov 01 15:47:59.874463 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:59.874212 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-d9spc" Nov 01 15:47:59.874463 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:59.874242 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d9spc" event={"ID":"c51a9296-d37f-4161-936d-c1de0a2c2c82","Type":"ContainerStarted","Data":"58ada7473e0092e4ebf858f0a8457c61fc4d3eab0a432de02a9fbb54e69f5e83"} Nov 01 15:47:59.890373 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:47:59.890321 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-d9spc" podStartSLOduration=1.7889244629999999 podStartE2EDuration="3.890299861s" totalImagesPullingTime="2.101375398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:47:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:59.889975288 +0000 UTC m=+13.221712938" watchObservedRunningTime="2026-11-01 15:47:59.890299861 +0000 UTC m=+13.222037514" Nov 01 15:48:00.037057 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:00.036532 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-qt7lp"] Nov 01 15:48:00.039761 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:00.039733 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qt7lp" Nov 01 15:48:00.048974 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:00.048951 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 01 15:48:00.048974 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:00.048968 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-fsfz2\"" Nov 01 15:48:00.049158 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:00.048953 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 01 15:48:00.050770 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:00.050750 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 01 15:48:00.050770 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:00.050759 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 01 15:48:00.050959 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:00.050938 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 01 15:48:00.058858 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:00.058834 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-qt7lp"] Nov 01 15:48:00.160090 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:00.160053 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cc6b2feb-cc68-454c-9805-6db660e294b8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-qt7lp\" (UID: \"cc6b2feb-cc68-454c-9805-6db660e294b8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qt7lp" Nov 01 15:48:00.160290 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:00.160101 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc6b2feb-cc68-454c-9805-6db660e294b8-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-qt7lp\" (UID: \"cc6b2feb-cc68-454c-9805-6db660e294b8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qt7lp" Nov 01 15:48:00.160290 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:00.160180 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc6b2feb-cc68-454c-9805-6db660e294b8-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-qt7lp\" (UID: \"cc6b2feb-cc68-454c-9805-6db660e294b8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qt7lp" Nov 01 15:48:00.160290 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:00.160212 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pkk4\" (UniqueName: \"kubernetes.io/projected/cc6b2feb-cc68-454c-9805-6db660e294b8-kube-api-access-8pkk4\") pod \"prometheus-operator-9cd89b74c-qt7lp\" (UID: \"cc6b2feb-cc68-454c-9805-6db660e294b8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qt7lp" Nov 01 15:48:00.261408 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:00.261383 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cc6b2feb-cc68-454c-9805-6db660e294b8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-qt7lp\" (UID: \"cc6b2feb-cc68-454c-9805-6db660e294b8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qt7lp" Nov 01 15:48:00.261492 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:00.261440 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc6b2feb-cc68-454c-9805-6db660e294b8-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-qt7lp\" (UID: \"cc6b2feb-cc68-454c-9805-6db660e294b8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qt7lp" Nov 01 15:48:00.261492 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:00.261482 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc6b2feb-cc68-454c-9805-6db660e294b8-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-qt7lp\" (UID: \"cc6b2feb-cc68-454c-9805-6db660e294b8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qt7lp" Nov 01 15:48:00.261586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:00.261511 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pkk4\" (UniqueName: \"kubernetes.io/projected/cc6b2feb-cc68-454c-9805-6db660e294b8-kube-api-access-8pkk4\") pod \"prometheus-operator-9cd89b74c-qt7lp\" (UID: \"cc6b2feb-cc68-454c-9805-6db660e294b8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qt7lp" Nov 01 15:48:00.262285 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:00.262257 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc6b2feb-cc68-454c-9805-6db660e294b8-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-qt7lp\" (UID: \"cc6b2feb-cc68-454c-9805-6db660e294b8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qt7lp" Nov 01 15:48:00.264040 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:00.264003 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc6b2feb-cc68-454c-9805-6db660e294b8-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-qt7lp\" (UID: \"cc6b2feb-cc68-454c-9805-6db660e294b8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qt7lp" Nov 01 15:48:00.264142 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:00.264104 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cc6b2feb-cc68-454c-9805-6db660e294b8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-qt7lp\" (UID: \"cc6b2feb-cc68-454c-9805-6db660e294b8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qt7lp" Nov 01 15:48:00.269567 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:00.269549 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pkk4\" (UniqueName: \"kubernetes.io/projected/cc6b2feb-cc68-454c-9805-6db660e294b8-kube-api-access-8pkk4\") pod \"prometheus-operator-9cd89b74c-qt7lp\" (UID: \"cc6b2feb-cc68-454c-9805-6db660e294b8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qt7lp" Nov 01 15:48:00.355655 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:00.355619 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qt7lp" Nov 01 15:48:00.628092 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:00.628056 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-qt7lp"] Nov 01 15:48:00.631871 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:48:00.631844 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6b2feb_cc68_454c_9805_6db660e294b8.slice/crio-487dbe60fbff87573a7aed4df55e0ea1e61e3991b5515fe2872b56b2d49ed3eb WatchSource:0}: Error finding container 487dbe60fbff87573a7aed4df55e0ea1e61e3991b5515fe2872b56b2d49ed3eb: Status 404 returned error can't find the container with id 487dbe60fbff87573a7aed4df55e0ea1e61e3991b5515fe2872b56b2d49ed3eb Nov 01 15:48:00.878457 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:00.878377 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sjjtf" event={"ID":"2d417d8c-560b-4038-9971-76592795cc92","Type":"ContainerStarted","Data":"fb09f0ed3659c2480ddc8b171319af5c723ca3792f1e93720afa795fef134076"} Nov 01 15:48:00.880881 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:00.880852 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vw5gf" event={"ID":"d0820c10-7071-4f70-82e6-fdbf2c0d6c76","Type":"ContainerStarted","Data":"7c31ca89db024c449b7a04510baea8e0cb8c48f5db975b5c614f0aaed8abb912"} Nov 01 15:48:00.882058 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:00.882035 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qt7lp" event={"ID":"cc6b2feb-cc68-454c-9805-6db660e294b8","Type":"ContainerStarted","Data":"487dbe60fbff87573a7aed4df55e0ea1e61e3991b5515fe2872b56b2d49ed3eb"} Nov 01 15:48:00.893600 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:00.893452 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-sjjtf" podStartSLOduration=2.039861015 podStartE2EDuration="4.89344036s" totalImagesPullingTime="2.853579345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:47:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:48:00.892361591 +0000 UTC m=+14.224099241" watchObservedRunningTime="2026-11-01 15:48:00.89344036 +0000 UTC m=+14.225178008" Nov 01 15:48:00.909246 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:00.909079 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vw5gf" podStartSLOduration=2.575889634 podStartE2EDuration="4.909065086s" totalImagesPullingTime="2.333175452s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 15:47:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:48:00.908599367 +0000 UTC m=+14.240336998" watchObservedRunningTime="2026-11-01 15:48:00.909065086 +0000 UTC m=+14.240802749" Nov 01 15:48:02.890310 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:02.890264 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qt7lp" event={"ID":"cc6b2feb-cc68-454c-9805-6db660e294b8","Type":"ContainerStarted","Data":"e67800e4ba915c1c49f6ba771194756db115ac73271a5f127f04b06240855865"} Nov 01 15:48:02.890720 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:02.890347 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qt7lp" event={"ID":"cc6b2feb-cc68-454c-9805-6db660e294b8","Type":"ContainerStarted","Data":"5cc34e7a086424dce9c941cd747885710895d7eb983d74ce89a1b0ab9f1b16b2"} Nov 01 15:48:02.904780 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:02.904731 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qt7lp" podStartSLOduration=1.6974425979999999 podStartE2EDuration="2.904714754s" totalImagesPullingTime="1.207272156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:48:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:48:02.903604712 +0000 UTC m=+16.235342363" watchObservedRunningTime="2026-11-01 15:48:02.904714754 +0000 UTC m=+16.236452406" Nov 01 15:48:04.352972 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.352938 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-jqtqm"] Nov 01 15:48:04.357584 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.357561 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jqtqm" Nov 01 15:48:04.365235 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.365214 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-58bfr\"" Nov 01 15:48:04.365386 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.365236 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 01 15:48:04.365386 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.365239 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 01 15:48:04.365801 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.365779 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-jqtqm"] Nov 01 15:48:04.387584 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.386379 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pwkct"] Nov 01 15:48:04.390613 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.390591 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-8b544"] Nov 01 15:48:04.390738 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.390716 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pwkct" Nov 01 15:48:04.391211 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.391188 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjqr5\" (UniqueName: \"kubernetes.io/projected/457afc43-0d86-4348-9348-29b4fa1931af-kube-api-access-bjqr5\") pod \"openshift-state-metrics-7bd7489957-jqtqm\" (UID: \"457afc43-0d86-4348-9348-29b4fa1931af\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jqtqm" Nov 01 15:48:04.391331 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.391226 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/457afc43-0d86-4348-9348-29b4fa1931af-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-jqtqm\" (UID: \"457afc43-0d86-4348-9348-29b4fa1931af\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jqtqm" Nov 01 15:48:04.391331 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.391246 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/457afc43-0d86-4348-9348-29b4fa1931af-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-jqtqm\" (UID: \"457afc43-0d86-4348-9348-29b4fa1931af\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jqtqm" Nov 01 15:48:04.391451 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.391346 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/457afc43-0d86-4348-9348-29b4fa1931af-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-jqtqm\" (UID: \"457afc43-0d86-4348-9348-29b4fa1931af\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jqtqm" Nov 01 15:48:04.393671 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.393652 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8b544" Nov 01 15:48:04.399098 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.399075 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 01 15:48:04.405572 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.405545 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-8b544"] Nov 01 15:48:04.467005 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.466976 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 01 15:48:04.467169 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.467152 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-87knk\"" Nov 01 15:48:04.467231 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.467164 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 01 15:48:04.467267 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.467235 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-z6wzh\"" Nov 01 15:48:04.467267 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.467254 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 01 15:48:04.467438 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.467426 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 01 15:48:04.491666 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.491642 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9c69f277-d3e5-457d-a4f9-328d96b681d3-node-exporter-textfile\") pod \"node-exporter-pwkct\" (UID: \"9c69f277-d3e5-457d-a4f9-328d96b681d3\") " pod="openshift-monitoring/node-exporter-pwkct" Nov 01 15:48:04.491746 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.491673 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjqr5\" (UniqueName: \"kubernetes.io/projected/457afc43-0d86-4348-9348-29b4fa1931af-kube-api-access-bjqr5\") pod \"openshift-state-metrics-7bd7489957-jqtqm\" (UID: \"457afc43-0d86-4348-9348-29b4fa1931af\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jqtqm" Nov 01 15:48:04.491746 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.491704 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9c69f277-d3e5-457d-a4f9-328d96b681d3-root\") pod \"node-exporter-pwkct\" (UID: \"9c69f277-d3e5-457d-a4f9-328d96b681d3\") " pod="openshift-monitoring/node-exporter-pwkct" Nov 01 15:48:04.491821 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.491782 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c69f277-d3e5-457d-a4f9-328d96b681d3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pwkct\" (UID: \"9c69f277-d3e5-457d-a4f9-328d96b681d3\") " pod="openshift-monitoring/node-exporter-pwkct" Nov 01 15:48:04.491858 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.491820 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjc2w\" (UniqueName: \"kubernetes.io/projected/9c69f277-d3e5-457d-a4f9-328d96b681d3-kube-api-access-kjc2w\") pod \"node-exporter-pwkct\" (UID: \"9c69f277-d3e5-457d-a4f9-328d96b681d3\") " pod="openshift-monitoring/node-exporter-pwkct" Nov 01 15:48:04.491858 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.491849 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/457afc43-0d86-4348-9348-29b4fa1931af-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-jqtqm\" (UID: \"457afc43-0d86-4348-9348-29b4fa1931af\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jqtqm" Nov 01 15:48:04.491910 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.491867 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7adfe5b2-8d23-4150-a89d-d8725623a6c0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-8b544\" (UID: \"7adfe5b2-8d23-4150-a89d-d8725623a6c0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8b544" Nov 01 15:48:04.491910 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.491895 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9c69f277-d3e5-457d-a4f9-328d96b681d3-node-exporter-tls\") pod \"node-exporter-pwkct\" (UID: \"9c69f277-d3e5-457d-a4f9-328d96b681d3\") " pod="openshift-monitoring/node-exporter-pwkct" Nov 01 15:48:04.492046 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.492033 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/457afc43-0d86-4348-9348-29b4fa1931af-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-jqtqm\" (UID: \"457afc43-0d86-4348-9348-29b4fa1931af\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jqtqm" Nov 01 15:48:04.492095 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.492058 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7adfe5b2-8d23-4150-a89d-d8725623a6c0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-8b544\" (UID: \"7adfe5b2-8d23-4150-a89d-d8725623a6c0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8b544" Nov 01 15:48:04.492095 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.492076 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7adfe5b2-8d23-4150-a89d-d8725623a6c0-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-8b544\" (UID: \"7adfe5b2-8d23-4150-a89d-d8725623a6c0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8b544" Nov 01 15:48:04.492160 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.492103 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7adfe5b2-8d23-4150-a89d-d8725623a6c0-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-8b544\" (UID: \"7adfe5b2-8d23-4150-a89d-d8725623a6c0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8b544" Nov 01 15:48:04.492202 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.492145 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7adfe5b2-8d23-4150-a89d-d8725623a6c0-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-8b544\" (UID: \"7adfe5b2-8d23-4150-a89d-d8725623a6c0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8b544" Nov 01 15:48:04.492202 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.492192 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sttf8\" (UniqueName: \"kubernetes.io/projected/7adfe5b2-8d23-4150-a89d-d8725623a6c0-kube-api-access-sttf8\") pod \"kube-state-metrics-5dc8d7d89b-8b544\" (UID: \"7adfe5b2-8d23-4150-a89d-d8725623a6c0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8b544" Nov 01 15:48:04.492271 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.492232 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/457afc43-0d86-4348-9348-29b4fa1931af-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-jqtqm\" (UID: \"457afc43-0d86-4348-9348-29b4fa1931af\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jqtqm" Nov 01 15:48:04.492271 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.492255 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c69f277-d3e5-457d-a4f9-328d96b681d3-sys\") pod \"node-exporter-pwkct\" (UID: \"9c69f277-d3e5-457d-a4f9-328d96b681d3\") " pod="openshift-monitoring/node-exporter-pwkct" Nov 01 15:48:04.492348 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.492271 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9c69f277-d3e5-457d-a4f9-328d96b681d3-node-exporter-accelerators-collector-config\") pod \"node-exporter-pwkct\" (UID: \"9c69f277-d3e5-457d-a4f9-328d96b681d3\") " pod="openshift-monitoring/node-exporter-pwkct" Nov 01 15:48:04.492348 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.492291 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c69f277-d3e5-457d-a4f9-328d96b681d3-metrics-client-ca\") pod \"node-exporter-pwkct\" (UID: \"9c69f277-d3e5-457d-a4f9-328d96b681d3\") " pod="openshift-monitoring/node-exporter-pwkct" Nov 01 15:48:04.492348 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.492314 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9c69f277-d3e5-457d-a4f9-328d96b681d3-node-exporter-wtmp\") pod \"node-exporter-pwkct\" (UID: \"9c69f277-d3e5-457d-a4f9-328d96b681d3\") " pod="openshift-monitoring/node-exporter-pwkct" Nov 01 15:48:04.492539 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.492520 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/457afc43-0d86-4348-9348-29b4fa1931af-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-jqtqm\" (UID: \"457afc43-0d86-4348-9348-29b4fa1931af\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jqtqm" Nov 01 15:48:04.494425 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.494402 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/457afc43-0d86-4348-9348-29b4fa1931af-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-jqtqm\" (UID: \"457afc43-0d86-4348-9348-29b4fa1931af\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jqtqm" Nov 01 15:48:04.494566 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.494548 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/457afc43-0d86-4348-9348-29b4fa1931af-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-jqtqm\" (UID: \"457afc43-0d86-4348-9348-29b4fa1931af\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jqtqm" Nov 01 15:48:04.501736 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.501721 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 01 15:48:04.505591 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.505576 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjqr5\" (UniqueName: \"kubernetes.io/projected/457afc43-0d86-4348-9348-29b4fa1931af-kube-api-access-bjqr5\") pod \"openshift-state-metrics-7bd7489957-jqtqm\" (UID: \"457afc43-0d86-4348-9348-29b4fa1931af\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jqtqm" Nov 01 15:48:04.593670 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.593632 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7adfe5b2-8d23-4150-a89d-d8725623a6c0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-8b544\" (UID: \"7adfe5b2-8d23-4150-a89d-d8725623a6c0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8b544" Nov 01 15:48:04.593836 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.593680 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9c69f277-d3e5-457d-a4f9-328d96b681d3-node-exporter-tls\") pod \"node-exporter-pwkct\" (UID: \"9c69f277-d3e5-457d-a4f9-328d96b681d3\") " pod="openshift-monitoring/node-exporter-pwkct" Nov 01 15:48:04.593836 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.593715 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7adfe5b2-8d23-4150-a89d-d8725623a6c0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-8b544\" (UID: \"7adfe5b2-8d23-4150-a89d-d8725623a6c0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8b544" Nov 01 15:48:04.593836 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.593744 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7adfe5b2-8d23-4150-a89d-d8725623a6c0-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-8b544\" (UID: \"7adfe5b2-8d23-4150-a89d-d8725623a6c0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8b544" Nov 01 15:48:04.593836 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.593798 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7adfe5b2-8d23-4150-a89d-d8725623a6c0-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-8b544\" (UID: \"7adfe5b2-8d23-4150-a89d-d8725623a6c0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8b544" Nov 01 15:48:04.593836 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.593831 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7adfe5b2-8d23-4150-a89d-d8725623a6c0-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-8b544\" (UID: \"7adfe5b2-8d23-4150-a89d-d8725623a6c0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8b544" Nov 01 15:48:04.594116 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.593857 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sttf8\" (UniqueName: \"kubernetes.io/projected/7adfe5b2-8d23-4150-a89d-d8725623a6c0-kube-api-access-sttf8\") pod \"kube-state-metrics-5dc8d7d89b-8b544\" (UID: \"7adfe5b2-8d23-4150-a89d-d8725623a6c0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8b544" Nov 01 15:48:04.594116 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.593884 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c69f277-d3e5-457d-a4f9-328d96b681d3-sys\") pod \"node-exporter-pwkct\" (UID: \"9c69f277-d3e5-457d-a4f9-328d96b681d3\") " pod="openshift-monitoring/node-exporter-pwkct" Nov 01 15:48:04.594972 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.594232 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c69f277-d3e5-457d-a4f9-328d96b681d3-sys\") pod \"node-exporter-pwkct\" (UID: \"9c69f277-d3e5-457d-a4f9-328d96b681d3\") " pod="openshift-monitoring/node-exporter-pwkct" Nov 01 15:48:04.594972 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.594276 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9c69f277-d3e5-457d-a4f9-328d96b681d3-node-exporter-accelerators-collector-config\") pod \"node-exporter-pwkct\" (UID: \"9c69f277-d3e5-457d-a4f9-328d96b681d3\") " pod="openshift-monitoring/node-exporter-pwkct" Nov 01 15:48:04.594972 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.594324 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c69f277-d3e5-457d-a4f9-328d96b681d3-metrics-client-ca\") pod \"node-exporter-pwkct\" (UID: \"9c69f277-d3e5-457d-a4f9-328d96b681d3\") " pod="openshift-monitoring/node-exporter-pwkct" Nov 01 15:48:04.594972 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.594355 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9c69f277-d3e5-457d-a4f9-328d96b681d3-node-exporter-wtmp\") pod \"node-exporter-pwkct\" (UID: \"9c69f277-d3e5-457d-a4f9-328d96b681d3\") " pod="openshift-monitoring/node-exporter-pwkct" Nov 01 15:48:04.594972 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.594356 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7adfe5b2-8d23-4150-a89d-d8725623a6c0-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-8b544\" (UID: \"7adfe5b2-8d23-4150-a89d-d8725623a6c0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8b544" Nov 01 15:48:04.594972 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.594413 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9c69f277-d3e5-457d-a4f9-328d96b681d3-node-exporter-textfile\") pod \"node-exporter-pwkct\" (UID: \"9c69f277-d3e5-457d-a4f9-328d96b681d3\") " pod="openshift-monitoring/node-exporter-pwkct" Nov 01 15:48:04.594972 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.594460 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9c69f277-d3e5-457d-a4f9-328d96b681d3-root\") pod \"node-exporter-pwkct\" (UID: \"9c69f277-d3e5-457d-a4f9-328d96b681d3\") " pod="openshift-monitoring/node-exporter-pwkct" Nov 01 15:48:04.594972 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.594485 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c69f277-d3e5-457d-a4f9-328d96b681d3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pwkct\" (UID: \"9c69f277-d3e5-457d-a4f9-328d96b681d3\") " pod="openshift-monitoring/node-exporter-pwkct" Nov 01 15:48:04.594972 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.594511 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjc2w\" (UniqueName: \"kubernetes.io/projected/9c69f277-d3e5-457d-a4f9-328d96b681d3-kube-api-access-kjc2w\") pod \"node-exporter-pwkct\" (UID: \"9c69f277-d3e5-457d-a4f9-328d96b681d3\") " pod="openshift-monitoring/node-exporter-pwkct" Nov 01 15:48:04.594972 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.594673 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7adfe5b2-8d23-4150-a89d-d8725623a6c0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-8b544\" (UID: \"7adfe5b2-8d23-4150-a89d-d8725623a6c0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8b544" Nov 01 15:48:04.594972 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.594842 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9c69f277-d3e5-457d-a4f9-328d96b681d3-root\") pod \"node-exporter-pwkct\" (UID: \"9c69f277-d3e5-457d-a4f9-328d96b681d3\") " pod="openshift-monitoring/node-exporter-pwkct" Nov 01 15:48:04.594972 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.594881 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9c69f277-d3e5-457d-a4f9-328d96b681d3-node-exporter-accelerators-collector-config\") pod \"node-exporter-pwkct\" (UID: \"9c69f277-d3e5-457d-a4f9-328d96b681d3\") " pod="openshift-monitoring/node-exporter-pwkct" Nov 01 15:48:04.594972 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.594893 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c69f277-d3e5-457d-a4f9-328d96b681d3-metrics-client-ca\") pod \"node-exporter-pwkct\" (UID: \"9c69f277-d3e5-457d-a4f9-328d96b681d3\") " pod="openshift-monitoring/node-exporter-pwkct" Nov 01 15:48:04.594972 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.594924 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9c69f277-d3e5-457d-a4f9-328d96b681d3-node-exporter-textfile\") pod \"node-exporter-pwkct\" (UID: \"9c69f277-d3e5-457d-a4f9-328d96b681d3\") " pod="openshift-monitoring/node-exporter-pwkct" Nov 01 15:48:04.595603 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.595061 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9c69f277-d3e5-457d-a4f9-328d96b681d3-node-exporter-wtmp\") pod \"node-exporter-pwkct\" (UID: \"9c69f277-d3e5-457d-a4f9-328d96b681d3\") " pod="openshift-monitoring/node-exporter-pwkct" Nov 01 15:48:04.595603 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.595381 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7adfe5b2-8d23-4150-a89d-d8725623a6c0-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-8b544\" (UID: \"7adfe5b2-8d23-4150-a89d-d8725623a6c0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8b544" Nov 01 15:48:04.596302 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.596275 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7adfe5b2-8d23-4150-a89d-d8725623a6c0-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-8b544\" (UID: \"7adfe5b2-8d23-4150-a89d-d8725623a6c0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8b544" Nov 01 15:48:04.596395 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.596309 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9c69f277-d3e5-457d-a4f9-328d96b681d3-node-exporter-tls\") pod \"node-exporter-pwkct\" (UID: \"9c69f277-d3e5-457d-a4f9-328d96b681d3\") " pod="openshift-monitoring/node-exporter-pwkct" Nov 01 15:48:04.596395 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.596315 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7adfe5b2-8d23-4150-a89d-d8725623a6c0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-8b544\" (UID: \"7adfe5b2-8d23-4150-a89d-d8725623a6c0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8b544" Nov 01 15:48:04.596974 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.596953 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c69f277-d3e5-457d-a4f9-328d96b681d3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pwkct\" (UID: \"9c69f277-d3e5-457d-a4f9-328d96b681d3\") " pod="openshift-monitoring/node-exporter-pwkct" Nov 01 15:48:04.603570 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.603521 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjc2w\" (UniqueName: \"kubernetes.io/projected/9c69f277-d3e5-457d-a4f9-328d96b681d3-kube-api-access-kjc2w\") pod \"node-exporter-pwkct\" (UID: \"9c69f277-d3e5-457d-a4f9-328d96b681d3\") " pod="openshift-monitoring/node-exporter-pwkct" Nov 01 15:48:04.603646 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.603631 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sttf8\" (UniqueName: \"kubernetes.io/projected/7adfe5b2-8d23-4150-a89d-d8725623a6c0-kube-api-access-sttf8\") pod \"kube-state-metrics-5dc8d7d89b-8b544\" (UID: \"7adfe5b2-8d23-4150-a89d-d8725623a6c0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8b544" Nov 01 15:48:04.674564 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.674527 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jqtqm" Nov 01 15:48:04.707667 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.707629 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pwkct" Nov 01 15:48:04.715091 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.715062 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8b544" Nov 01 15:48:04.719095 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:48:04.719064 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c69f277_d3e5_457d_a4f9_328d96b681d3.slice/crio-3f3ef7b612ac2a0a3c83bd0dde663a9bc77ba2c74657dc481a350456ac57bab6 WatchSource:0}: Error finding container 3f3ef7b612ac2a0a3c83bd0dde663a9bc77ba2c74657dc481a350456ac57bab6: Status 404 returned error can't find the container with id 3f3ef7b612ac2a0a3c83bd0dde663a9bc77ba2c74657dc481a350456ac57bab6 Nov 01 15:48:04.897341 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.897254 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pwkct" event={"ID":"9c69f277-d3e5-457d-a4f9-328d96b681d3","Type":"ContainerStarted","Data":"3f3ef7b612ac2a0a3c83bd0dde663a9bc77ba2c74657dc481a350456ac57bab6"} Nov 01 15:48:04.953395 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.953362 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-jqtqm"] Nov 01 15:48:04.956646 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:48:04.956616 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod457afc43_0d86_4348_9348_29b4fa1931af.slice/crio-ef223891f90433af3f4e26dec10ca750acecce3b09888c7dfe881f2f165e2505 WatchSource:0}: Error finding container ef223891f90433af3f4e26dec10ca750acecce3b09888c7dfe881f2f165e2505: Status 404 returned error can't find the container with id ef223891f90433af3f4e26dec10ca750acecce3b09888c7dfe881f2f165e2505 Nov 01 15:48:04.993159 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:04.993134 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-8b544"] Nov 01 15:48:04.997958 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:48:04.997932 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7adfe5b2_8d23_4150_a89d_d8725623a6c0.slice/crio-d789202863a9b2e2488c2276d89168bd6e4a03ab271bde6f1affe30c4418431d WatchSource:0}: Error finding container d789202863a9b2e2488c2276d89168bd6e4a03ab271bde6f1affe30c4418431d: Status 404 returned error can't find the container with id d789202863a9b2e2488c2276d89168bd6e4a03ab271bde6f1affe30c4418431d Nov 01 15:48:05.902818 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:05.902780 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8b544" event={"ID":"7adfe5b2-8d23-4150-a89d-d8725623a6c0","Type":"ContainerStarted","Data":"d789202863a9b2e2488c2276d89168bd6e4a03ab271bde6f1affe30c4418431d"} Nov 01 15:48:05.905537 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:05.905503 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jqtqm" event={"ID":"457afc43-0d86-4348-9348-29b4fa1931af","Type":"ContainerStarted","Data":"c29ab58729f08a818977151ef4ea679a6169967ecc9cea6ca864f6b95edb16c0"} Nov 01 15:48:05.905660 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:05.905560 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jqtqm" event={"ID":"457afc43-0d86-4348-9348-29b4fa1931af","Type":"ContainerStarted","Data":"8fd66acc0ab61f48645d7dfa3716dfa143e661049ac87964275019271dda2f59"} Nov 01 15:48:05.905660 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:05.905579 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jqtqm" event={"ID":"457afc43-0d86-4348-9348-29b4fa1931af","Type":"ContainerStarted","Data":"ef223891f90433af3f4e26dec10ca750acecce3b09888c7dfe881f2f165e2505"} Nov 01 15:48:06.449270 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.449244 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-777b758c86-h4jsj"] Nov 01 15:48:06.453861 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.453842 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-777b758c86-h4jsj" Nov 01 15:48:06.463552 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.463478 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 01 15:48:06.463552 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.463527 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-fe2fcbp1jek22\"" Nov 01 15:48:06.463806 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.463629 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-bhhmt\"" Nov 01 15:48:06.463806 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.463652 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 01 15:48:06.464055 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.463997 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 01 15:48:06.464055 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.464032 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 01 15:48:06.464218 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.464136 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 01 15:48:06.465362 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.465338 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-777b758c86-h4jsj"] Nov 01 15:48:06.509106 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.509078 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/88d3d141-d4fa-4033-a3cb-372752fa2c67-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-777b758c86-h4jsj\" (UID: \"88d3d141-d4fa-4033-a3cb-372752fa2c67\") " pod="openshift-monitoring/thanos-querier-777b758c86-h4jsj" Nov 01 15:48:06.509210 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.509125 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42pzr\" (UniqueName: \"kubernetes.io/projected/88d3d141-d4fa-4033-a3cb-372752fa2c67-kube-api-access-42pzr\") pod \"thanos-querier-777b758c86-h4jsj\" (UID: \"88d3d141-d4fa-4033-a3cb-372752fa2c67\") " pod="openshift-monitoring/thanos-querier-777b758c86-h4jsj" Nov 01 15:48:06.509255 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.509235 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88d3d141-d4fa-4033-a3cb-372752fa2c67-metrics-client-ca\") pod \"thanos-querier-777b758c86-h4jsj\" (UID: \"88d3d141-d4fa-4033-a3cb-372752fa2c67\") " pod="openshift-monitoring/thanos-querier-777b758c86-h4jsj" Nov 01 15:48:06.509342 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.509273 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/88d3d141-d4fa-4033-a3cb-372752fa2c67-secret-thanos-querier-tls\") pod \"thanos-querier-777b758c86-h4jsj\" (UID: \"88d3d141-d4fa-4033-a3cb-372752fa2c67\") " pod="openshift-monitoring/thanos-querier-777b758c86-h4jsj" Nov 01 15:48:06.509403 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.509386 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88d3d141-d4fa-4033-a3cb-372752fa2c67-tmp\") pod \"thanos-querier-777b758c86-h4jsj\" (UID: \"88d3d141-d4fa-4033-a3cb-372752fa2c67\") " pod="openshift-monitoring/thanos-querier-777b758c86-h4jsj" Nov 01 15:48:06.509442 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.509426 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/88d3d141-d4fa-4033-a3cb-372752fa2c67-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-777b758c86-h4jsj\" (UID: \"88d3d141-d4fa-4033-a3cb-372752fa2c67\") " pod="openshift-monitoring/thanos-querier-777b758c86-h4jsj" Nov 01 15:48:06.509472 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.509449 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/88d3d141-d4fa-4033-a3cb-372752fa2c67-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-777b758c86-h4jsj\" (UID: \"88d3d141-d4fa-4033-a3cb-372752fa2c67\") " pod="openshift-monitoring/thanos-querier-777b758c86-h4jsj" Nov 01 15:48:06.509472 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.509466 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/88d3d141-d4fa-4033-a3cb-372752fa2c67-secret-grpc-tls\") pod \"thanos-querier-777b758c86-h4jsj\" (UID: \"88d3d141-d4fa-4033-a3cb-372752fa2c67\") " pod="openshift-monitoring/thanos-querier-777b758c86-h4jsj" Nov 01 15:48:06.509528 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.509483 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/88d3d141-d4fa-4033-a3cb-372752fa2c67-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-777b758c86-h4jsj\" (UID: \"88d3d141-d4fa-4033-a3cb-372752fa2c67\") " pod="openshift-monitoring/thanos-querier-777b758c86-h4jsj" Nov 01 15:48:06.610053 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.609955 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88d3d141-d4fa-4033-a3cb-372752fa2c67-metrics-client-ca\") pod \"thanos-querier-777b758c86-h4jsj\" (UID: \"88d3d141-d4fa-4033-a3cb-372752fa2c67\") " pod="openshift-monitoring/thanos-querier-777b758c86-h4jsj" Nov 01 15:48:06.610053 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.609999 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/88d3d141-d4fa-4033-a3cb-372752fa2c67-secret-thanos-querier-tls\") pod \"thanos-querier-777b758c86-h4jsj\" (UID: \"88d3d141-d4fa-4033-a3cb-372752fa2c67\") " pod="openshift-monitoring/thanos-querier-777b758c86-h4jsj" Nov 01 15:48:06.610189 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.610064 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88d3d141-d4fa-4033-a3cb-372752fa2c67-tmp\") pod \"thanos-querier-777b758c86-h4jsj\" (UID: \"88d3d141-d4fa-4033-a3cb-372752fa2c67\") " pod="openshift-monitoring/thanos-querier-777b758c86-h4jsj" Nov 01 15:48:06.610189 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.610098 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/88d3d141-d4fa-4033-a3cb-372752fa2c67-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-777b758c86-h4jsj\" (UID: \"88d3d141-d4fa-4033-a3cb-372752fa2c67\") " pod="openshift-monitoring/thanos-querier-777b758c86-h4jsj" Nov 01 15:48:06.610189 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.610130 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/88d3d141-d4fa-4033-a3cb-372752fa2c67-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-777b758c86-h4jsj\" (UID: \"88d3d141-d4fa-4033-a3cb-372752fa2c67\") " pod="openshift-monitoring/thanos-querier-777b758c86-h4jsj" Nov 01 15:48:06.610189 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.610147 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/88d3d141-d4fa-4033-a3cb-372752fa2c67-secret-grpc-tls\") pod \"thanos-querier-777b758c86-h4jsj\" (UID: \"88d3d141-d4fa-4033-a3cb-372752fa2c67\") " pod="openshift-monitoring/thanos-querier-777b758c86-h4jsj" Nov 01 15:48:06.610189 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.610166 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/88d3d141-d4fa-4033-a3cb-372752fa2c67-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-777b758c86-h4jsj\" (UID: \"88d3d141-d4fa-4033-a3cb-372752fa2c67\") " pod="openshift-monitoring/thanos-querier-777b758c86-h4jsj" Nov 01 15:48:06.610424 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.610202 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/88d3d141-d4fa-4033-a3cb-372752fa2c67-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-777b758c86-h4jsj\" (UID: \"88d3d141-d4fa-4033-a3cb-372752fa2c67\") " pod="openshift-monitoring/thanos-querier-777b758c86-h4jsj" Nov 01 15:48:06.610424 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.610224 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42pzr\" (UniqueName: \"kubernetes.io/projected/88d3d141-d4fa-4033-a3cb-372752fa2c67-kube-api-access-42pzr\") pod \"thanos-querier-777b758c86-h4jsj\" (UID: \"88d3d141-d4fa-4033-a3cb-372752fa2c67\") " pod="openshift-monitoring/thanos-querier-777b758c86-h4jsj" Nov 01 15:48:06.610833 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.610809 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88d3d141-d4fa-4033-a3cb-372752fa2c67-tmp\") pod \"thanos-querier-777b758c86-h4jsj\" (UID: \"88d3d141-d4fa-4033-a3cb-372752fa2c67\") " pod="openshift-monitoring/thanos-querier-777b758c86-h4jsj" Nov 01 15:48:06.610916 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.610859 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88d3d141-d4fa-4033-a3cb-372752fa2c67-metrics-client-ca\") pod \"thanos-querier-777b758c86-h4jsj\" (UID: \"88d3d141-d4fa-4033-a3cb-372752fa2c67\") " pod="openshift-monitoring/thanos-querier-777b758c86-h4jsj" Nov 01 15:48:06.613634 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.613605 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/88d3d141-d4fa-4033-a3cb-372752fa2c67-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-777b758c86-h4jsj\" (UID: \"88d3d141-d4fa-4033-a3cb-372752fa2c67\") " pod="openshift-monitoring/thanos-querier-777b758c86-h4jsj" Nov 01 15:48:06.614863 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.614571 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/88d3d141-d4fa-4033-a3cb-372752fa2c67-secret-grpc-tls\") pod \"thanos-querier-777b758c86-h4jsj\" (UID: \"88d3d141-d4fa-4033-a3cb-372752fa2c67\") " pod="openshift-monitoring/thanos-querier-777b758c86-h4jsj" Nov 01 15:48:06.616736 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.616689 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/88d3d141-d4fa-4033-a3cb-372752fa2c67-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-777b758c86-h4jsj\" (UID: \"88d3d141-d4fa-4033-a3cb-372752fa2c67\") " pod="openshift-monitoring/thanos-querier-777b758c86-h4jsj" Nov 01 15:48:06.616924 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.616898 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/88d3d141-d4fa-4033-a3cb-372752fa2c67-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-777b758c86-h4jsj\" (UID: \"88d3d141-d4fa-4033-a3cb-372752fa2c67\") " pod="openshift-monitoring/thanos-querier-777b758c86-h4jsj" Nov 01 15:48:06.617345 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.617315 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/88d3d141-d4fa-4033-a3cb-372752fa2c67-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-777b758c86-h4jsj\" (UID: \"88d3d141-d4fa-4033-a3cb-372752fa2c67\") " pod="openshift-monitoring/thanos-querier-777b758c86-h4jsj" Nov 01 15:48:06.618294 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.618270 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/88d3d141-d4fa-4033-a3cb-372752fa2c67-secret-thanos-querier-tls\") pod \"thanos-querier-777b758c86-h4jsj\" (UID: \"88d3d141-d4fa-4033-a3cb-372752fa2c67\") " pod="openshift-monitoring/thanos-querier-777b758c86-h4jsj" Nov 01 15:48:06.620343 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.620326 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42pzr\" (UniqueName: \"kubernetes.io/projected/88d3d141-d4fa-4033-a3cb-372752fa2c67-kube-api-access-42pzr\") pod \"thanos-querier-777b758c86-h4jsj\" (UID: \"88d3d141-d4fa-4033-a3cb-372752fa2c67\") " pod="openshift-monitoring/thanos-querier-777b758c86-h4jsj" Nov 01 15:48:06.797744 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.797675 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-777b758c86-h4jsj" Nov 01 15:48:06.910529 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.910490 4363 generic.go:356] "Generic (PLEG): container finished" podID="9c69f277-d3e5-457d-a4f9-328d96b681d3" containerID="3d644a4c8794f7d0a5c44f63694d79fede5659f94c57150e732977985b11f54a" exitCode=0 Nov 01 15:48:06.910529 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.910524 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pwkct" event={"ID":"9c69f277-d3e5-457d-a4f9-328d96b681d3","Type":"ContainerDied","Data":"3d644a4c8794f7d0a5c44f63694d79fede5659f94c57150e732977985b11f54a"} Nov 01 15:48:06.915310 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.915280 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8b544" event={"ID":"7adfe5b2-8d23-4150-a89d-d8725623a6c0","Type":"ContainerStarted","Data":"c150e71354472932a1ac55f30a6c339df66894bd9e2f82a579f223821a4b3264"} Nov 01 15:48:06.915427 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.915343 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8b544" event={"ID":"7adfe5b2-8d23-4150-a89d-d8725623a6c0","Type":"ContainerStarted","Data":"5919f3d34092c7274dfb55f067f88822e940a0366d9b6b015fd1131aa2634b13"} Nov 01 15:48:06.915427 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.915370 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8b544" event={"ID":"7adfe5b2-8d23-4150-a89d-d8725623a6c0","Type":"ContainerStarted","Data":"77207b579ca08dd8088e78d25c75d5d522fbbb380916d57bf7a3dabc47a1f68f"} Nov 01 15:48:06.917921 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.917891 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jqtqm" event={"ID":"457afc43-0d86-4348-9348-29b4fa1931af","Type":"ContainerStarted","Data":"465f09d46b660754aad52cc6809ce49a935858f6548f17ebd8ce407d2ca9486c"} Nov 01 15:48:06.943662 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.943610 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jqtqm" podStartSLOduration=1.743828616 podStartE2EDuration="2.943593778s" totalImagesPullingTime="1.199765162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:48:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:48:06.942782419 +0000 UTC m=+20.274520070" watchObservedRunningTime="2026-11-01 15:48:06.943593778 +0000 UTC m=+20.275331429" Nov 01 15:48:06.961161 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:06.961116 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8b544" podStartSLOduration=1.527525171 podStartE2EDuration="2.961100866s" totalImagesPullingTime="1.433575695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:48:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:48:06.960240147 +0000 UTC m=+20.291977796" watchObservedRunningTime="2026-11-01 15:48:06.961100866 +0000 UTC m=+20.292838516" Nov 01 15:48:07.073819 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:07.073605 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-777b758c86-h4jsj"] Nov 01 15:48:07.076133 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:48:07.076106 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d3d141_d4fa_4033_a3cb_372752fa2c67.slice/crio-01dcc6e58587969bc96a52996bbe6e75ca913362ae76e0e0e846f3e872e0ba51 WatchSource:0}: Error finding container 01dcc6e58587969bc96a52996bbe6e75ca913362ae76e0e0e846f3e872e0ba51: Status 404 returned error can't find the container with id 01dcc6e58587969bc96a52996bbe6e75ca913362ae76e0e0e846f3e872e0ba51 Nov 01 15:48:07.924611 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:07.924560 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pwkct" event={"ID":"9c69f277-d3e5-457d-a4f9-328d96b681d3","Type":"ContainerStarted","Data":"790597c4e7b25b2c1baa2d49d867d953b0ae0c3e8f22c2c41144342802041ebb"} Nov 01 15:48:07.925130 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:07.924638 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pwkct" event={"ID":"9c69f277-d3e5-457d-a4f9-328d96b681d3","Type":"ContainerStarted","Data":"7e580ee4b0a0cf8932710c316a11a703854f0434478d5a5ee8c6f9791398241e"} Nov 01 15:48:07.926284 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:07.926254 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-777b758c86-h4jsj" event={"ID":"88d3d141-d4fa-4033-a3cb-372752fa2c67","Type":"ContainerStarted","Data":"01dcc6e58587969bc96a52996bbe6e75ca913362ae76e0e0e846f3e872e0ba51"} Nov 01 15:48:07.943222 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:07.943165 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pwkct" podStartSLOduration=2.6393249880000003 podStartE2EDuration="3.943151938s" totalImagesPullingTime="1.229993277s" totalInitContainerRuntime="73.833673ms" isStatefulPod=true podCreationTimestamp="2026-11-01 15:48:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:48:07.942264928 +0000 UTC m=+21.274002582" watchObservedRunningTime="2026-11-01 15:48:07.943151938 +0000 UTC m=+21.274889587" Nov 01 15:48:08.064097 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:08.064067 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-d7slg" Nov 01 15:48:08.931958 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:08.931928 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-777b758c86-h4jsj" event={"ID":"88d3d141-d4fa-4033-a3cb-372752fa2c67","Type":"ContainerStarted","Data":"1aeca0bdad4eb05daa039e0d220c8906afecaef1538fe08c513cde8465c2f984"} Nov 01 15:48:08.932421 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:08.932010 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-777b758c86-h4jsj" event={"ID":"88d3d141-d4fa-4033-a3cb-372752fa2c67","Type":"ContainerStarted","Data":"8949f3bf689d73f548f93a81bb465da56e11162e7ac0e75834b17ea5f8bd2618"} Nov 01 15:48:09.884555 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:09.884521 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-d9spc" Nov 01 15:48:09.938855 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:09.938816 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-777b758c86-h4jsj" event={"ID":"88d3d141-d4fa-4033-a3cb-372752fa2c67","Type":"ContainerStarted","Data":"a66fe1c0362a94403fc75e96fc7d036479e16eb21dd8de514931bc7a1a1785b1"} Nov 01 15:48:09.939320 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:09.938875 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-777b758c86-h4jsj" event={"ID":"88d3d141-d4fa-4033-a3cb-372752fa2c67","Type":"ContainerStarted","Data":"17fb4b035bd6a0553f7c62c32d00faa6a7658ab253619f7a871860ee9c24316c"} Nov 01 15:48:09.939320 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:09.938901 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-777b758c86-h4jsj" event={"ID":"88d3d141-d4fa-4033-a3cb-372752fa2c67","Type":"ContainerStarted","Data":"6e99a35c35c9426cfa3d0ccfe5a5f359eda7b456fe3ce8fe878de4b397dfd123"} Nov 01 15:48:09.939320 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:09.938925 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-777b758c86-h4jsj" event={"ID":"88d3d141-d4fa-4033-a3cb-372752fa2c67","Type":"ContainerStarted","Data":"ef5ecf75f16d42b7a6a87d48d092bcafa261cb612acff05c7140a105ec2749ab"} Nov 01 15:48:09.939320 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:09.938977 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-777b758c86-h4jsj" Nov 01 15:48:09.959984 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:09.959931 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-777b758c86-h4jsj" podStartSLOduration=1.4580434119999999 podStartE2EDuration="3.959912825s" totalImagesPullingTime="2.501869413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:48:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:48:09.958521905 +0000 UTC m=+23.290259567" watchObservedRunningTime="2026-11-01 15:48:09.959912825 +0000 UTC m=+23.291650476" Nov 01 15:48:13.261096 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:13.261066 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jtwhw" Nov 01 15:48:15.950514 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:15.950470 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-777b758c86-h4jsj" Nov 01 15:48:16.222783 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:16.222706 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-l26rw" Nov 01 15:48:18.307489 ip-10-0-128-238 kubenswrapper[4363]: E1101 15:48:18.307452 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 15:48:18.872631 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:18.872609 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-85755df759-sswt8" Nov 01 15:48:19.235607 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:19.235535 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-8vq2s"] Nov 01 15:48:19.268331 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:19.268303 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-8vq2s"] Nov 01 15:48:19.268474 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:19.268392 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-8vq2s" Nov 01 15:48:19.271550 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:19.271515 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-dvztt\"" Nov 01 15:48:19.271706 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:19.271689 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 01 15:48:19.271773 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:19.271747 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 01 15:48:19.416462 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:19.416426 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czxd4\" (UniqueName: \"kubernetes.io/projected/455d1f7d-3a33-43de-8561-6439dcab4589-kube-api-access-czxd4\") pod \"downloads-6dd789c8b7-8vq2s\" (UID: \"455d1f7d-3a33-43de-8561-6439dcab4589\") " pod="openshift-console/downloads-6dd789c8b7-8vq2s" Nov 01 15:48:19.416849 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:19.416467 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/455d1f7d-3a33-43de-8561-6439dcab4589-tmp\") pod \"downloads-6dd789c8b7-8vq2s\" (UID: \"455d1f7d-3a33-43de-8561-6439dcab4589\") " pod="openshift-console/downloads-6dd789c8b7-8vq2s" Nov 01 15:48:19.517683 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:19.517606 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/455d1f7d-3a33-43de-8561-6439dcab4589-tmp\") pod \"downloads-6dd789c8b7-8vq2s\" (UID: \"455d1f7d-3a33-43de-8561-6439dcab4589\") " pod="openshift-console/downloads-6dd789c8b7-8vq2s" Nov 01 15:48:19.517683 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:19.517671 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czxd4\" (UniqueName: \"kubernetes.io/projected/455d1f7d-3a33-43de-8561-6439dcab4589-kube-api-access-czxd4\") pod \"downloads-6dd789c8b7-8vq2s\" (UID: \"455d1f7d-3a33-43de-8561-6439dcab4589\") " pod="openshift-console/downloads-6dd789c8b7-8vq2s" Nov 01 15:48:19.517989 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:19.517971 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/455d1f7d-3a33-43de-8561-6439dcab4589-tmp\") pod \"downloads-6dd789c8b7-8vq2s\" (UID: \"455d1f7d-3a33-43de-8561-6439dcab4589\") " pod="openshift-console/downloads-6dd789c8b7-8vq2s" Nov 01 15:48:19.526065 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:19.526039 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czxd4\" (UniqueName: \"kubernetes.io/projected/455d1f7d-3a33-43de-8561-6439dcab4589-kube-api-access-czxd4\") pod \"downloads-6dd789c8b7-8vq2s\" (UID: \"455d1f7d-3a33-43de-8561-6439dcab4589\") " pod="openshift-console/downloads-6dd789c8b7-8vq2s" Nov 01 15:48:19.591037 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:19.590989 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-8vq2s" Nov 01 15:48:19.882043 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:19.881873 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-8vq2s"] Nov 01 15:48:19.884650 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:48:19.884622 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod455d1f7d_3a33_43de_8561_6439dcab4589.slice/crio-73cd12f958cf12c17343781a9322a2f9bd59a903adf270455bf426757e0044d9 WatchSource:0}: Error finding container 73cd12f958cf12c17343781a9322a2f9bd59a903adf270455bf426757e0044d9: Status 404 returned error can't find the container with id 73cd12f958cf12c17343781a9322a2f9bd59a903adf270455bf426757e0044d9 Nov 01 15:48:19.977597 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:19.977564 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-8vq2s" event={"ID":"455d1f7d-3a33-43de-8561-6439dcab4589","Type":"ContainerStarted","Data":"73cd12f958cf12c17343781a9322a2f9bd59a903adf270455bf426757e0044d9"} Nov 01 15:48:20.688424 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:20.688386 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-68bb497dfc-xs9s9"] Nov 01 15:48:20.690550 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:20.690525 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68bb497dfc-xs9s9" Nov 01 15:48:20.694294 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:20.694274 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 01 15:48:20.694418 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:20.694274 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-crd6q\"" Nov 01 15:48:20.694418 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:20.694318 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 01 15:48:20.694418 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:20.694330 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 01 15:48:20.694418 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:20.694364 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 01 15:48:20.694418 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:20.694392 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 01 15:48:20.703488 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:20.703468 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68bb497dfc-xs9s9"] Nov 01 15:48:20.827685 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:20.827647 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db59b0e4-187a-4fe2-b50c-9028b871a70f-console-oauth-config\") pod \"console-68bb497dfc-xs9s9\" (UID: \"db59b0e4-187a-4fe2-b50c-9028b871a70f\") " pod="openshift-console/console-68bb497dfc-xs9s9" Nov 01 15:48:20.827874 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:20.827714 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db59b0e4-187a-4fe2-b50c-9028b871a70f-service-ca\") pod \"console-68bb497dfc-xs9s9\" (UID: \"db59b0e4-187a-4fe2-b50c-9028b871a70f\") " pod="openshift-console/console-68bb497dfc-xs9s9" Nov 01 15:48:20.827874 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:20.827805 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db59b0e4-187a-4fe2-b50c-9028b871a70f-oauth-serving-cert\") pod \"console-68bb497dfc-xs9s9\" (UID: \"db59b0e4-187a-4fe2-b50c-9028b871a70f\") " pod="openshift-console/console-68bb497dfc-xs9s9" Nov 01 15:48:20.827874 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:20.827841 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db59b0e4-187a-4fe2-b50c-9028b871a70f-console-serving-cert\") pod \"console-68bb497dfc-xs9s9\" (UID: \"db59b0e4-187a-4fe2-b50c-9028b871a70f\") " pod="openshift-console/console-68bb497dfc-xs9s9" Nov 01 15:48:20.827874 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:20.827864 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb4lw\" (UniqueName: \"kubernetes.io/projected/db59b0e4-187a-4fe2-b50c-9028b871a70f-kube-api-access-mb4lw\") pod \"console-68bb497dfc-xs9s9\" (UID: \"db59b0e4-187a-4fe2-b50c-9028b871a70f\") " pod="openshift-console/console-68bb497dfc-xs9s9" Nov 01 15:48:20.828063 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:20.827967 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db59b0e4-187a-4fe2-b50c-9028b871a70f-console-config\") pod \"console-68bb497dfc-xs9s9\" (UID: \"db59b0e4-187a-4fe2-b50c-9028b871a70f\") " pod="openshift-console/console-68bb497dfc-xs9s9" Nov 01 15:48:20.828063 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:20.828006 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db59b0e4-187a-4fe2-b50c-9028b871a70f-tmp\") pod \"console-68bb497dfc-xs9s9\" (UID: \"db59b0e4-187a-4fe2-b50c-9028b871a70f\") " pod="openshift-console/console-68bb497dfc-xs9s9" Nov 01 15:48:20.929120 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:20.929083 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db59b0e4-187a-4fe2-b50c-9028b871a70f-tmp\") pod \"console-68bb497dfc-xs9s9\" (UID: \"db59b0e4-187a-4fe2-b50c-9028b871a70f\") " pod="openshift-console/console-68bb497dfc-xs9s9" Nov 01 15:48:20.929346 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:20.929143 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db59b0e4-187a-4fe2-b50c-9028b871a70f-console-oauth-config\") pod \"console-68bb497dfc-xs9s9\" (UID: \"db59b0e4-187a-4fe2-b50c-9028b871a70f\") " pod="openshift-console/console-68bb497dfc-xs9s9" Nov 01 15:48:20.929346 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:20.929208 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db59b0e4-187a-4fe2-b50c-9028b871a70f-service-ca\") pod \"console-68bb497dfc-xs9s9\" (UID: \"db59b0e4-187a-4fe2-b50c-9028b871a70f\") " pod="openshift-console/console-68bb497dfc-xs9s9" Nov 01 15:48:20.929346 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:20.929245 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db59b0e4-187a-4fe2-b50c-9028b871a70f-oauth-serving-cert\") pod \"console-68bb497dfc-xs9s9\" (UID: \"db59b0e4-187a-4fe2-b50c-9028b871a70f\") " pod="openshift-console/console-68bb497dfc-xs9s9" Nov 01 15:48:20.929346 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:20.929269 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db59b0e4-187a-4fe2-b50c-9028b871a70f-console-serving-cert\") pod \"console-68bb497dfc-xs9s9\" (UID: \"db59b0e4-187a-4fe2-b50c-9028b871a70f\") " pod="openshift-console/console-68bb497dfc-xs9s9" Nov 01 15:48:20.929346 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:20.929286 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb4lw\" (UniqueName: \"kubernetes.io/projected/db59b0e4-187a-4fe2-b50c-9028b871a70f-kube-api-access-mb4lw\") pod \"console-68bb497dfc-xs9s9\" (UID: \"db59b0e4-187a-4fe2-b50c-9028b871a70f\") " pod="openshift-console/console-68bb497dfc-xs9s9" Nov 01 15:48:20.929346 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:20.929307 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db59b0e4-187a-4fe2-b50c-9028b871a70f-console-config\") pod \"console-68bb497dfc-xs9s9\" (UID: \"db59b0e4-187a-4fe2-b50c-9028b871a70f\") " pod="openshift-console/console-68bb497dfc-xs9s9" Nov 01 15:48:20.929556 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:20.929528 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db59b0e4-187a-4fe2-b50c-9028b871a70f-tmp\") pod \"console-68bb497dfc-xs9s9\" (UID: \"db59b0e4-187a-4fe2-b50c-9028b871a70f\") " pod="openshift-console/console-68bb497dfc-xs9s9" Nov 01 15:48:20.929873 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:20.929857 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db59b0e4-187a-4fe2-b50c-9028b871a70f-console-config\") pod \"console-68bb497dfc-xs9s9\" (UID: \"db59b0e4-187a-4fe2-b50c-9028b871a70f\") " pod="openshift-console/console-68bb497dfc-xs9s9" Nov 01 15:48:20.929968 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:20.929952 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db59b0e4-187a-4fe2-b50c-9028b871a70f-oauth-serving-cert\") pod \"console-68bb497dfc-xs9s9\" (UID: \"db59b0e4-187a-4fe2-b50c-9028b871a70f\") " pod="openshift-console/console-68bb497dfc-xs9s9" Nov 01 15:48:20.930778 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:20.930749 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db59b0e4-187a-4fe2-b50c-9028b871a70f-service-ca\") pod \"console-68bb497dfc-xs9s9\" (UID: \"db59b0e4-187a-4fe2-b50c-9028b871a70f\") " pod="openshift-console/console-68bb497dfc-xs9s9" Nov 01 15:48:20.932375 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:20.932352 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db59b0e4-187a-4fe2-b50c-9028b871a70f-console-oauth-config\") pod \"console-68bb497dfc-xs9s9\" (UID: \"db59b0e4-187a-4fe2-b50c-9028b871a70f\") " pod="openshift-console/console-68bb497dfc-xs9s9" Nov 01 15:48:20.932696 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:20.932676 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db59b0e4-187a-4fe2-b50c-9028b871a70f-console-serving-cert\") pod \"console-68bb497dfc-xs9s9\" (UID: \"db59b0e4-187a-4fe2-b50c-9028b871a70f\") " pod="openshift-console/console-68bb497dfc-xs9s9" Nov 01 15:48:20.938695 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:20.938639 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb4lw\" (UniqueName: \"kubernetes.io/projected/db59b0e4-187a-4fe2-b50c-9028b871a70f-kube-api-access-mb4lw\") pod \"console-68bb497dfc-xs9s9\" (UID: \"db59b0e4-187a-4fe2-b50c-9028b871a70f\") " pod="openshift-console/console-68bb497dfc-xs9s9" Nov 01 15:48:21.003770 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:21.003736 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68bb497dfc-xs9s9" Nov 01 15:48:21.287921 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:21.287848 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68bb497dfc-xs9s9"] Nov 01 15:48:21.986724 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:21.986690 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68bb497dfc-xs9s9" event={"ID":"db59b0e4-187a-4fe2-b50c-9028b871a70f","Type":"ContainerStarted","Data":"59c20282ccfec90db408643bfeb0a4af768e2e92220f6165c8b7dd3502c47ec7"} Nov 01 15:48:25.000757 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:25.000719 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68bb497dfc-xs9s9" event={"ID":"db59b0e4-187a-4fe2-b50c-9028b871a70f","Type":"ContainerStarted","Data":"97529ad126188abe4946944db13e9d4eb8c7ffacc4bb5024a72172c447a6bac4"} Nov 01 15:48:25.019390 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:25.019335 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-68bb497dfc-xs9s9" podStartSLOduration=1.865967023 podStartE2EDuration="5.01932096s" totalImagesPullingTime="3.153353937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:48:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:48:25.018158257 +0000 UTC m=+38.349895906" watchObservedRunningTime="2026-11-01 15:48:25.01932096 +0000 UTC m=+38.351058610" Nov 01 15:48:31.004633 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:31.004542 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-68bb497dfc-xs9s9" Nov 01 15:48:31.004633 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:31.004617 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68bb497dfc-xs9s9" Nov 01 15:48:31.009968 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:31.009944 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-68bb497dfc-xs9s9" Nov 01 15:48:31.030074 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:31.030051 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-68bb497dfc-xs9s9" Nov 01 15:48:32.717268 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:32.717222 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-855d4fbdf5-6kgtq"] Nov 01 15:48:32.746629 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:32.746597 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-855d4fbdf5-6kgtq"] Nov 01 15:48:32.746798 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:32.746697 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-855d4fbdf5-6kgtq" Nov 01 15:48:32.753527 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:32.753484 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 01 15:48:32.937614 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:32.937576 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc7adf5a-3da8-4856-91a3-03b8677e2af7-console-config\") pod \"console-855d4fbdf5-6kgtq\" (UID: \"bc7adf5a-3da8-4856-91a3-03b8677e2af7\") " pod="openshift-console/console-855d4fbdf5-6kgtq" Nov 01 15:48:32.937614 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:32.937617 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc7adf5a-3da8-4856-91a3-03b8677e2af7-service-ca\") pod \"console-855d4fbdf5-6kgtq\" (UID: \"bc7adf5a-3da8-4856-91a3-03b8677e2af7\") " pod="openshift-console/console-855d4fbdf5-6kgtq" Nov 01 15:48:32.937853 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:32.937650 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc7adf5a-3da8-4856-91a3-03b8677e2af7-console-serving-cert\") pod \"console-855d4fbdf5-6kgtq\" (UID: \"bc7adf5a-3da8-4856-91a3-03b8677e2af7\") " pod="openshift-console/console-855d4fbdf5-6kgtq" Nov 01 15:48:32.937853 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:32.937767 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc7adf5a-3da8-4856-91a3-03b8677e2af7-trusted-ca-bundle\") pod \"console-855d4fbdf5-6kgtq\" (UID: \"bc7adf5a-3da8-4856-91a3-03b8677e2af7\") " pod="openshift-console/console-855d4fbdf5-6kgtq" Nov 01 15:48:32.937853 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:32.937806 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc7adf5a-3da8-4856-91a3-03b8677e2af7-oauth-serving-cert\") pod \"console-855d4fbdf5-6kgtq\" (UID: \"bc7adf5a-3da8-4856-91a3-03b8677e2af7\") " pod="openshift-console/console-855d4fbdf5-6kgtq" Nov 01 15:48:32.937853 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:32.937835 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc7adf5a-3da8-4856-91a3-03b8677e2af7-console-oauth-config\") pod \"console-855d4fbdf5-6kgtq\" (UID: \"bc7adf5a-3da8-4856-91a3-03b8677e2af7\") " pod="openshift-console/console-855d4fbdf5-6kgtq" Nov 01 15:48:32.938051 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:32.937859 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwvvt\" (UniqueName: \"kubernetes.io/projected/bc7adf5a-3da8-4856-91a3-03b8677e2af7-kube-api-access-kwvvt\") pod \"console-855d4fbdf5-6kgtq\" (UID: \"bc7adf5a-3da8-4856-91a3-03b8677e2af7\") " pod="openshift-console/console-855d4fbdf5-6kgtq" Nov 01 15:48:32.938051 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:32.937951 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc7adf5a-3da8-4856-91a3-03b8677e2af7-tmp\") pod \"console-855d4fbdf5-6kgtq\" (UID: \"bc7adf5a-3da8-4856-91a3-03b8677e2af7\") " pod="openshift-console/console-855d4fbdf5-6kgtq" Nov 01 15:48:33.038701 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:33.038618 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc7adf5a-3da8-4856-91a3-03b8677e2af7-tmp\") pod \"console-855d4fbdf5-6kgtq\" (UID: \"bc7adf5a-3da8-4856-91a3-03b8677e2af7\") " pod="openshift-console/console-855d4fbdf5-6kgtq" Nov 01 15:48:33.038701 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:33.038679 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc7adf5a-3da8-4856-91a3-03b8677e2af7-console-config\") pod \"console-855d4fbdf5-6kgtq\" (UID: \"bc7adf5a-3da8-4856-91a3-03b8677e2af7\") " pod="openshift-console/console-855d4fbdf5-6kgtq" Nov 01 15:48:33.038701 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:33.038697 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc7adf5a-3da8-4856-91a3-03b8677e2af7-service-ca\") pod \"console-855d4fbdf5-6kgtq\" (UID: \"bc7adf5a-3da8-4856-91a3-03b8677e2af7\") " pod="openshift-console/console-855d4fbdf5-6kgtq" Nov 01 15:48:33.038969 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:33.038723 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc7adf5a-3da8-4856-91a3-03b8677e2af7-console-serving-cert\") pod \"console-855d4fbdf5-6kgtq\" (UID: \"bc7adf5a-3da8-4856-91a3-03b8677e2af7\") " pod="openshift-console/console-855d4fbdf5-6kgtq" Nov 01 15:48:33.038969 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:33.038766 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc7adf5a-3da8-4856-91a3-03b8677e2af7-trusted-ca-bundle\") pod \"console-855d4fbdf5-6kgtq\" (UID: \"bc7adf5a-3da8-4856-91a3-03b8677e2af7\") " pod="openshift-console/console-855d4fbdf5-6kgtq" Nov 01 15:48:33.038969 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:33.038793 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc7adf5a-3da8-4856-91a3-03b8677e2af7-oauth-serving-cert\") pod \"console-855d4fbdf5-6kgtq\" (UID: \"bc7adf5a-3da8-4856-91a3-03b8677e2af7\") " pod="openshift-console/console-855d4fbdf5-6kgtq" Nov 01 15:48:33.038969 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:33.038826 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc7adf5a-3da8-4856-91a3-03b8677e2af7-console-oauth-config\") pod \"console-855d4fbdf5-6kgtq\" (UID: \"bc7adf5a-3da8-4856-91a3-03b8677e2af7\") " pod="openshift-console/console-855d4fbdf5-6kgtq" Nov 01 15:48:33.038969 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:33.038862 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwvvt\" (UniqueName: \"kubernetes.io/projected/bc7adf5a-3da8-4856-91a3-03b8677e2af7-kube-api-access-kwvvt\") pod \"console-855d4fbdf5-6kgtq\" (UID: \"bc7adf5a-3da8-4856-91a3-03b8677e2af7\") " pod="openshift-console/console-855d4fbdf5-6kgtq" Nov 01 15:48:33.039246 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:33.039113 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc7adf5a-3da8-4856-91a3-03b8677e2af7-tmp\") pod \"console-855d4fbdf5-6kgtq\" (UID: \"bc7adf5a-3da8-4856-91a3-03b8677e2af7\") " pod="openshift-console/console-855d4fbdf5-6kgtq" Nov 01 15:48:33.039766 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:33.039740 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc7adf5a-3da8-4856-91a3-03b8677e2af7-console-config\") pod \"console-855d4fbdf5-6kgtq\" (UID: \"bc7adf5a-3da8-4856-91a3-03b8677e2af7\") " pod="openshift-console/console-855d4fbdf5-6kgtq" Nov 01 15:48:33.039896 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:33.039744 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc7adf5a-3da8-4856-91a3-03b8677e2af7-service-ca\") pod \"console-855d4fbdf5-6kgtq\" (UID: \"bc7adf5a-3da8-4856-91a3-03b8677e2af7\") " pod="openshift-console/console-855d4fbdf5-6kgtq" Nov 01 15:48:33.039896 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:33.039835 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc7adf5a-3da8-4856-91a3-03b8677e2af7-oauth-serving-cert\") pod \"console-855d4fbdf5-6kgtq\" (UID: \"bc7adf5a-3da8-4856-91a3-03b8677e2af7\") " pod="openshift-console/console-855d4fbdf5-6kgtq" Nov 01 15:48:33.040031 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:33.039908 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc7adf5a-3da8-4856-91a3-03b8677e2af7-trusted-ca-bundle\") pod \"console-855d4fbdf5-6kgtq\" (UID: \"bc7adf5a-3da8-4856-91a3-03b8677e2af7\") " pod="openshift-console/console-855d4fbdf5-6kgtq" Nov 01 15:48:33.041519 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:33.041496 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc7adf5a-3da8-4856-91a3-03b8677e2af7-console-oauth-config\") pod \"console-855d4fbdf5-6kgtq\" (UID: \"bc7adf5a-3da8-4856-91a3-03b8677e2af7\") " pod="openshift-console/console-855d4fbdf5-6kgtq" Nov 01 15:48:33.041612 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:33.041521 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc7adf5a-3da8-4856-91a3-03b8677e2af7-console-serving-cert\") pod \"console-855d4fbdf5-6kgtq\" (UID: \"bc7adf5a-3da8-4856-91a3-03b8677e2af7\") " pod="openshift-console/console-855d4fbdf5-6kgtq" Nov 01 15:48:33.048269 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:33.048245 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwvvt\" (UniqueName: \"kubernetes.io/projected/bc7adf5a-3da8-4856-91a3-03b8677e2af7-kube-api-access-kwvvt\") pod \"console-855d4fbdf5-6kgtq\" (UID: \"bc7adf5a-3da8-4856-91a3-03b8677e2af7\") " pod="openshift-console/console-855d4fbdf5-6kgtq" Nov 01 15:48:33.068936 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:33.068911 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-855d4fbdf5-6kgtq" Nov 01 15:48:33.344588 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:33.344557 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-855d4fbdf5-6kgtq"] Nov 01 15:48:33.348644 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:48:33.348615 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc7adf5a_3da8_4856_91a3_03b8677e2af7.slice/crio-809bab93bcfdf3936baac5f702ea3fc2c812f8e816620f48d765be260559e376 WatchSource:0}: Error finding container 809bab93bcfdf3936baac5f702ea3fc2c812f8e816620f48d765be260559e376: Status 404 returned error can't find the container with id 809bab93bcfdf3936baac5f702ea3fc2c812f8e816620f48d765be260559e376 Nov 01 15:48:34.038000 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:34.037960 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-855d4fbdf5-6kgtq" event={"ID":"bc7adf5a-3da8-4856-91a3-03b8677e2af7","Type":"ContainerStarted","Data":"58c6d76585252e66073f2b17dd5efe03469656d0c42f18e666625f814dad3803"} Nov 01 15:48:34.038574 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:34.038077 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-855d4fbdf5-6kgtq" event={"ID":"bc7adf5a-3da8-4856-91a3-03b8677e2af7","Type":"ContainerStarted","Data":"809bab93bcfdf3936baac5f702ea3fc2c812f8e816620f48d765be260559e376"} Nov 01 15:48:34.081874 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:34.081822 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-855d4fbdf5-6kgtq" podStartSLOduration=2.081803587 podStartE2EDuration="2.081803587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:48:34.080791367 +0000 UTC m=+47.412529018" watchObservedRunningTime="2026-11-01 15:48:34.081803587 +0000 UTC m=+47.413541237" Nov 01 15:48:36.048681 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:36.048634 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-8vq2s" event={"ID":"455d1f7d-3a33-43de-8561-6439dcab4589","Type":"ContainerStarted","Data":"3805b73575207f8eadda888f4ab2619c69f0a57fd58622b212f20ec044c6c0bc"} Nov 01 15:48:36.049178 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:36.048889 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-8vq2s" Nov 01 15:48:36.060139 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:36.060115 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-8vq2s" Nov 01 15:48:36.068549 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:36.068498 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-8vq2s" podStartSLOduration=1.557006891 podStartE2EDuration="17.068482993s" totalImagesPullingTime="15.511476102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:48:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:48:36.067889467 +0000 UTC m=+49.399627116" watchObservedRunningTime="2026-11-01 15:48:36.068482993 +0000 UTC m=+49.400220645" Nov 01 15:48:43.069929 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:43.069887 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-855d4fbdf5-6kgtq" Nov 01 15:48:43.070493 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:43.070148 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-855d4fbdf5-6kgtq" Nov 01 15:48:43.075497 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:43.075472 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-855d4fbdf5-6kgtq" Nov 01 15:48:44.083713 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:44.083687 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-855d4fbdf5-6kgtq" Nov 01 15:48:44.128575 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:48:44.128547 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68bb497dfc-xs9s9"] Nov 01 15:48:48.307779 ip-10-0-128-238 kubenswrapper[4363]: E1101 15:48:48.307740 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 15:49:00.931306 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:00.931272 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-9b5784bf-hvxsb"] Nov 01 15:49:00.953878 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:00.953850 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9b5784bf-hvxsb"] Nov 01 15:49:00.954046 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:00.953965 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9b5784bf-hvxsb" Nov 01 15:49:01.042214 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:01.042188 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/39d2972b-d3ff-4b75-a329-37827adb7c9d-console-config\") pod \"console-9b5784bf-hvxsb\" (UID: \"39d2972b-d3ff-4b75-a329-37827adb7c9d\") " pod="openshift-console/console-9b5784bf-hvxsb" Nov 01 15:49:01.042350 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:01.042223 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/39d2972b-d3ff-4b75-a329-37827adb7c9d-console-serving-cert\") pod \"console-9b5784bf-hvxsb\" (UID: \"39d2972b-d3ff-4b75-a329-37827adb7c9d\") " pod="openshift-console/console-9b5784bf-hvxsb" Nov 01 15:49:01.042350 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:01.042238 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39d2972b-d3ff-4b75-a329-37827adb7c9d-tmp\") pod \"console-9b5784bf-hvxsb\" (UID: \"39d2972b-d3ff-4b75-a329-37827adb7c9d\") " pod="openshift-console/console-9b5784bf-hvxsb" Nov 01 15:49:01.042350 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:01.042314 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39d2972b-d3ff-4b75-a329-37827adb7c9d-trusted-ca-bundle\") pod \"console-9b5784bf-hvxsb\" (UID: \"39d2972b-d3ff-4b75-a329-37827adb7c9d\") " pod="openshift-console/console-9b5784bf-hvxsb" Nov 01 15:49:01.042453 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:01.042400 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/39d2972b-d3ff-4b75-a329-37827adb7c9d-console-oauth-config\") pod \"console-9b5784bf-hvxsb\" (UID: \"39d2972b-d3ff-4b75-a329-37827adb7c9d\") " pod="openshift-console/console-9b5784bf-hvxsb" Nov 01 15:49:01.042453 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:01.042443 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/39d2972b-d3ff-4b75-a329-37827adb7c9d-oauth-serving-cert\") pod \"console-9b5784bf-hvxsb\" (UID: \"39d2972b-d3ff-4b75-a329-37827adb7c9d\") " pod="openshift-console/console-9b5784bf-hvxsb" Nov 01 15:49:01.042532 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:01.042519 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/39d2972b-d3ff-4b75-a329-37827adb7c9d-service-ca\") pod \"console-9b5784bf-hvxsb\" (UID: \"39d2972b-d3ff-4b75-a329-37827adb7c9d\") " pod="openshift-console/console-9b5784bf-hvxsb" Nov 01 15:49:01.042567 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:01.042545 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgj4z\" (UniqueName: \"kubernetes.io/projected/39d2972b-d3ff-4b75-a329-37827adb7c9d-kube-api-access-hgj4z\") pod \"console-9b5784bf-hvxsb\" (UID: \"39d2972b-d3ff-4b75-a329-37827adb7c9d\") " pod="openshift-console/console-9b5784bf-hvxsb" Nov 01 15:49:01.143390 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:01.143357 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/39d2972b-d3ff-4b75-a329-37827adb7c9d-oauth-serving-cert\") pod \"console-9b5784bf-hvxsb\" (UID: \"39d2972b-d3ff-4b75-a329-37827adb7c9d\") " pod="openshift-console/console-9b5784bf-hvxsb" Nov 01 15:49:01.144091 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:01.144062 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/39d2972b-d3ff-4b75-a329-37827adb7c9d-service-ca\") pod \"console-9b5784bf-hvxsb\" (UID: \"39d2972b-d3ff-4b75-a329-37827adb7c9d\") " pod="openshift-console/console-9b5784bf-hvxsb" Nov 01 15:49:01.144215 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:01.144119 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgj4z\" (UniqueName: \"kubernetes.io/projected/39d2972b-d3ff-4b75-a329-37827adb7c9d-kube-api-access-hgj4z\") pod \"console-9b5784bf-hvxsb\" (UID: \"39d2972b-d3ff-4b75-a329-37827adb7c9d\") " pod="openshift-console/console-9b5784bf-hvxsb" Nov 01 15:49:01.144465 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:01.144445 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/39d2972b-d3ff-4b75-a329-37827adb7c9d-console-config\") pod \"console-9b5784bf-hvxsb\" (UID: \"39d2972b-d3ff-4b75-a329-37827adb7c9d\") " pod="openshift-console/console-9b5784bf-hvxsb" Nov 01 15:49:01.144510 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:01.144497 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/39d2972b-d3ff-4b75-a329-37827adb7c9d-console-serving-cert\") pod \"console-9b5784bf-hvxsb\" (UID: \"39d2972b-d3ff-4b75-a329-37827adb7c9d\") " pod="openshift-console/console-9b5784bf-hvxsb" Nov 01 15:49:01.144545 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:01.144520 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39d2972b-d3ff-4b75-a329-37827adb7c9d-tmp\") pod \"console-9b5784bf-hvxsb\" (UID: \"39d2972b-d3ff-4b75-a329-37827adb7c9d\") " pod="openshift-console/console-9b5784bf-hvxsb" Nov 01 15:49:01.144545 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:01.144529 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/39d2972b-d3ff-4b75-a329-37827adb7c9d-oauth-serving-cert\") pod \"console-9b5784bf-hvxsb\" (UID: \"39d2972b-d3ff-4b75-a329-37827adb7c9d\") " pod="openshift-console/console-9b5784bf-hvxsb" Nov 01 15:49:01.144607 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:01.144578 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39d2972b-d3ff-4b75-a329-37827adb7c9d-trusted-ca-bundle\") pod \"console-9b5784bf-hvxsb\" (UID: \"39d2972b-d3ff-4b75-a329-37827adb7c9d\") " pod="openshift-console/console-9b5784bf-hvxsb" Nov 01 15:49:01.144645 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:01.144605 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/39d2972b-d3ff-4b75-a329-37827adb7c9d-console-oauth-config\") pod \"console-9b5784bf-hvxsb\" (UID: \"39d2972b-d3ff-4b75-a329-37827adb7c9d\") " pod="openshift-console/console-9b5784bf-hvxsb" Nov 01 15:49:01.144836 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:01.144819 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39d2972b-d3ff-4b75-a329-37827adb7c9d-tmp\") pod \"console-9b5784bf-hvxsb\" (UID: \"39d2972b-d3ff-4b75-a329-37827adb7c9d\") " pod="openshift-console/console-9b5784bf-hvxsb" Nov 01 15:49:01.145390 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:01.145363 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/39d2972b-d3ff-4b75-a329-37827adb7c9d-console-config\") pod \"console-9b5784bf-hvxsb\" (UID: \"39d2972b-d3ff-4b75-a329-37827adb7c9d\") " pod="openshift-console/console-9b5784bf-hvxsb" Nov 01 15:49:01.146057 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:01.146009 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/39d2972b-d3ff-4b75-a329-37827adb7c9d-service-ca\") pod \"console-9b5784bf-hvxsb\" (UID: \"39d2972b-d3ff-4b75-a329-37827adb7c9d\") " pod="openshift-console/console-9b5784bf-hvxsb" Nov 01 15:49:01.146395 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:01.146375 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39d2972b-d3ff-4b75-a329-37827adb7c9d-trusted-ca-bundle\") pod \"console-9b5784bf-hvxsb\" (UID: \"39d2972b-d3ff-4b75-a329-37827adb7c9d\") " pod="openshift-console/console-9b5784bf-hvxsb" Nov 01 15:49:01.146898 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:01.146880 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/39d2972b-d3ff-4b75-a329-37827adb7c9d-console-oauth-config\") pod \"console-9b5784bf-hvxsb\" (UID: \"39d2972b-d3ff-4b75-a329-37827adb7c9d\") " pod="openshift-console/console-9b5784bf-hvxsb" Nov 01 15:49:01.147731 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:01.147715 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/39d2972b-d3ff-4b75-a329-37827adb7c9d-console-serving-cert\") pod \"console-9b5784bf-hvxsb\" (UID: \"39d2972b-d3ff-4b75-a329-37827adb7c9d\") " pod="openshift-console/console-9b5784bf-hvxsb" Nov 01 15:49:01.152103 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:01.152085 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgj4z\" (UniqueName: \"kubernetes.io/projected/39d2972b-d3ff-4b75-a329-37827adb7c9d-kube-api-access-hgj4z\") pod \"console-9b5784bf-hvxsb\" (UID: \"39d2972b-d3ff-4b75-a329-37827adb7c9d\") " pod="openshift-console/console-9b5784bf-hvxsb" Nov 01 15:49:01.279918 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:01.279842 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9b5784bf-hvxsb" Nov 01 15:49:01.579047 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:01.578998 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9b5784bf-hvxsb"] Nov 01 15:49:01.581900 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:49:01.581863 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39d2972b_d3ff_4b75_a329_37827adb7c9d.slice/crio-a85c1207ecc0337d0f38b6680aa931a1046063bd8bed088b3bd554829246b0a9 WatchSource:0}: Error finding container a85c1207ecc0337d0f38b6680aa931a1046063bd8bed088b3bd554829246b0a9: Status 404 returned error can't find the container with id a85c1207ecc0337d0f38b6680aa931a1046063bd8bed088b3bd554829246b0a9 Nov 01 15:49:02.151464 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:02.151431 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9b5784bf-hvxsb" event={"ID":"39d2972b-d3ff-4b75-a329-37827adb7c9d","Type":"ContainerStarted","Data":"e449bf5c69cad0508e0939a2a41e6eb66b22a7420d1e223fe42eee2de9652f8e"} Nov 01 15:49:02.151819 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:02.151479 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9b5784bf-hvxsb" event={"ID":"39d2972b-d3ff-4b75-a329-37827adb7c9d","Type":"ContainerStarted","Data":"a85c1207ecc0337d0f38b6680aa931a1046063bd8bed088b3bd554829246b0a9"} Nov 01 15:49:02.167924 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:02.167876 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-9b5784bf-hvxsb" podStartSLOduration=2.167862332 podStartE2EDuration="2.167862332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:49:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:49:02.166634855 +0000 UTC m=+75.498372505" watchObservedRunningTime="2026-11-01 15:49:02.167862332 +0000 UTC m=+75.499599981" Nov 01 15:49:09.148227 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:09.148188 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-68bb497dfc-xs9s9" podUID="db59b0e4-187a-4fe2-b50c-9028b871a70f" containerName="console" containerID="cri-o://97529ad126188abe4946944db13e9d4eb8c7ffacc4bb5024a72172c447a6bac4" gracePeriod=15 Nov 01 15:49:09.417965 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:09.417887 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68bb497dfc-xs9s9" Nov 01 15:49:09.515239 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:09.515211 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db59b0e4-187a-4fe2-b50c-9028b871a70f-tmp\" (UniqueName: \"kubernetes.io/empty-dir/db59b0e4-187a-4fe2-b50c-9028b871a70f-tmp\") pod \"db59b0e4-187a-4fe2-b50c-9028b871a70f\" (UID: \"db59b0e4-187a-4fe2-b50c-9028b871a70f\") " Nov 01 15:49:09.515409 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:09.515253 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db59b0e4-187a-4fe2-b50c-9028b871a70f-service-ca\" (UniqueName: \"kubernetes.io/configmap/db59b0e4-187a-4fe2-b50c-9028b871a70f-service-ca\") pod \"db59b0e4-187a-4fe2-b50c-9028b871a70f\" (UID: \"db59b0e4-187a-4fe2-b50c-9028b871a70f\") " Nov 01 15:49:09.515409 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:09.515282 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db59b0e4-187a-4fe2-b50c-9028b871a70f-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db59b0e4-187a-4fe2-b50c-9028b871a70f-oauth-serving-cert\") pod \"db59b0e4-187a-4fe2-b50c-9028b871a70f\" (UID: \"db59b0e4-187a-4fe2-b50c-9028b871a70f\") " Nov 01 15:49:09.515409 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:09.515319 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db59b0e4-187a-4fe2-b50c-9028b871a70f-kube-api-access-mb4lw\" (UniqueName: \"kubernetes.io/projected/db59b0e4-187a-4fe2-b50c-9028b871a70f-kube-api-access-mb4lw\") pod \"db59b0e4-187a-4fe2-b50c-9028b871a70f\" (UID: \"db59b0e4-187a-4fe2-b50c-9028b871a70f\") " Nov 01 15:49:09.515409 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:09.515335 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db59b0e4-187a-4fe2-b50c-9028b871a70f-console-config\" (UniqueName: \"kubernetes.io/configmap/db59b0e4-187a-4fe2-b50c-9028b871a70f-console-config\") pod \"db59b0e4-187a-4fe2-b50c-9028b871a70f\" (UID: \"db59b0e4-187a-4fe2-b50c-9028b871a70f\") " Nov 01 15:49:09.515409 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:09.515360 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db59b0e4-187a-4fe2-b50c-9028b871a70f-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db59b0e4-187a-4fe2-b50c-9028b871a70f-console-serving-cert\") pod \"db59b0e4-187a-4fe2-b50c-9028b871a70f\" (UID: \"db59b0e4-187a-4fe2-b50c-9028b871a70f\") " Nov 01 15:49:09.515685 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:09.515419 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db59b0e4-187a-4fe2-b50c-9028b871a70f-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db59b0e4-187a-4fe2-b50c-9028b871a70f-console-oauth-config\") pod \"db59b0e4-187a-4fe2-b50c-9028b871a70f\" (UID: \"db59b0e4-187a-4fe2-b50c-9028b871a70f\") " Nov 01 15:49:09.515685 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:09.515469 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db59b0e4-187a-4fe2-b50c-9028b871a70f-tmp" pod "db59b0e4-187a-4fe2-b50c-9028b871a70f" (UID: "db59b0e4-187a-4fe2-b50c-9028b871a70f"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:49:09.515685 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:09.515668 4363 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db59b0e4-187a-4fe2-b50c-9028b871a70f-tmp\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:49:09.515841 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:09.515735 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db59b0e4-187a-4fe2-b50c-9028b871a70f-oauth-serving-cert" pod "db59b0e4-187a-4fe2-b50c-9028b871a70f" (UID: "db59b0e4-187a-4fe2-b50c-9028b871a70f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:49:09.515841 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:09.515748 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db59b0e4-187a-4fe2-b50c-9028b871a70f-service-ca" pod "db59b0e4-187a-4fe2-b50c-9028b871a70f" (UID: "db59b0e4-187a-4fe2-b50c-9028b871a70f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:49:09.515841 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:09.515827 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db59b0e4-187a-4fe2-b50c-9028b871a70f-console-config" pod "db59b0e4-187a-4fe2-b50c-9028b871a70f" (UID: "db59b0e4-187a-4fe2-b50c-9028b871a70f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:49:09.517624 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:09.517595 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db59b0e4-187a-4fe2-b50c-9028b871a70f-kube-api-access-mb4lw" pod "db59b0e4-187a-4fe2-b50c-9028b871a70f" (UID: "db59b0e4-187a-4fe2-b50c-9028b871a70f"). InnerVolumeSpecName "kube-api-access-mb4lw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:49:09.517736 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:09.517595 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db59b0e4-187a-4fe2-b50c-9028b871a70f-console-serving-cert" pod "db59b0e4-187a-4fe2-b50c-9028b871a70f" (UID: "db59b0e4-187a-4fe2-b50c-9028b871a70f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:49:09.517736 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:09.517668 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db59b0e4-187a-4fe2-b50c-9028b871a70f-console-oauth-config" pod "db59b0e4-187a-4fe2-b50c-9028b871a70f" (UID: "db59b0e4-187a-4fe2-b50c-9028b871a70f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:49:09.616839 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:09.616805 4363 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db59b0e4-187a-4fe2-b50c-9028b871a70f-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:49:09.616839 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:09.616834 4363 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db59b0e4-187a-4fe2-b50c-9028b871a70f-oauth-serving-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:49:09.616839 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:09.616844 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb4lw\" (UniqueName: \"kubernetes.io/projected/db59b0e4-187a-4fe2-b50c-9028b871a70f-kube-api-access-mb4lw\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:49:09.617083 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:09.616855 4363 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db59b0e4-187a-4fe2-b50c-9028b871a70f-console-config\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:49:09.617083 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:09.616864 4363 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db59b0e4-187a-4fe2-b50c-9028b871a70f-console-serving-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:49:09.617083 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:09.616872 4363 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db59b0e4-187a-4fe2-b50c-9028b871a70f-console-oauth-config\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:49:10.183541 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:10.183503 4363 generic.go:356] "Generic (PLEG): container finished" podID="db59b0e4-187a-4fe2-b50c-9028b871a70f" containerID="97529ad126188abe4946944db13e9d4eb8c7ffacc4bb5024a72172c447a6bac4" exitCode=0 Nov 01 15:49:10.183923 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:10.183556 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68bb497dfc-xs9s9" event={"ID":"db59b0e4-187a-4fe2-b50c-9028b871a70f","Type":"ContainerDied","Data":"97529ad126188abe4946944db13e9d4eb8c7ffacc4bb5024a72172c447a6bac4"} Nov 01 15:49:10.183923 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:10.183579 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68bb497dfc-xs9s9" Nov 01 15:49:10.183923 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:10.183623 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68bb497dfc-xs9s9" event={"ID":"db59b0e4-187a-4fe2-b50c-9028b871a70f","Type":"ContainerDied","Data":"59c20282ccfec90db408643bfeb0a4af768e2e92220f6165c8b7dd3502c47ec7"} Nov 01 15:49:10.183923 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:10.183645 4363 scope.go:122] "RemoveContainer" containerID="97529ad126188abe4946944db13e9d4eb8c7ffacc4bb5024a72172c447a6bac4" Nov 01 15:49:10.193723 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:10.193704 4363 scope.go:122] "RemoveContainer" containerID="97529ad126188abe4946944db13e9d4eb8c7ffacc4bb5024a72172c447a6bac4" Nov 01 15:49:10.193972 ip-10-0-128-238 kubenswrapper[4363]: E1101 15:49:10.193953 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97529ad126188abe4946944db13e9d4eb8c7ffacc4bb5024a72172c447a6bac4\": container with ID starting with 97529ad126188abe4946944db13e9d4eb8c7ffacc4bb5024a72172c447a6bac4 not found: ID does not exist" containerID="97529ad126188abe4946944db13e9d4eb8c7ffacc4bb5024a72172c447a6bac4" Nov 01 15:49:10.194146 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:10.193980 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97529ad126188abe4946944db13e9d4eb8c7ffacc4bb5024a72172c447a6bac4"} err="failed to get container status \"97529ad126188abe4946944db13e9d4eb8c7ffacc4bb5024a72172c447a6bac4\": rpc error: code = NotFound desc = could not find container \"97529ad126188abe4946944db13e9d4eb8c7ffacc4bb5024a72172c447a6bac4\": container with ID starting with 97529ad126188abe4946944db13e9d4eb8c7ffacc4bb5024a72172c447a6bac4 not found: ID does not exist" Nov 01 15:49:10.204111 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:10.204089 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68bb497dfc-xs9s9"] Nov 01 15:49:10.206826 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:10.206803 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-68bb497dfc-xs9s9"] Nov 01 15:49:10.794926 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:10.794896 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db59b0e4-187a-4fe2-b50c-9028b871a70f" path="/var/lib/kubelet/pods/db59b0e4-187a-4fe2-b50c-9028b871a70f/volumes" Nov 01 15:49:11.282060 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:11.280220 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9b5784bf-hvxsb" Nov 01 15:49:11.282060 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:11.280298 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9b5784bf-hvxsb" Nov 01 15:49:11.287533 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:11.287510 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9b5784bf-hvxsb" Nov 01 15:49:12.195577 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:12.195551 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9b5784bf-hvxsb" Nov 01 15:49:12.241595 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:12.241566 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-855d4fbdf5-6kgtq"] Nov 01 15:49:18.308524 ip-10-0-128-238 kubenswrapper[4363]: E1101 15:49:18.308485 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 15:49:37.262702 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:37.262645 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-855d4fbdf5-6kgtq" podUID="bc7adf5a-3da8-4856-91a3-03b8677e2af7" containerName="console" containerID="cri-o://58c6d76585252e66073f2b17dd5efe03469656d0c42f18e666625f814dad3803" gracePeriod=15 Nov 01 15:49:37.493907 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:37.493883 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-855d4fbdf5-6kgtq" Nov 01 15:49:37.635080 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:37.635052 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc7adf5a-3da8-4856-91a3-03b8677e2af7-tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc7adf5a-3da8-4856-91a3-03b8677e2af7-tmp\") pod \"bc7adf5a-3da8-4856-91a3-03b8677e2af7\" (UID: \"bc7adf5a-3da8-4856-91a3-03b8677e2af7\") " Nov 01 15:49:37.635236 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:37.635094 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc7adf5a-3da8-4856-91a3-03b8677e2af7-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc7adf5a-3da8-4856-91a3-03b8677e2af7-console-serving-cert\") pod \"bc7adf5a-3da8-4856-91a3-03b8677e2af7\" (UID: \"bc7adf5a-3da8-4856-91a3-03b8677e2af7\") " Nov 01 15:49:37.635236 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:37.635168 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc7adf5a-3da8-4856-91a3-03b8677e2af7-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc7adf5a-3da8-4856-91a3-03b8677e2af7-service-ca\") pod \"bc7adf5a-3da8-4856-91a3-03b8677e2af7\" (UID: \"bc7adf5a-3da8-4856-91a3-03b8677e2af7\") " Nov 01 15:49:37.635236 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:37.635200 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc7adf5a-3da8-4856-91a3-03b8677e2af7-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc7adf5a-3da8-4856-91a3-03b8677e2af7-oauth-serving-cert\") pod \"bc7adf5a-3da8-4856-91a3-03b8677e2af7\" (UID: \"bc7adf5a-3da8-4856-91a3-03b8677e2af7\") " Nov 01 15:49:37.635407 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:37.635279 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc7adf5a-3da8-4856-91a3-03b8677e2af7-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc7adf5a-3da8-4856-91a3-03b8677e2af7-trusted-ca-bundle\") pod \"bc7adf5a-3da8-4856-91a3-03b8677e2af7\" (UID: \"bc7adf5a-3da8-4856-91a3-03b8677e2af7\") " Nov 01 15:49:37.635407 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:37.635309 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc7adf5a-3da8-4856-91a3-03b8677e2af7-kube-api-access-kwvvt\" (UniqueName: \"kubernetes.io/projected/bc7adf5a-3da8-4856-91a3-03b8677e2af7-kube-api-access-kwvvt\") pod \"bc7adf5a-3da8-4856-91a3-03b8677e2af7\" (UID: \"bc7adf5a-3da8-4856-91a3-03b8677e2af7\") " Nov 01 15:49:37.635407 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:37.635338 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc7adf5a-3da8-4856-91a3-03b8677e2af7-console-config\" (UniqueName: \"kubernetes.io/configmap/bc7adf5a-3da8-4856-91a3-03b8677e2af7-console-config\") pod \"bc7adf5a-3da8-4856-91a3-03b8677e2af7\" (UID: \"bc7adf5a-3da8-4856-91a3-03b8677e2af7\") " Nov 01 15:49:37.635407 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:37.635356 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc7adf5a-3da8-4856-91a3-03b8677e2af7-tmp" pod "bc7adf5a-3da8-4856-91a3-03b8677e2af7" (UID: "bc7adf5a-3da8-4856-91a3-03b8677e2af7"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:49:37.635407 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:37.635374 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc7adf5a-3da8-4856-91a3-03b8677e2af7-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc7adf5a-3da8-4856-91a3-03b8677e2af7-console-oauth-config\") pod \"bc7adf5a-3da8-4856-91a3-03b8677e2af7\" (UID: \"bc7adf5a-3da8-4856-91a3-03b8677e2af7\") " Nov 01 15:49:37.635657 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:37.635606 4363 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc7adf5a-3da8-4856-91a3-03b8677e2af7-tmp\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:49:37.635710 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:37.635680 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc7adf5a-3da8-4856-91a3-03b8677e2af7-service-ca" pod "bc7adf5a-3da8-4856-91a3-03b8677e2af7" (UID: "bc7adf5a-3da8-4856-91a3-03b8677e2af7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:49:37.635756 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:37.635711 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc7adf5a-3da8-4856-91a3-03b8677e2af7-oauth-serving-cert" pod "bc7adf5a-3da8-4856-91a3-03b8677e2af7" (UID: "bc7adf5a-3da8-4856-91a3-03b8677e2af7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:49:37.635756 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:37.635717 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc7adf5a-3da8-4856-91a3-03b8677e2af7-console-config" pod "bc7adf5a-3da8-4856-91a3-03b8677e2af7" (UID: "bc7adf5a-3da8-4856-91a3-03b8677e2af7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:49:37.635866 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:37.635770 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc7adf5a-3da8-4856-91a3-03b8677e2af7-trusted-ca-bundle" pod "bc7adf5a-3da8-4856-91a3-03b8677e2af7" (UID: "bc7adf5a-3da8-4856-91a3-03b8677e2af7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:49:37.637309 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:37.637279 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc7adf5a-3da8-4856-91a3-03b8677e2af7-console-serving-cert" pod "bc7adf5a-3da8-4856-91a3-03b8677e2af7" (UID: "bc7adf5a-3da8-4856-91a3-03b8677e2af7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:49:37.637459 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:37.637434 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc7adf5a-3da8-4856-91a3-03b8677e2af7-kube-api-access-kwvvt" pod "bc7adf5a-3da8-4856-91a3-03b8677e2af7" (UID: "bc7adf5a-3da8-4856-91a3-03b8677e2af7"). InnerVolumeSpecName "kube-api-access-kwvvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:49:37.637514 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:37.637496 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc7adf5a-3da8-4856-91a3-03b8677e2af7-console-oauth-config" pod "bc7adf5a-3da8-4856-91a3-03b8677e2af7" (UID: "bc7adf5a-3da8-4856-91a3-03b8677e2af7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:49:37.736032 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:37.735988 4363 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc7adf5a-3da8-4856-91a3-03b8677e2af7-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:49:37.736032 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:37.736029 4363 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc7adf5a-3da8-4856-91a3-03b8677e2af7-oauth-serving-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:49:37.736216 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:37.736043 4363 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc7adf5a-3da8-4856-91a3-03b8677e2af7-trusted-ca-bundle\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:49:37.736216 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:37.736053 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwvvt\" (UniqueName: \"kubernetes.io/projected/bc7adf5a-3da8-4856-91a3-03b8677e2af7-kube-api-access-kwvvt\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:49:37.736216 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:37.736062 4363 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc7adf5a-3da8-4856-91a3-03b8677e2af7-console-config\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:49:37.736216 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:37.736071 4363 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc7adf5a-3da8-4856-91a3-03b8677e2af7-console-oauth-config\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:49:37.736216 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:37.736080 4363 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc7adf5a-3da8-4856-91a3-03b8677e2af7-console-serving-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:49:38.281391 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:38.281354 4363 generic.go:356] "Generic (PLEG): container finished" podID="bc7adf5a-3da8-4856-91a3-03b8677e2af7" containerID="58c6d76585252e66073f2b17dd5efe03469656d0c42f18e666625f814dad3803" exitCode=0 Nov 01 15:49:38.281825 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:38.281400 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-855d4fbdf5-6kgtq" Nov 01 15:49:38.281825 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:38.281409 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-855d4fbdf5-6kgtq" event={"ID":"bc7adf5a-3da8-4856-91a3-03b8677e2af7","Type":"ContainerDied","Data":"58c6d76585252e66073f2b17dd5efe03469656d0c42f18e666625f814dad3803"} Nov 01 15:49:38.281825 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:38.281461 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-855d4fbdf5-6kgtq" event={"ID":"bc7adf5a-3da8-4856-91a3-03b8677e2af7","Type":"ContainerDied","Data":"809bab93bcfdf3936baac5f702ea3fc2c812f8e816620f48d765be260559e376"} Nov 01 15:49:38.281825 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:38.281489 4363 scope.go:122] "RemoveContainer" containerID="58c6d76585252e66073f2b17dd5efe03469656d0c42f18e666625f814dad3803" Nov 01 15:49:38.289591 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:38.289575 4363 scope.go:122] "RemoveContainer" containerID="58c6d76585252e66073f2b17dd5efe03469656d0c42f18e666625f814dad3803" Nov 01 15:49:38.289828 ip-10-0-128-238 kubenswrapper[4363]: E1101 15:49:38.289811 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58c6d76585252e66073f2b17dd5efe03469656d0c42f18e666625f814dad3803\": container with ID starting with 58c6d76585252e66073f2b17dd5efe03469656d0c42f18e666625f814dad3803 not found: ID does not exist" containerID="58c6d76585252e66073f2b17dd5efe03469656d0c42f18e666625f814dad3803" Nov 01 15:49:38.289882 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:38.289835 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58c6d76585252e66073f2b17dd5efe03469656d0c42f18e666625f814dad3803"} err="failed to get container status \"58c6d76585252e66073f2b17dd5efe03469656d0c42f18e666625f814dad3803\": rpc error: code = NotFound desc = could not find container \"58c6d76585252e66073f2b17dd5efe03469656d0c42f18e666625f814dad3803\": container with ID starting with 58c6d76585252e66073f2b17dd5efe03469656d0c42f18e666625f814dad3803 not found: ID does not exist" Nov 01 15:49:38.299468 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:38.299446 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-855d4fbdf5-6kgtq"] Nov 01 15:49:38.302351 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:38.302331 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-855d4fbdf5-6kgtq"] Nov 01 15:49:38.794631 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:49:38.794601 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc7adf5a-3da8-4856-91a3-03b8677e2af7" path="/var/lib/kubelet/pods/bc7adf5a-3da8-4856-91a3-03b8677e2af7/volumes" Nov 01 15:49:48.306583 ip-10-0-128-238 kubenswrapper[4363]: E1101 15:49:48.306539 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 15:50:18.307622 ip-10-0-128-238 kubenswrapper[4363]: E1101 15:50:18.307537 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 15:50:18.717177 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:18.717143 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-67647cbf64-wn65v"] Nov 01 15:50:18.717490 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:18.717471 4363 cpu_manager.go:414] "containerMap: removing container" podUID="bc7adf5a-3da8-4856-91a3-03b8677e2af7" containerName="console" Nov 01 15:50:18.717571 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:18.717492 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc7adf5a-3da8-4856-91a3-03b8677e2af7" containerName="console" Nov 01 15:50:18.717571 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:18.717516 4363 cpu_manager.go:414] "containerMap: removing container" podUID="db59b0e4-187a-4fe2-b50c-9028b871a70f" containerName="console" Nov 01 15:50:18.717571 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:18.717523 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db59b0e4-187a-4fe2-b50c-9028b871a70f" containerName="console" Nov 01 15:50:18.717716 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:18.717617 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="db59b0e4-187a-4fe2-b50c-9028b871a70f" containerName="console" Nov 01 15:50:18.717716 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:18.717635 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc7adf5a-3da8-4856-91a3-03b8677e2af7" containerName="console" Nov 01 15:50:18.719493 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:18.719472 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67647cbf64-wn65v" Nov 01 15:50:18.735374 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:18.735354 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67647cbf64-wn65v"] Nov 01 15:50:18.824967 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:18.824938 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4fdc9534-ef36-455b-9ea0-ea4bb28d28e6-console-serving-cert\") pod \"console-67647cbf64-wn65v\" (UID: \"4fdc9534-ef36-455b-9ea0-ea4bb28d28e6\") " pod="openshift-console/console-67647cbf64-wn65v" Nov 01 15:50:18.824967 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:18.824969 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4fdc9534-ef36-455b-9ea0-ea4bb28d28e6-tmp\") pod \"console-67647cbf64-wn65v\" (UID: \"4fdc9534-ef36-455b-9ea0-ea4bb28d28e6\") " pod="openshift-console/console-67647cbf64-wn65v" Nov 01 15:50:18.825167 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:18.825004 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4fdc9534-ef36-455b-9ea0-ea4bb28d28e6-console-config\") pod \"console-67647cbf64-wn65v\" (UID: \"4fdc9534-ef36-455b-9ea0-ea4bb28d28e6\") " pod="openshift-console/console-67647cbf64-wn65v" Nov 01 15:50:18.825167 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:18.825091 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4fdc9534-ef36-455b-9ea0-ea4bb28d28e6-service-ca\") pod \"console-67647cbf64-wn65v\" (UID: \"4fdc9534-ef36-455b-9ea0-ea4bb28d28e6\") " pod="openshift-console/console-67647cbf64-wn65v" Nov 01 15:50:18.825167 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:18.825113 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4fdc9534-ef36-455b-9ea0-ea4bb28d28e6-oauth-serving-cert\") pod \"console-67647cbf64-wn65v\" (UID: \"4fdc9534-ef36-455b-9ea0-ea4bb28d28e6\") " pod="openshift-console/console-67647cbf64-wn65v" Nov 01 15:50:18.825167 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:18.825132 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4fdc9534-ef36-455b-9ea0-ea4bb28d28e6-console-oauth-config\") pod \"console-67647cbf64-wn65v\" (UID: \"4fdc9534-ef36-455b-9ea0-ea4bb28d28e6\") " pod="openshift-console/console-67647cbf64-wn65v" Nov 01 15:50:18.825167 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:18.825146 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qp2z\" (UniqueName: \"kubernetes.io/projected/4fdc9534-ef36-455b-9ea0-ea4bb28d28e6-kube-api-access-2qp2z\") pod \"console-67647cbf64-wn65v\" (UID: \"4fdc9534-ef36-455b-9ea0-ea4bb28d28e6\") " pod="openshift-console/console-67647cbf64-wn65v" Nov 01 15:50:18.825322 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:18.825213 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fdc9534-ef36-455b-9ea0-ea4bb28d28e6-trusted-ca-bundle\") pod \"console-67647cbf64-wn65v\" (UID: \"4fdc9534-ef36-455b-9ea0-ea4bb28d28e6\") " pod="openshift-console/console-67647cbf64-wn65v" Nov 01 15:50:18.925953 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:18.925924 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4fdc9534-ef36-455b-9ea0-ea4bb28d28e6-console-serving-cert\") pod \"console-67647cbf64-wn65v\" (UID: \"4fdc9534-ef36-455b-9ea0-ea4bb28d28e6\") " pod="openshift-console/console-67647cbf64-wn65v" Nov 01 15:50:18.926165 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:18.925964 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4fdc9534-ef36-455b-9ea0-ea4bb28d28e6-tmp\") pod \"console-67647cbf64-wn65v\" (UID: \"4fdc9534-ef36-455b-9ea0-ea4bb28d28e6\") " pod="openshift-console/console-67647cbf64-wn65v" Nov 01 15:50:18.926165 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:18.926001 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4fdc9534-ef36-455b-9ea0-ea4bb28d28e6-console-config\") pod \"console-67647cbf64-wn65v\" (UID: \"4fdc9534-ef36-455b-9ea0-ea4bb28d28e6\") " pod="openshift-console/console-67647cbf64-wn65v" Nov 01 15:50:18.926165 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:18.926050 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4fdc9534-ef36-455b-9ea0-ea4bb28d28e6-service-ca\") pod \"console-67647cbf64-wn65v\" (UID: \"4fdc9534-ef36-455b-9ea0-ea4bb28d28e6\") " pod="openshift-console/console-67647cbf64-wn65v" Nov 01 15:50:18.926165 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:18.926075 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4fdc9534-ef36-455b-9ea0-ea4bb28d28e6-oauth-serving-cert\") pod \"console-67647cbf64-wn65v\" (UID: \"4fdc9534-ef36-455b-9ea0-ea4bb28d28e6\") " pod="openshift-console/console-67647cbf64-wn65v" Nov 01 15:50:18.926165 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:18.926100 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4fdc9534-ef36-455b-9ea0-ea4bb28d28e6-console-oauth-config\") pod \"console-67647cbf64-wn65v\" (UID: \"4fdc9534-ef36-455b-9ea0-ea4bb28d28e6\") " pod="openshift-console/console-67647cbf64-wn65v" Nov 01 15:50:18.926165 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:18.926121 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qp2z\" (UniqueName: \"kubernetes.io/projected/4fdc9534-ef36-455b-9ea0-ea4bb28d28e6-kube-api-access-2qp2z\") pod \"console-67647cbf64-wn65v\" (UID: \"4fdc9534-ef36-455b-9ea0-ea4bb28d28e6\") " pod="openshift-console/console-67647cbf64-wn65v" Nov 01 15:50:18.926463 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:18.926255 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fdc9534-ef36-455b-9ea0-ea4bb28d28e6-trusted-ca-bundle\") pod \"console-67647cbf64-wn65v\" (UID: \"4fdc9534-ef36-455b-9ea0-ea4bb28d28e6\") " pod="openshift-console/console-67647cbf64-wn65v" Nov 01 15:50:18.926463 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:18.926379 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4fdc9534-ef36-455b-9ea0-ea4bb28d28e6-tmp\") pod \"console-67647cbf64-wn65v\" (UID: \"4fdc9534-ef36-455b-9ea0-ea4bb28d28e6\") " pod="openshift-console/console-67647cbf64-wn65v" Nov 01 15:50:18.927061 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:18.927006 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4fdc9534-ef36-455b-9ea0-ea4bb28d28e6-service-ca\") pod \"console-67647cbf64-wn65v\" (UID: \"4fdc9534-ef36-455b-9ea0-ea4bb28d28e6\") " pod="openshift-console/console-67647cbf64-wn65v" Nov 01 15:50:18.927188 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:18.927101 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fdc9534-ef36-455b-9ea0-ea4bb28d28e6-trusted-ca-bundle\") pod \"console-67647cbf64-wn65v\" (UID: \"4fdc9534-ef36-455b-9ea0-ea4bb28d28e6\") " pod="openshift-console/console-67647cbf64-wn65v" Nov 01 15:50:18.927188 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:18.927139 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4fdc9534-ef36-455b-9ea0-ea4bb28d28e6-console-config\") pod \"console-67647cbf64-wn65v\" (UID: \"4fdc9534-ef36-455b-9ea0-ea4bb28d28e6\") " pod="openshift-console/console-67647cbf64-wn65v" Nov 01 15:50:18.927188 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:18.927139 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4fdc9534-ef36-455b-9ea0-ea4bb28d28e6-oauth-serving-cert\") pod \"console-67647cbf64-wn65v\" (UID: \"4fdc9534-ef36-455b-9ea0-ea4bb28d28e6\") " pod="openshift-console/console-67647cbf64-wn65v" Nov 01 15:50:18.928804 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:18.928779 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4fdc9534-ef36-455b-9ea0-ea4bb28d28e6-console-oauth-config\") pod \"console-67647cbf64-wn65v\" (UID: \"4fdc9534-ef36-455b-9ea0-ea4bb28d28e6\") " pod="openshift-console/console-67647cbf64-wn65v" Nov 01 15:50:18.928936 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:18.928914 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4fdc9534-ef36-455b-9ea0-ea4bb28d28e6-console-serving-cert\") pod \"console-67647cbf64-wn65v\" (UID: \"4fdc9534-ef36-455b-9ea0-ea4bb28d28e6\") " pod="openshift-console/console-67647cbf64-wn65v" Nov 01 15:50:18.935429 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:18.935411 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qp2z\" (UniqueName: \"kubernetes.io/projected/4fdc9534-ef36-455b-9ea0-ea4bb28d28e6-kube-api-access-2qp2z\") pod \"console-67647cbf64-wn65v\" (UID: \"4fdc9534-ef36-455b-9ea0-ea4bb28d28e6\") " pod="openshift-console/console-67647cbf64-wn65v" Nov 01 15:50:19.029288 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:19.029205 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67647cbf64-wn65v" Nov 01 15:50:19.316937 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:19.316862 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67647cbf64-wn65v"] Nov 01 15:50:19.320367 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:50:19.320340 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fdc9534_ef36_455b_9ea0_ea4bb28d28e6.slice/crio-c4cfcb329b142334eb09d47841f89623bbcc93a2e91b0d318a8e82196e16d60a WatchSource:0}: Error finding container c4cfcb329b142334eb09d47841f89623bbcc93a2e91b0d318a8e82196e16d60a: Status 404 returned error can't find the container with id c4cfcb329b142334eb09d47841f89623bbcc93a2e91b0d318a8e82196e16d60a Nov 01 15:50:19.430954 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:19.430916 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67647cbf64-wn65v" event={"ID":"4fdc9534-ef36-455b-9ea0-ea4bb28d28e6","Type":"ContainerStarted","Data":"900daf0307b2e7eff2460d34406257561b6c4088375dca58f80da25514262269"} Nov 01 15:50:19.431179 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:19.430968 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67647cbf64-wn65v" event={"ID":"4fdc9534-ef36-455b-9ea0-ea4bb28d28e6","Type":"ContainerStarted","Data":"c4cfcb329b142334eb09d47841f89623bbcc93a2e91b0d318a8e82196e16d60a"} Nov 01 15:50:19.452261 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:19.452068 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-67647cbf64-wn65v" podStartSLOduration=1.452050083 podStartE2EDuration="1.452050083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:50:19.451755668 +0000 UTC m=+152.783493319" watchObservedRunningTime="2026-11-01 15:50:19.452050083 +0000 UTC m=+152.783787733" Nov 01 15:50:29.029549 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:29.029511 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67647cbf64-wn65v" Nov 01 15:50:29.030051 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:29.029574 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67647cbf64-wn65v" Nov 01 15:50:29.034373 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:29.034350 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67647cbf64-wn65v" Nov 01 15:50:29.470507 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:29.470481 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67647cbf64-wn65v" Nov 01 15:50:29.517905 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:29.517876 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9b5784bf-hvxsb"] Nov 01 15:50:48.306690 ip-10-0-128-238 kubenswrapper[4363]: E1101 15:50:48.306655 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 15:50:54.537364 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:54.537319 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-9b5784bf-hvxsb" podUID="39d2972b-d3ff-4b75-a329-37827adb7c9d" containerName="console" containerID="cri-o://e449bf5c69cad0508e0939a2a41e6eb66b22a7420d1e223fe42eee2de9652f8e" gracePeriod=15 Nov 01 15:50:54.784789 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:54.784764 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9b5784bf-hvxsb" Nov 01 15:50:54.911565 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:54.911533 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39d2972b-d3ff-4b75-a329-37827adb7c9d-tmp\" (UniqueName: \"kubernetes.io/empty-dir/39d2972b-d3ff-4b75-a329-37827adb7c9d-tmp\") pod \"39d2972b-d3ff-4b75-a329-37827adb7c9d\" (UID: \"39d2972b-d3ff-4b75-a329-37827adb7c9d\") " Nov 01 15:50:54.911741 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:54.911582 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39d2972b-d3ff-4b75-a329-37827adb7c9d-service-ca\" (UniqueName: \"kubernetes.io/configmap/39d2972b-d3ff-4b75-a329-37827adb7c9d-service-ca\") pod \"39d2972b-d3ff-4b75-a329-37827adb7c9d\" (UID: \"39d2972b-d3ff-4b75-a329-37827adb7c9d\") " Nov 01 15:50:54.911741 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:54.911613 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39d2972b-d3ff-4b75-a329-37827adb7c9d-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39d2972b-d3ff-4b75-a329-37827adb7c9d-trusted-ca-bundle\") pod \"39d2972b-d3ff-4b75-a329-37827adb7c9d\" (UID: \"39d2972b-d3ff-4b75-a329-37827adb7c9d\") " Nov 01 15:50:54.911741 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:54.911661 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39d2972b-d3ff-4b75-a329-37827adb7c9d-console-config\" (UniqueName: \"kubernetes.io/configmap/39d2972b-d3ff-4b75-a329-37827adb7c9d-console-config\") pod \"39d2972b-d3ff-4b75-a329-37827adb7c9d\" (UID: \"39d2972b-d3ff-4b75-a329-37827adb7c9d\") " Nov 01 15:50:54.911741 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:54.911712 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39d2972b-d3ff-4b75-a329-37827adb7c9d-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/39d2972b-d3ff-4b75-a329-37827adb7c9d-console-serving-cert\") pod \"39d2972b-d3ff-4b75-a329-37827adb7c9d\" (UID: \"39d2972b-d3ff-4b75-a329-37827adb7c9d\") " Nov 01 15:50:54.911741 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:54.911738 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39d2972b-d3ff-4b75-a329-37827adb7c9d-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/39d2972b-d3ff-4b75-a329-37827adb7c9d-oauth-serving-cert\") pod \"39d2972b-d3ff-4b75-a329-37827adb7c9d\" (UID: \"39d2972b-d3ff-4b75-a329-37827adb7c9d\") " Nov 01 15:50:54.911980 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:54.911786 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39d2972b-d3ff-4b75-a329-37827adb7c9d-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/39d2972b-d3ff-4b75-a329-37827adb7c9d-console-oauth-config\") pod \"39d2972b-d3ff-4b75-a329-37827adb7c9d\" (UID: \"39d2972b-d3ff-4b75-a329-37827adb7c9d\") " Nov 01 15:50:54.911980 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:54.911816 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39d2972b-d3ff-4b75-a329-37827adb7c9d-tmp" pod "39d2972b-d3ff-4b75-a329-37827adb7c9d" (UID: "39d2972b-d3ff-4b75-a329-37827adb7c9d"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:50:54.911980 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:54.911820 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39d2972b-d3ff-4b75-a329-37827adb7c9d-kube-api-access-hgj4z\" (UniqueName: \"kubernetes.io/projected/39d2972b-d3ff-4b75-a329-37827adb7c9d-kube-api-access-hgj4z\") pod \"39d2972b-d3ff-4b75-a329-37827adb7c9d\" (UID: \"39d2972b-d3ff-4b75-a329-37827adb7c9d\") " Nov 01 15:50:54.912153 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:54.912072 4363 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39d2972b-d3ff-4b75-a329-37827adb7c9d-tmp\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:50:54.912153 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:54.912117 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39d2972b-d3ff-4b75-a329-37827adb7c9d-service-ca" pod "39d2972b-d3ff-4b75-a329-37827adb7c9d" (UID: "39d2972b-d3ff-4b75-a329-37827adb7c9d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:50:54.912253 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:54.912214 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39d2972b-d3ff-4b75-a329-37827adb7c9d-trusted-ca-bundle" pod "39d2972b-d3ff-4b75-a329-37827adb7c9d" (UID: "39d2972b-d3ff-4b75-a329-37827adb7c9d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:50:54.912376 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:54.912242 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39d2972b-d3ff-4b75-a329-37827adb7c9d-oauth-serving-cert" pod "39d2972b-d3ff-4b75-a329-37827adb7c9d" (UID: "39d2972b-d3ff-4b75-a329-37827adb7c9d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:50:54.912376 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:54.912246 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39d2972b-d3ff-4b75-a329-37827adb7c9d-console-config" pod "39d2972b-d3ff-4b75-a329-37827adb7c9d" (UID: "39d2972b-d3ff-4b75-a329-37827adb7c9d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:50:54.913893 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:54.913863 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39d2972b-d3ff-4b75-a329-37827adb7c9d-console-oauth-config" pod "39d2972b-d3ff-4b75-a329-37827adb7c9d" (UID: "39d2972b-d3ff-4b75-a329-37827adb7c9d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:50:54.914055 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:54.914011 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39d2972b-d3ff-4b75-a329-37827adb7c9d-kube-api-access-hgj4z" pod "39d2972b-d3ff-4b75-a329-37827adb7c9d" (UID: "39d2972b-d3ff-4b75-a329-37827adb7c9d"). InnerVolumeSpecName "kube-api-access-hgj4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:50:54.914206 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:54.914182 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39d2972b-d3ff-4b75-a329-37827adb7c9d-console-serving-cert" pod "39d2972b-d3ff-4b75-a329-37827adb7c9d" (UID: "39d2972b-d3ff-4b75-a329-37827adb7c9d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:50:55.012704 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:55.012667 4363 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/39d2972b-d3ff-4b75-a329-37827adb7c9d-console-config\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:50:55.012704 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:55.012699 4363 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/39d2972b-d3ff-4b75-a329-37827adb7c9d-console-serving-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:50:55.012704 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:55.012709 4363 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/39d2972b-d3ff-4b75-a329-37827adb7c9d-oauth-serving-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:50:55.012925 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:55.012717 4363 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/39d2972b-d3ff-4b75-a329-37827adb7c9d-console-oauth-config\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:50:55.012925 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:55.012726 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgj4z\" (UniqueName: \"kubernetes.io/projected/39d2972b-d3ff-4b75-a329-37827adb7c9d-kube-api-access-hgj4z\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:50:55.012925 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:55.012737 4363 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/39d2972b-d3ff-4b75-a329-37827adb7c9d-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:50:55.012925 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:55.012745 4363 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39d2972b-d3ff-4b75-a329-37827adb7c9d-trusted-ca-bundle\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:50:55.556681 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:55.556649 4363 generic.go:356] "Generic (PLEG): container finished" podID="39d2972b-d3ff-4b75-a329-37827adb7c9d" containerID="e449bf5c69cad0508e0939a2a41e6eb66b22a7420d1e223fe42eee2de9652f8e" exitCode=0 Nov 01 15:50:55.557083 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:55.556697 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9b5784bf-hvxsb" Nov 01 15:50:55.557083 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:55.556726 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9b5784bf-hvxsb" event={"ID":"39d2972b-d3ff-4b75-a329-37827adb7c9d","Type":"ContainerDied","Data":"e449bf5c69cad0508e0939a2a41e6eb66b22a7420d1e223fe42eee2de9652f8e"} Nov 01 15:50:55.557083 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:55.556766 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9b5784bf-hvxsb" event={"ID":"39d2972b-d3ff-4b75-a329-37827adb7c9d","Type":"ContainerDied","Data":"a85c1207ecc0337d0f38b6680aa931a1046063bd8bed088b3bd554829246b0a9"} Nov 01 15:50:55.557083 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:55.556785 4363 scope.go:122] "RemoveContainer" containerID="e449bf5c69cad0508e0939a2a41e6eb66b22a7420d1e223fe42eee2de9652f8e" Nov 01 15:50:55.565004 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:55.564988 4363 scope.go:122] "RemoveContainer" containerID="e449bf5c69cad0508e0939a2a41e6eb66b22a7420d1e223fe42eee2de9652f8e" Nov 01 15:50:55.565297 ip-10-0-128-238 kubenswrapper[4363]: E1101 15:50:55.565278 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e449bf5c69cad0508e0939a2a41e6eb66b22a7420d1e223fe42eee2de9652f8e\": container with ID starting with e449bf5c69cad0508e0939a2a41e6eb66b22a7420d1e223fe42eee2de9652f8e not found: ID does not exist" containerID="e449bf5c69cad0508e0939a2a41e6eb66b22a7420d1e223fe42eee2de9652f8e" Nov 01 15:50:55.565361 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:55.565306 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e449bf5c69cad0508e0939a2a41e6eb66b22a7420d1e223fe42eee2de9652f8e"} err="failed to get container status \"e449bf5c69cad0508e0939a2a41e6eb66b22a7420d1e223fe42eee2de9652f8e\": rpc error: code = NotFound desc = could not find container \"e449bf5c69cad0508e0939a2a41e6eb66b22a7420d1e223fe42eee2de9652f8e\": container with ID starting with e449bf5c69cad0508e0939a2a41e6eb66b22a7420d1e223fe42eee2de9652f8e not found: ID does not exist" Nov 01 15:50:55.576292 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:55.576272 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9b5784bf-hvxsb"] Nov 01 15:50:55.580221 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:55.580200 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9b5784bf-hvxsb"] Nov 01 15:50:56.794345 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:50:56.794316 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39d2972b-d3ff-4b75-a329-37827adb7c9d" path="/var/lib/kubelet/pods/39d2972b-d3ff-4b75-a329-37827adb7c9d/volumes" Nov 01 15:51:18.307366 ip-10-0-128-238 kubenswrapper[4363]: E1101 15:51:18.307326 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 15:51:48.306689 ip-10-0-128-238 kubenswrapper[4363]: E1101 15:51:48.306653 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 15:52:18.307089 ip-10-0-128-238 kubenswrapper[4363]: E1101 15:52:18.307040 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 15:52:26.646279 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:26.646242 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-vl7qf"] Nov 01 15:52:26.646636 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:26.646541 4363 cpu_manager.go:414] "containerMap: removing container" podUID="39d2972b-d3ff-4b75-a329-37827adb7c9d" containerName="console" Nov 01 15:52:26.646636 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:26.646551 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39d2972b-d3ff-4b75-a329-37827adb7c9d" containerName="console" Nov 01 15:52:26.646636 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:26.646613 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="39d2972b-d3ff-4b75-a329-37827adb7c9d" containerName="console" Nov 01 15:52:26.648277 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:26.648255 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vl7qf" Nov 01 15:52:26.653472 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:26.653429 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 01 15:52:26.653576 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:26.653490 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 01 15:52:26.653576 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:26.653560 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 01 15:52:26.656593 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:26.656572 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 01 15:52:26.656593 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:26.656574 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 01 15:52:26.656716 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:26.656600 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-k9xjt\"" Nov 01 15:52:26.656716 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:26.656618 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 01 15:52:26.665620 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:26.665594 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-vl7qf"] Nov 01 15:52:26.757595 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:26.753180 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-vl7qf\" (UID: \"a7f6b776-e53a-4ed9-8287-a2e1d615cef3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vl7qf" Nov 01 15:52:26.757595 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:26.753491 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6mkz\" (UniqueName: \"kubernetes.io/projected/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-kube-api-access-z6mkz\") pod \"istiod-openshift-gateway-7876bfcb74-vl7qf\" (UID: \"a7f6b776-e53a-4ed9-8287-a2e1d615cef3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vl7qf" Nov 01 15:52:26.757595 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:26.753613 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-vl7qf\" (UID: \"a7f6b776-e53a-4ed9-8287-a2e1d615cef3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vl7qf" Nov 01 15:52:26.757595 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:26.753649 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-vl7qf\" (UID: \"a7f6b776-e53a-4ed9-8287-a2e1d615cef3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vl7qf" Nov 01 15:52:26.757595 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:26.753687 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-vl7qf\" (UID: \"a7f6b776-e53a-4ed9-8287-a2e1d615cef3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vl7qf" Nov 01 15:52:26.757595 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:26.753759 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-vl7qf\" (UID: \"a7f6b776-e53a-4ed9-8287-a2e1d615cef3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vl7qf" Nov 01 15:52:26.757595 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:26.753793 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-vl7qf\" (UID: \"a7f6b776-e53a-4ed9-8287-a2e1d615cef3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vl7qf" Nov 01 15:52:26.854817 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:26.854789 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-vl7qf\" (UID: \"a7f6b776-e53a-4ed9-8287-a2e1d615cef3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vl7qf" Nov 01 15:52:26.854966 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:26.854824 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6mkz\" (UniqueName: \"kubernetes.io/projected/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-kube-api-access-z6mkz\") pod \"istiod-openshift-gateway-7876bfcb74-vl7qf\" (UID: \"a7f6b776-e53a-4ed9-8287-a2e1d615cef3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vl7qf" Nov 01 15:52:26.854966 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:26.854850 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-vl7qf\" (UID: \"a7f6b776-e53a-4ed9-8287-a2e1d615cef3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vl7qf" Nov 01 15:52:26.854966 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:26.854868 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-vl7qf\" (UID: \"a7f6b776-e53a-4ed9-8287-a2e1d615cef3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vl7qf" Nov 01 15:52:26.855162 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:26.854993 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-vl7qf\" (UID: \"a7f6b776-e53a-4ed9-8287-a2e1d615cef3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vl7qf" Nov 01 15:52:26.855162 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:26.855095 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-vl7qf\" (UID: \"a7f6b776-e53a-4ed9-8287-a2e1d615cef3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vl7qf" Nov 01 15:52:26.855162 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:26.855127 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-vl7qf\" (UID: \"a7f6b776-e53a-4ed9-8287-a2e1d615cef3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vl7qf" Nov 01 15:52:26.855684 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:26.855655 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-vl7qf\" (UID: \"a7f6b776-e53a-4ed9-8287-a2e1d615cef3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vl7qf" Nov 01 15:52:26.857307 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:26.857281 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-vl7qf\" (UID: \"a7f6b776-e53a-4ed9-8287-a2e1d615cef3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vl7qf" Nov 01 15:52:26.857475 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:26.857456 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-vl7qf\" (UID: \"a7f6b776-e53a-4ed9-8287-a2e1d615cef3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vl7qf" Nov 01 15:52:26.857535 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:26.857462 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-vl7qf\" (UID: \"a7f6b776-e53a-4ed9-8287-a2e1d615cef3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vl7qf" Nov 01 15:52:26.857630 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:26.857600 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-vl7qf\" (UID: \"a7f6b776-e53a-4ed9-8287-a2e1d615cef3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vl7qf" Nov 01 15:52:26.865655 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:26.865631 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-vl7qf\" (UID: \"a7f6b776-e53a-4ed9-8287-a2e1d615cef3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vl7qf" Nov 01 15:52:26.865965 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:26.865948 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6mkz\" (UniqueName: \"kubernetes.io/projected/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-kube-api-access-z6mkz\") pod \"istiod-openshift-gateway-7876bfcb74-vl7qf\" (UID: \"a7f6b776-e53a-4ed9-8287-a2e1d615cef3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vl7qf" Nov 01 15:52:26.958693 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:26.958648 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vl7qf" Nov 01 15:52:27.236279 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:27.233390 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-vl7qf"] Nov 01 15:52:27.238939 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:52:27.238911 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7f6b776_e53a_4ed9_8287_a2e1d615cef3.slice/crio-bdca7bae2d28154d1a3513a976757198d2af4201da9782199aea6051bb3c6249 WatchSource:0}: Error finding container bdca7bae2d28154d1a3513a976757198d2af4201da9782199aea6051bb3c6249: Status 404 returned error can't find the container with id bdca7bae2d28154d1a3513a976757198d2af4201da9782199aea6051bb3c6249 Nov 01 15:52:27.864123 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:27.864085 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vl7qf" event={"ID":"a7f6b776-e53a-4ed9-8287-a2e1d615cef3","Type":"ContainerStarted","Data":"bdca7bae2d28154d1a3513a976757198d2af4201da9782199aea6051bb3c6249"} Nov 01 15:52:30.199602 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:30.199572 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:52:30.199889 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:30.199632 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:52:30.880712 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:30.880675 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vl7qf" event={"ID":"a7f6b776-e53a-4ed9-8287-a2e1d615cef3","Type":"ContainerStarted","Data":"81f07906d69857b44a24b25ae3487f977b7215b4406f570f46b081355ec1785c"} Nov 01 15:52:30.880904 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:30.880749 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vl7qf" Nov 01 15:52:30.900840 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:30.900792 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vl7qf" podStartSLOduration=1.942674533 podStartE2EDuration="4.900777252s" totalImagesPullingTime="2.958102719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:52:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:52:30.899130487 +0000 UTC m=+284.230868139" watchObservedRunningTime="2026-11-01 15:52:30.900777252 +0000 UTC m=+284.232514902" Nov 01 15:52:31.885899 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:31.885868 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vl7qf" Nov 01 15:52:46.804390 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:46.804359 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtwhw_8254aafa-dd5b-4150-87e4-d165dab6bcb5/ovn-acl-logging/0.log" Nov 01 15:52:46.810235 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:46.810207 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtwhw_8254aafa-dd5b-4150-87e4-d165dab6bcb5/ovn-acl-logging/0.log" Nov 01 15:52:48.307199 ip-10-0-128-238 kubenswrapper[4363]: E1101 15:52:48.307156 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 15:52:58.692347 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:58.692312 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-l7sxc"] Nov 01 15:52:58.695746 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:58.695724 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-l7sxc" Nov 01 15:52:58.754923 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:58.754893 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-l7sxc"] Nov 01 15:52:58.794012 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:58.793983 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmmrt\" (UniqueName: \"kubernetes.io/projected/c90d6816-410a-4eca-8a89-efe5f0c25c68-kube-api-access-vmmrt\") pod \"opendatahub-operator-controller-manager-6d5d696d67-l7sxc\" (UID: \"c90d6816-410a-4eca-8a89-efe5f0c25c68\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-l7sxc" Nov 01 15:52:58.794012 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:58.794038 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c90d6816-410a-4eca-8a89-efe5f0c25c68-metrics-cert\") pod \"opendatahub-operator-controller-manager-6d5d696d67-l7sxc\" (UID: \"c90d6816-410a-4eca-8a89-efe5f0c25c68\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-l7sxc" Nov 01 15:52:58.794315 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:58.794153 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c90d6816-410a-4eca-8a89-efe5f0c25c68-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d5d696d67-l7sxc\" (UID: \"c90d6816-410a-4eca-8a89-efe5f0c25c68\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-l7sxc" Nov 01 15:52:58.794315 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:58.794211 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c90d6816-410a-4eca-8a89-efe5f0c25c68-tmp\") pod \"opendatahub-operator-controller-manager-6d5d696d67-l7sxc\" (UID: \"c90d6816-410a-4eca-8a89-efe5f0c25c68\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-l7sxc" Nov 01 15:52:58.794315 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:58.794242 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c90d6816-410a-4eca-8a89-efe5f0c25c68-manifests\") pod \"opendatahub-operator-controller-manager-6d5d696d67-l7sxc\" (UID: \"c90d6816-410a-4eca-8a89-efe5f0c25c68\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-l7sxc" Nov 01 15:52:58.794463 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:58.794419 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c90d6816-410a-4eca-8a89-efe5f0c25c68-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d5d696d67-l7sxc\" (UID: \"c90d6816-410a-4eca-8a89-efe5f0c25c68\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-l7sxc" Nov 01 15:52:58.797599 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:58.797577 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 01 15:52:58.797758 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:58.797709 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 01 15:52:58.801140 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:58.801121 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 01 15:52:58.801275 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:58.801262 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-4c5x6\"" Nov 01 15:52:58.801397 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:58.801382 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 01 15:52:58.801435 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:58.801414 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 01 15:52:58.894952 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:58.894922 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c90d6816-410a-4eca-8a89-efe5f0c25c68-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d5d696d67-l7sxc\" (UID: \"c90d6816-410a-4eca-8a89-efe5f0c25c68\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-l7sxc" Nov 01 15:52:58.895146 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:58.894990 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c90d6816-410a-4eca-8a89-efe5f0c25c68-tmp\") pod \"opendatahub-operator-controller-manager-6d5d696d67-l7sxc\" (UID: \"c90d6816-410a-4eca-8a89-efe5f0c25c68\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-l7sxc" Nov 01 15:52:58.895146 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:58.895126 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c90d6816-410a-4eca-8a89-efe5f0c25c68-manifests\") pod \"opendatahub-operator-controller-manager-6d5d696d67-l7sxc\" (UID: \"c90d6816-410a-4eca-8a89-efe5f0c25c68\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-l7sxc" Nov 01 15:52:58.895299 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:58.895152 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c90d6816-410a-4eca-8a89-efe5f0c25c68-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d5d696d67-l7sxc\" (UID: \"c90d6816-410a-4eca-8a89-efe5f0c25c68\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-l7sxc" Nov 01 15:52:58.895350 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:58.895332 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmmrt\" (UniqueName: \"kubernetes.io/projected/c90d6816-410a-4eca-8a89-efe5f0c25c68-kube-api-access-vmmrt\") pod \"opendatahub-operator-controller-manager-6d5d696d67-l7sxc\" (UID: \"c90d6816-410a-4eca-8a89-efe5f0c25c68\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-l7sxc" Nov 01 15:52:58.895417 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:58.895363 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c90d6816-410a-4eca-8a89-efe5f0c25c68-metrics-cert\") pod \"opendatahub-operator-controller-manager-6d5d696d67-l7sxc\" (UID: \"c90d6816-410a-4eca-8a89-efe5f0c25c68\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-l7sxc" Nov 01 15:52:58.895588 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:58.895564 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c90d6816-410a-4eca-8a89-efe5f0c25c68-manifests\") pod \"opendatahub-operator-controller-manager-6d5d696d67-l7sxc\" (UID: \"c90d6816-410a-4eca-8a89-efe5f0c25c68\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-l7sxc" Nov 01 15:52:58.897435 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:58.897408 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c90d6816-410a-4eca-8a89-efe5f0c25c68-tmp\") pod \"opendatahub-operator-controller-manager-6d5d696d67-l7sxc\" (UID: \"c90d6816-410a-4eca-8a89-efe5f0c25c68\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-l7sxc" Nov 01 15:52:58.897562 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:58.897488 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c90d6816-410a-4eca-8a89-efe5f0c25c68-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d5d696d67-l7sxc\" (UID: \"c90d6816-410a-4eca-8a89-efe5f0c25c68\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-l7sxc" Nov 01 15:52:58.897647 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:58.897630 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c90d6816-410a-4eca-8a89-efe5f0c25c68-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d5d696d67-l7sxc\" (UID: \"c90d6816-410a-4eca-8a89-efe5f0c25c68\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-l7sxc" Nov 01 15:52:58.897746 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:58.897729 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c90d6816-410a-4eca-8a89-efe5f0c25c68-metrics-cert\") pod \"opendatahub-operator-controller-manager-6d5d696d67-l7sxc\" (UID: \"c90d6816-410a-4eca-8a89-efe5f0c25c68\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-l7sxc" Nov 01 15:52:58.903942 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:58.903921 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmmrt\" (UniqueName: \"kubernetes.io/projected/c90d6816-410a-4eca-8a89-efe5f0c25c68-kube-api-access-vmmrt\") pod \"opendatahub-operator-controller-manager-6d5d696d67-l7sxc\" (UID: \"c90d6816-410a-4eca-8a89-efe5f0c25c68\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-l7sxc" Nov 01 15:52:59.010471 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:59.010401 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-l7sxc" Nov 01 15:52:59.300412 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:59.300309 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-l7sxc"] Nov 01 15:52:59.304061 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:52:59.304029 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc90d6816_410a_4eca_8a89_efe5f0c25c68.slice/crio-c11129da0ebb4aded135fd27d389fa0f839a1703132a1b2597c3ed6a73fe9173 WatchSource:0}: Error finding container c11129da0ebb4aded135fd27d389fa0f839a1703132a1b2597c3ed6a73fe9173: Status 404 returned error can't find the container with id c11129da0ebb4aded135fd27d389fa0f839a1703132a1b2597c3ed6a73fe9173 Nov 01 15:52:59.305893 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:59.305871 4363 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 15:52:59.981841 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:52:59.981797 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-l7sxc" event={"ID":"c90d6816-410a-4eca-8a89-efe5f0c25c68","Type":"ContainerStarted","Data":"c11129da0ebb4aded135fd27d389fa0f839a1703132a1b2597c3ed6a73fe9173"} Nov 01 15:53:02.997157 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:02.997074 4363 generic.go:356] "Generic (PLEG): container finished" podID="c90d6816-410a-4eca-8a89-efe5f0c25c68" containerID="37291e9ea332dadf6f5e0561e54e60e41f86ef3daadf345cbb02850db0d77a98" exitCode=0 Nov 01 15:53:02.997157 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:02.997144 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-l7sxc" event={"ID":"c90d6816-410a-4eca-8a89-efe5f0c25c68","Type":"ContainerDied","Data":"37291e9ea332dadf6f5e0561e54e60e41f86ef3daadf345cbb02850db0d77a98"} Nov 01 15:53:04.002250 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:04.002211 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-l7sxc" event={"ID":"c90d6816-410a-4eca-8a89-efe5f0c25c68","Type":"ContainerStarted","Data":"a2b7e22289ce903d7400a2cf87ea66624a3c44cffde2921479497c3984be59ad"} Nov 01 15:53:04.002684 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:04.002365 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-l7sxc" Nov 01 15:53:04.034389 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:04.034331 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-l7sxc" podStartSLOduration=2.272477853 podStartE2EDuration="6.034318673s" totalImagesPullingTime="3.355568632s" totalInitContainerRuntime="406.272188ms" isStatefulPod=false podCreationTimestamp="2026-11-01 15:52:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:53:04.032571936 +0000 UTC m=+317.364309586" watchObservedRunningTime="2026-11-01 15:53:04.034318673 +0000 UTC m=+317.366056322" Nov 01 15:53:15.008677 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:15.008647 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-l7sxc" Nov 01 15:53:18.307428 ip-10-0-128-238 kubenswrapper[4363]: E1101 15:53:18.307389 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 15:53:37.668649 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:37.668617 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6759f4f87c-mhx92"] Nov 01 15:53:37.674317 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:37.674293 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6759f4f87c-mhx92" Nov 01 15:53:37.682477 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:37.682451 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6759f4f87c-mhx92"] Nov 01 15:53:37.729907 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:37.729879 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 01 15:53:37.730194 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:37.730173 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 01 15:53:37.730365 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:37.730199 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-89t2n\"" Nov 01 15:53:37.821676 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:37.821646 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19531d99-b1bc-4632-9ced-ec214a278244-tmp\") pod \"kube-auth-proxy-6759f4f87c-mhx92\" (UID: \"19531d99-b1bc-4632-9ced-ec214a278244\") " pod="openshift-ingress/kube-auth-proxy-6759f4f87c-mhx92" Nov 01 15:53:37.821846 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:37.821686 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19531d99-b1bc-4632-9ced-ec214a278244-tls-certs\") pod \"kube-auth-proxy-6759f4f87c-mhx92\" (UID: \"19531d99-b1bc-4632-9ced-ec214a278244\") " pod="openshift-ingress/kube-auth-proxy-6759f4f87c-mhx92" Nov 01 15:53:37.821846 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:37.821729 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4knnf\" (UniqueName: \"kubernetes.io/projected/19531d99-b1bc-4632-9ced-ec214a278244-kube-api-access-4knnf\") pod \"kube-auth-proxy-6759f4f87c-mhx92\" (UID: \"19531d99-b1bc-4632-9ced-ec214a278244\") " pod="openshift-ingress/kube-auth-proxy-6759f4f87c-mhx92" Nov 01 15:53:37.922956 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:37.922870 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19531d99-b1bc-4632-9ced-ec214a278244-tls-certs\") pod \"kube-auth-proxy-6759f4f87c-mhx92\" (UID: \"19531d99-b1bc-4632-9ced-ec214a278244\") " pod="openshift-ingress/kube-auth-proxy-6759f4f87c-mhx92" Nov 01 15:53:37.923157 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:37.922956 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4knnf\" (UniqueName: \"kubernetes.io/projected/19531d99-b1bc-4632-9ced-ec214a278244-kube-api-access-4knnf\") pod \"kube-auth-proxy-6759f4f87c-mhx92\" (UID: \"19531d99-b1bc-4632-9ced-ec214a278244\") " pod="openshift-ingress/kube-auth-proxy-6759f4f87c-mhx92" Nov 01 15:53:37.923157 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:37.923009 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19531d99-b1bc-4632-9ced-ec214a278244-tmp\") pod \"kube-auth-proxy-6759f4f87c-mhx92\" (UID: \"19531d99-b1bc-4632-9ced-ec214a278244\") " pod="openshift-ingress/kube-auth-proxy-6759f4f87c-mhx92" Nov 01 15:53:37.925183 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:37.925159 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19531d99-b1bc-4632-9ced-ec214a278244-tmp\") pod \"kube-auth-proxy-6759f4f87c-mhx92\" (UID: \"19531d99-b1bc-4632-9ced-ec214a278244\") " pod="openshift-ingress/kube-auth-proxy-6759f4f87c-mhx92" Nov 01 15:53:37.925305 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:37.925251 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19531d99-b1bc-4632-9ced-ec214a278244-tls-certs\") pod \"kube-auth-proxy-6759f4f87c-mhx92\" (UID: \"19531d99-b1bc-4632-9ced-ec214a278244\") " pod="openshift-ingress/kube-auth-proxy-6759f4f87c-mhx92" Nov 01 15:53:37.936634 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:37.936611 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4knnf\" (UniqueName: \"kubernetes.io/projected/19531d99-b1bc-4632-9ced-ec214a278244-kube-api-access-4knnf\") pod \"kube-auth-proxy-6759f4f87c-mhx92\" (UID: \"19531d99-b1bc-4632-9ced-ec214a278244\") " pod="openshift-ingress/kube-auth-proxy-6759f4f87c-mhx92" Nov 01 15:53:37.987406 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:37.987379 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6759f4f87c-mhx92" Nov 01 15:53:38.264188 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:38.264076 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6759f4f87c-mhx92"] Nov 01 15:53:38.266806 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:53:38.266779 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19531d99_b1bc_4632_9ced_ec214a278244.slice/crio-a748c39d56d470f422bfa474e59a72f7da76884f902fad1a0bc5221decff5cad WatchSource:0}: Error finding container a748c39d56d470f422bfa474e59a72f7da76884f902fad1a0bc5221decff5cad: Status 404 returned error can't find the container with id a748c39d56d470f422bfa474e59a72f7da76884f902fad1a0bc5221decff5cad Nov 01 15:53:39.126905 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:39.126862 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6759f4f87c-mhx92" event={"ID":"19531d99-b1bc-4632-9ced-ec214a278244","Type":"ContainerStarted","Data":"a748c39d56d470f422bfa474e59a72f7da76884f902fad1a0bc5221decff5cad"} Nov 01 15:53:40.024590 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:40.024512 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-j7vgt"] Nov 01 15:53:40.030923 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:40.030897 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j7vgt" Nov 01 15:53:40.039933 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:40.039895 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 01 15:53:40.040145 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:40.040106 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-pq4x4\"" Nov 01 15:53:40.040270 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:40.040200 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 01 15:53:40.041383 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:40.041317 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-j7vgt"] Nov 01 15:53:40.042045 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:40.042010 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 01 15:53:40.142913 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:40.142879 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e85c52e-56da-4d66-913f-12bafed607c8-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-j7vgt\" (UID: \"2e85c52e-56da-4d66-913f-12bafed607c8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j7vgt" Nov 01 15:53:40.142913 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:40.142924 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2e85c52e-56da-4d66-913f-12bafed607c8-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-j7vgt\" (UID: \"2e85c52e-56da-4d66-913f-12bafed607c8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j7vgt" Nov 01 15:53:40.143422 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:40.143086 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29t2x\" (UniqueName: \"kubernetes.io/projected/2e85c52e-56da-4d66-913f-12bafed607c8-kube-api-access-29t2x\") pod \"kserve-module-controller-manager-7566cf5fc8-j7vgt\" (UID: \"2e85c52e-56da-4d66-913f-12bafed607c8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j7vgt" Nov 01 15:53:40.143422 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:40.143120 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e85c52e-56da-4d66-913f-12bafed607c8-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-j7vgt\" (UID: \"2e85c52e-56da-4d66-913f-12bafed607c8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j7vgt" Nov 01 15:53:40.243918 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:40.243871 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e85c52e-56da-4d66-913f-12bafed607c8-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-j7vgt\" (UID: \"2e85c52e-56da-4d66-913f-12bafed607c8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j7vgt" Nov 01 15:53:40.243918 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:40.243923 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2e85c52e-56da-4d66-913f-12bafed607c8-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-j7vgt\" (UID: \"2e85c52e-56da-4d66-913f-12bafed607c8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j7vgt" Nov 01 15:53:40.244198 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:40.244003 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29t2x\" (UniqueName: \"kubernetes.io/projected/2e85c52e-56da-4d66-913f-12bafed607c8-kube-api-access-29t2x\") pod \"kserve-module-controller-manager-7566cf5fc8-j7vgt\" (UID: \"2e85c52e-56da-4d66-913f-12bafed607c8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j7vgt" Nov 01 15:53:40.244198 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:40.244058 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e85c52e-56da-4d66-913f-12bafed607c8-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-j7vgt\" (UID: \"2e85c52e-56da-4d66-913f-12bafed607c8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j7vgt" Nov 01 15:53:40.244446 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:40.244427 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e85c52e-56da-4d66-913f-12bafed607c8-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-j7vgt\" (UID: \"2e85c52e-56da-4d66-913f-12bafed607c8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j7vgt" Nov 01 15:53:40.244586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:40.244563 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2e85c52e-56da-4d66-913f-12bafed607c8-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-j7vgt\" (UID: \"2e85c52e-56da-4d66-913f-12bafed607c8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j7vgt" Nov 01 15:53:40.244826 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:40.244805 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e85c52e-56da-4d66-913f-12bafed607c8-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-j7vgt\" (UID: \"2e85c52e-56da-4d66-913f-12bafed607c8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j7vgt" Nov 01 15:53:40.252480 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:40.252450 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29t2x\" (UniqueName: \"kubernetes.io/projected/2e85c52e-56da-4d66-913f-12bafed607c8-kube-api-access-29t2x\") pod \"kserve-module-controller-manager-7566cf5fc8-j7vgt\" (UID: \"2e85c52e-56da-4d66-913f-12bafed607c8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j7vgt" Nov 01 15:53:40.346067 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:40.346036 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j7vgt" Nov 01 15:53:40.647993 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:40.647911 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-j7vgt"] Nov 01 15:53:40.654418 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:53:40.654376 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e85c52e_56da_4d66_913f_12bafed607c8.slice/crio-aa7cfd29af949c57e1eebf404b1b0864de1df1336eba3bb023137f37c730622f WatchSource:0}: Error finding container aa7cfd29af949c57e1eebf404b1b0864de1df1336eba3bb023137f37c730622f: Status 404 returned error can't find the container with id aa7cfd29af949c57e1eebf404b1b0864de1df1336eba3bb023137f37c730622f Nov 01 15:53:41.135999 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:41.135962 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6759f4f87c-mhx92" event={"ID":"19531d99-b1bc-4632-9ced-ec214a278244","Type":"ContainerStarted","Data":"8ff9887a17ac8abe3f3c074235dd757a845298f3e2c3ca0e4891c544bbe17a20"} Nov 01 15:53:41.137155 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:41.137129 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j7vgt" event={"ID":"2e85c52e-56da-4d66-913f-12bafed607c8","Type":"ContainerStarted","Data":"aa7cfd29af949c57e1eebf404b1b0864de1df1336eba3bb023137f37c730622f"} Nov 01 15:53:41.151283 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:41.151240 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6759f4f87c-mhx92" podStartSLOduration=1.5862031170000002 podStartE2EDuration="4.151227132s" totalImagesPullingTime="2.565024015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:53:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:53:41.149686222 +0000 UTC m=+354.481423873" watchObservedRunningTime="2026-11-01 15:53:41.151227132 +0000 UTC m=+354.482964796" Nov 01 15:53:43.148244 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:43.148210 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j7vgt" event={"ID":"2e85c52e-56da-4d66-913f-12bafed607c8","Type":"ContainerStarted","Data":"a98b69b708c98983a560c9c68c9ecbd6aa1434861664e6ddc64143b8ff8e9435"} Nov 01 15:53:43.148649 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:43.148299 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j7vgt" Nov 01 15:53:43.165795 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:43.165751 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j7vgt" podStartSLOduration=1.475246751 podStartE2EDuration="3.165737708s" totalImagesPullingTime="1.690490957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:53:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:53:43.165106964 +0000 UTC m=+356.496844614" watchObservedRunningTime="2026-11-01 15:53:43.165737708 +0000 UTC m=+356.497475360" Nov 01 15:53:45.939459 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:45.939423 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v8vcm"] Nov 01 15:53:45.942811 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:45.942791 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v8vcm" Nov 01 15:53:45.948407 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:45.948386 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 01 15:53:45.951138 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:45.951118 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-r57c6\"" Nov 01 15:53:45.962556 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:45.962534 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v8vcm"] Nov 01 15:53:45.989005 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:45.988975 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7dqx\" (UniqueName: \"kubernetes.io/projected/5a739d41-eca2-41b0-8758-d8236e0eccd6-kube-api-access-c7dqx\") pod \"kuadrant-operator-controller-manager-678d87766f-v8vcm\" (UID: \"5a739d41-eca2-41b0-8758-d8236e0eccd6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v8vcm" Nov 01 15:53:45.989116 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:45.989039 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5a739d41-eca2-41b0-8758-d8236e0eccd6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-v8vcm\" (UID: \"5a739d41-eca2-41b0-8758-d8236e0eccd6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v8vcm" Nov 01 15:53:46.047077 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:46.047056 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 01 15:53:46.089480 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:46.089448 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5a739d41-eca2-41b0-8758-d8236e0eccd6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-v8vcm\" (UID: \"5a739d41-eca2-41b0-8758-d8236e0eccd6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v8vcm" Nov 01 15:53:46.089648 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:46.089525 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7dqx\" (UniqueName: \"kubernetes.io/projected/5a739d41-eca2-41b0-8758-d8236e0eccd6-kube-api-access-c7dqx\") pod \"kuadrant-operator-controller-manager-678d87766f-v8vcm\" (UID: \"5a739d41-eca2-41b0-8758-d8236e0eccd6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v8vcm" Nov 01 15:53:46.089824 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:46.089805 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5a739d41-eca2-41b0-8758-d8236e0eccd6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-v8vcm\" (UID: \"5a739d41-eca2-41b0-8758-d8236e0eccd6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v8vcm" Nov 01 15:53:46.097496 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:46.097468 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7dqx\" (UniqueName: \"kubernetes.io/projected/5a739d41-eca2-41b0-8758-d8236e0eccd6-kube-api-access-c7dqx\") pod \"kuadrant-operator-controller-manager-678d87766f-v8vcm\" (UID: \"5a739d41-eca2-41b0-8758-d8236e0eccd6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v8vcm" Nov 01 15:53:46.255728 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:46.255638 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v8vcm" Nov 01 15:53:46.532356 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:46.532267 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v8vcm"] Nov 01 15:53:46.535680 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:53:46.535646 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a739d41_eca2_41b0_8758_d8236e0eccd6.slice/crio-2dbe4c678277fc6fe5a432734b97e6b4f2fb990bf4284f6b71433014047cd490 WatchSource:0}: Error finding container 2dbe4c678277fc6fe5a432734b97e6b4f2fb990bf4284f6b71433014047cd490: Status 404 returned error can't find the container with id 2dbe4c678277fc6fe5a432734b97e6b4f2fb990bf4284f6b71433014047cd490 Nov 01 15:53:47.164601 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:47.164571 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v8vcm" event={"ID":"5a739d41-eca2-41b0-8758-d8236e0eccd6","Type":"ContainerStarted","Data":"2dbe4c678277fc6fe5a432734b97e6b4f2fb990bf4284f6b71433014047cd490"} Nov 01 15:53:48.312935 ip-10-0-128-238 kubenswrapper[4363]: E1101 15:53:48.312899 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 15:53:51.193541 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:51.193499 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v8vcm" event={"ID":"5a739d41-eca2-41b0-8758-d8236e0eccd6","Type":"ContainerStarted","Data":"d4039a5ba251de67806e5dc589592eecc9e830cb9b0c2d32416441847aa8ae72"} Nov 01 15:53:51.194038 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:51.193675 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v8vcm" Nov 01 15:53:51.213747 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:53:51.213699 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v8vcm" podStartSLOduration=2.02498275 podStartE2EDuration="6.2136855s" totalImagesPullingTime="4.18870275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:53:51.212069466 +0000 UTC m=+364.543807115" watchObservedRunningTime="2026-11-01 15:53:51.2136855 +0000 UTC m=+364.545423149" Nov 01 15:54:02.200830 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:02.200801 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v8vcm" Nov 01 15:54:04.153652 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:04.153620 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j7vgt" Nov 01 15:54:18.308112 ip-10-0-128-238 kubenswrapper[4363]: E1101 15:54:18.308073 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 15:54:32.769737 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:32.769651 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9q8c"] Nov 01 15:54:32.771977 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:32.771946 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9q8c" Nov 01 15:54:32.784935 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:32.784911 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9q8c"] Nov 01 15:54:32.878521 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:32.878490 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9208aac6-2f13-4f8b-ac7a-3453908209be-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9q8c\" (UID: \"9208aac6-2f13-4f8b-ac7a-3453908209be\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9q8c" Nov 01 15:54:32.878672 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:32.878534 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9208aac6-2f13-4f8b-ac7a-3453908209be-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9q8c\" (UID: \"9208aac6-2f13-4f8b-ac7a-3453908209be\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9q8c" Nov 01 15:54:32.878672 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:32.878627 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zplm\" (UniqueName: \"kubernetes.io/projected/9208aac6-2f13-4f8b-ac7a-3453908209be-kube-api-access-8zplm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9q8c\" (UID: \"9208aac6-2f13-4f8b-ac7a-3453908209be\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9q8c" Nov 01 15:54:32.878746 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:32.878672 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9208aac6-2f13-4f8b-ac7a-3453908209be-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9q8c\" (UID: \"9208aac6-2f13-4f8b-ac7a-3453908209be\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9q8c" Nov 01 15:54:32.878746 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:32.878721 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9208aac6-2f13-4f8b-ac7a-3453908209be-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9q8c\" (UID: \"9208aac6-2f13-4f8b-ac7a-3453908209be\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9q8c" Nov 01 15:54:32.878805 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:32.878747 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9208aac6-2f13-4f8b-ac7a-3453908209be-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9q8c\" (UID: \"9208aac6-2f13-4f8b-ac7a-3453908209be\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9q8c" Nov 01 15:54:32.878840 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:32.878817 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9208aac6-2f13-4f8b-ac7a-3453908209be-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9q8c\" (UID: \"9208aac6-2f13-4f8b-ac7a-3453908209be\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9q8c" Nov 01 15:54:32.979435 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:32.979407 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zplm\" (UniqueName: \"kubernetes.io/projected/9208aac6-2f13-4f8b-ac7a-3453908209be-kube-api-access-8zplm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9q8c\" (UID: \"9208aac6-2f13-4f8b-ac7a-3453908209be\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9q8c" Nov 01 15:54:32.979599 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:32.979446 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9208aac6-2f13-4f8b-ac7a-3453908209be-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9q8c\" (UID: \"9208aac6-2f13-4f8b-ac7a-3453908209be\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9q8c" Nov 01 15:54:32.979599 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:32.979463 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9208aac6-2f13-4f8b-ac7a-3453908209be-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9q8c\" (UID: \"9208aac6-2f13-4f8b-ac7a-3453908209be\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9q8c" Nov 01 15:54:32.979599 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:32.979484 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9208aac6-2f13-4f8b-ac7a-3453908209be-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9q8c\" (UID: \"9208aac6-2f13-4f8b-ac7a-3453908209be\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9q8c" Nov 01 15:54:32.979599 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:32.979504 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9208aac6-2f13-4f8b-ac7a-3453908209be-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9q8c\" (UID: \"9208aac6-2f13-4f8b-ac7a-3453908209be\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9q8c" Nov 01 15:54:32.979599 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:32.979535 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9208aac6-2f13-4f8b-ac7a-3453908209be-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9q8c\" (UID: \"9208aac6-2f13-4f8b-ac7a-3453908209be\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9q8c" Nov 01 15:54:32.979599 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:32.979554 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9208aac6-2f13-4f8b-ac7a-3453908209be-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9q8c\" (UID: \"9208aac6-2f13-4f8b-ac7a-3453908209be\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9q8c" Nov 01 15:54:32.980437 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:32.980408 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9208aac6-2f13-4f8b-ac7a-3453908209be-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9q8c\" (UID: \"9208aac6-2f13-4f8b-ac7a-3453908209be\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9q8c" Nov 01 15:54:32.981914 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:32.981890 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9208aac6-2f13-4f8b-ac7a-3453908209be-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9q8c\" (UID: \"9208aac6-2f13-4f8b-ac7a-3453908209be\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9q8c" Nov 01 15:54:32.982051 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:32.981997 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9208aac6-2f13-4f8b-ac7a-3453908209be-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9q8c\" (UID: \"9208aac6-2f13-4f8b-ac7a-3453908209be\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9q8c" Nov 01 15:54:32.982119 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:32.982073 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9208aac6-2f13-4f8b-ac7a-3453908209be-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9q8c\" (UID: \"9208aac6-2f13-4f8b-ac7a-3453908209be\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9q8c" Nov 01 15:54:32.982119 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:32.982086 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9208aac6-2f13-4f8b-ac7a-3453908209be-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9q8c\" (UID: \"9208aac6-2f13-4f8b-ac7a-3453908209be\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9q8c" Nov 01 15:54:32.989720 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:32.989699 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9208aac6-2f13-4f8b-ac7a-3453908209be-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9q8c\" (UID: \"9208aac6-2f13-4f8b-ac7a-3453908209be\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9q8c" Nov 01 15:54:32.998815 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:32.998790 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zplm\" (UniqueName: \"kubernetes.io/projected/9208aac6-2f13-4f8b-ac7a-3453908209be-kube-api-access-8zplm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9q8c\" (UID: \"9208aac6-2f13-4f8b-ac7a-3453908209be\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9q8c" Nov 01 15:54:33.083535 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:33.083459 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9q8c" Nov 01 15:54:33.357185 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:33.357159 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9q8c"] Nov 01 15:54:33.359703 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:54:33.359676 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9208aac6_2f13_4f8b_ac7a_3453908209be.slice/crio-3b10679d667d956d47e1497fb25a351fa2f705c5c776f96ece27e832a0053dd5 WatchSource:0}: Error finding container 3b10679d667d956d47e1497fb25a351fa2f705c5c776f96ece27e832a0053dd5: Status 404 returned error can't find the container with id 3b10679d667d956d47e1497fb25a351fa2f705c5c776f96ece27e832a0053dd5 Nov 01 15:54:33.361929 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:33.361894 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:54:33.362036 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:33.361962 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:54:33.365460 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:33.365432 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9q8c" event={"ID":"9208aac6-2f13-4f8b-ac7a-3453908209be","Type":"ContainerStarted","Data":"3b10679d667d956d47e1497fb25a351fa2f705c5c776f96ece27e832a0053dd5"} Nov 01 15:54:34.371368 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:34.371334 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9q8c" event={"ID":"9208aac6-2f13-4f8b-ac7a-3453908209be","Type":"ContainerStarted","Data":"8694853f234a5834b8a6ef951cbd149d9a123c678d44894998ff2aee4305fff5"} Nov 01 15:54:34.371833 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:34.371526 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9q8c" Nov 01 15:54:34.373226 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:34.373205 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9q8c" Nov 01 15:54:34.392561 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:34.392508 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9q8c" podStartSLOduration=2.392493244 podStartE2EDuration="2.392493244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:54:34.391680489 +0000 UTC m=+407.723418152" watchObservedRunningTime="2026-11-01 15:54:34.392493244 +0000 UTC m=+407.724230894" Nov 01 15:54:34.438027 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:34.437984 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-vl7qf"] Nov 01 15:54:34.438235 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:34.438214 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vl7qf" podUID="a7f6b776-e53a-4ed9-8287-a2e1d615cef3" containerName="discovery" containerID="cri-o://81f07906d69857b44a24b25ae3487f977b7215b4406f570f46b081355ec1785c" gracePeriod=30 Nov 01 15:54:34.678126 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:34.678100 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vl7qf" Nov 01 15:54:34.795130 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:34.795098 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-istio-token\" (UniqueName: \"kubernetes.io/projected/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-istio-token\") pod \"a7f6b776-e53a-4ed9-8287-a2e1d615cef3\" (UID: \"a7f6b776-e53a-4ed9-8287-a2e1d615cef3\") " Nov 01 15:54:34.795285 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:34.795170 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-istio-csr-dns-cert\") pod \"a7f6b776-e53a-4ed9-8287-a2e1d615cef3\" (UID: \"a7f6b776-e53a-4ed9-8287-a2e1d615cef3\") " Nov 01 15:54:34.795285 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:34.795231 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-kube-api-access-z6mkz\" (UniqueName: \"kubernetes.io/projected/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-kube-api-access-z6mkz\") pod \"a7f6b776-e53a-4ed9-8287-a2e1d615cef3\" (UID: \"a7f6b776-e53a-4ed9-8287-a2e1d615cef3\") " Nov 01 15:54:34.795361 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:34.795292 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-local-certs\") pod \"a7f6b776-e53a-4ed9-8287-a2e1d615cef3\" (UID: \"a7f6b776-e53a-4ed9-8287-a2e1d615cef3\") " Nov 01 15:54:34.795361 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:34.795332 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-istio-kubeconfig\") pod \"a7f6b776-e53a-4ed9-8287-a2e1d615cef3\" (UID: \"a7f6b776-e53a-4ed9-8287-a2e1d615cef3\") " Nov 01 15:54:34.795426 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:34.795365 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-istio-csr-ca-configmap\") pod \"a7f6b776-e53a-4ed9-8287-a2e1d615cef3\" (UID: \"a7f6b776-e53a-4ed9-8287-a2e1d615cef3\") " Nov 01 15:54:34.795426 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:34.795399 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-cacerts\" (UniqueName: \"kubernetes.io/secret/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-cacerts\") pod \"a7f6b776-e53a-4ed9-8287-a2e1d615cef3\" (UID: \"a7f6b776-e53a-4ed9-8287-a2e1d615cef3\") " Nov 01 15:54:34.796053 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:34.795970 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-istio-csr-ca-configmap" pod "a7f6b776-e53a-4ed9-8287-a2e1d615cef3" (UID: "a7f6b776-e53a-4ed9-8287-a2e1d615cef3"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:54:34.797711 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:34.797668 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-istio-token" pod "a7f6b776-e53a-4ed9-8287-a2e1d615cef3" (UID: "a7f6b776-e53a-4ed9-8287-a2e1d615cef3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:54:34.797942 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:34.797914 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-kube-api-access-z6mkz" pod "a7f6b776-e53a-4ed9-8287-a2e1d615cef3" (UID: "a7f6b776-e53a-4ed9-8287-a2e1d615cef3"). InnerVolumeSpecName "kube-api-access-z6mkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:54:34.798083 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:34.797943 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-local-certs" pod "a7f6b776-e53a-4ed9-8287-a2e1d615cef3" (UID: "a7f6b776-e53a-4ed9-8287-a2e1d615cef3"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:54:34.798083 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:34.798001 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-istio-csr-dns-cert" pod "a7f6b776-e53a-4ed9-8287-a2e1d615cef3" (UID: "a7f6b776-e53a-4ed9-8287-a2e1d615cef3"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:54:34.798083 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:34.798064 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-istio-kubeconfig" pod "a7f6b776-e53a-4ed9-8287-a2e1d615cef3" (UID: "a7f6b776-e53a-4ed9-8287-a2e1d615cef3"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:54:34.798194 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:34.798136 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-cacerts" pod "a7f6b776-e53a-4ed9-8287-a2e1d615cef3" (UID: "a7f6b776-e53a-4ed9-8287-a2e1d615cef3"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:54:34.896927 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:34.896887 4363 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-cacerts\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:54:34.896927 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:34.896916 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:54:34.896927 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:34.896925 4363 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-istio-csr-dns-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:54:34.896927 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:34.896935 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6mkz\" (UniqueName: \"kubernetes.io/projected/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-kube-api-access-z6mkz\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:54:34.897197 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:34.896943 4363 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-local-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:54:34.897197 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:34.896953 4363 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-istio-kubeconfig\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:54:34.897197 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:34.896963 4363 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a7f6b776-e53a-4ed9-8287-a2e1d615cef3-istio-csr-ca-configmap\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:54:35.377265 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:35.377232 4363 generic.go:356] "Generic (PLEG): container finished" podID="a7f6b776-e53a-4ed9-8287-a2e1d615cef3" containerID="81f07906d69857b44a24b25ae3487f977b7215b4406f570f46b081355ec1785c" exitCode=0 Nov 01 15:54:35.377686 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:35.377275 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vl7qf" Nov 01 15:54:35.377686 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:35.377313 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vl7qf" event={"ID":"a7f6b776-e53a-4ed9-8287-a2e1d615cef3","Type":"ContainerDied","Data":"81f07906d69857b44a24b25ae3487f977b7215b4406f570f46b081355ec1785c"} Nov 01 15:54:35.377686 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:35.377363 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vl7qf" event={"ID":"a7f6b776-e53a-4ed9-8287-a2e1d615cef3","Type":"ContainerDied","Data":"bdca7bae2d28154d1a3513a976757198d2af4201da9782199aea6051bb3c6249"} Nov 01 15:54:35.377686 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:35.377383 4363 scope.go:122] "RemoveContainer" containerID="81f07906d69857b44a24b25ae3487f977b7215b4406f570f46b081355ec1785c" Nov 01 15:54:35.386256 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:35.386237 4363 scope.go:122] "RemoveContainer" containerID="81f07906d69857b44a24b25ae3487f977b7215b4406f570f46b081355ec1785c" Nov 01 15:54:35.386519 ip-10-0-128-238 kubenswrapper[4363]: E1101 15:54:35.386492 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81f07906d69857b44a24b25ae3487f977b7215b4406f570f46b081355ec1785c\": container with ID starting with 81f07906d69857b44a24b25ae3487f977b7215b4406f570f46b081355ec1785c not found: ID does not exist" containerID="81f07906d69857b44a24b25ae3487f977b7215b4406f570f46b081355ec1785c" Nov 01 15:54:35.386611 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:35.386526 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81f07906d69857b44a24b25ae3487f977b7215b4406f570f46b081355ec1785c"} err="failed to get container status \"81f07906d69857b44a24b25ae3487f977b7215b4406f570f46b081355ec1785c\": rpc error: code = NotFound desc = could not find container \"81f07906d69857b44a24b25ae3487f977b7215b4406f570f46b081355ec1785c\": container with ID starting with 81f07906d69857b44a24b25ae3487f977b7215b4406f570f46b081355ec1785c not found: ID does not exist" Nov 01 15:54:35.402160 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:35.402138 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-vl7qf"] Nov 01 15:54:35.412995 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:35.412974 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-vl7qf"] Nov 01 15:54:36.795242 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:54:36.795208 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7f6b776-e53a-4ed9-8287-a2e1d615cef3" path="/var/lib/kubelet/pods/a7f6b776-e53a-4ed9-8287-a2e1d615cef3/volumes" Nov 01 15:54:48.307753 ip-10-0-128-238 kubenswrapper[4363]: E1101 15:54:48.307719 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 15:55:05.163233 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:55:05.163195 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-655f6fc5df-b8pd9"] Nov 01 15:55:05.163701 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:55:05.163502 4363 cpu_manager.go:414] "containerMap: removing container" podUID="a7f6b776-e53a-4ed9-8287-a2e1d615cef3" containerName="discovery" Nov 01 15:55:05.163701 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:55:05.163513 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7f6b776-e53a-4ed9-8287-a2e1d615cef3" containerName="discovery" Nov 01 15:55:05.163701 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:55:05.163581 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7f6b776-e53a-4ed9-8287-a2e1d615cef3" containerName="discovery" Nov 01 15:55:05.166640 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:55:05.166619 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-655f6fc5df-b8pd9" Nov 01 15:55:05.170189 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:55:05.170168 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 15:55:05.170308 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:55:05.170276 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 15:55:05.175625 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:55:05.175597 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-655f6fc5df-b8pd9"] Nov 01 15:55:05.222263 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:55:05.222232 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 01 15:55:05.222431 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:55:05.222316 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-l5k8h\"" Nov 01 15:55:05.222431 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:55:05.222398 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 15:55:05.233830 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:55:05.233807 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk2km\" (UniqueName: \"kubernetes.io/projected/086dd0e1-6a4f-45dd-9b53-f78094bda8ab-kube-api-access-kk2km\") pod \"maas-api-655f6fc5df-b8pd9\" (UID: \"086dd0e1-6a4f-45dd-9b53-f78094bda8ab\") " pod="odh-ai-gateway-infra/maas-api-655f6fc5df-b8pd9" Nov 01 15:55:05.233909 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:55:05.233840 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/086dd0e1-6a4f-45dd-9b53-f78094bda8ab-metrics-certs\") pod \"maas-api-655f6fc5df-b8pd9\" (UID: \"086dd0e1-6a4f-45dd-9b53-f78094bda8ab\") " pod="odh-ai-gateway-infra/maas-api-655f6fc5df-b8pd9" Nov 01 15:55:05.233909 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:55:05.233859 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/086dd0e1-6a4f-45dd-9b53-f78094bda8ab-maas-api-tls\") pod \"maas-api-655f6fc5df-b8pd9\" (UID: \"086dd0e1-6a4f-45dd-9b53-f78094bda8ab\") " pod="odh-ai-gateway-infra/maas-api-655f6fc5df-b8pd9" Nov 01 15:55:05.233909 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:55:05.233887 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/086dd0e1-6a4f-45dd-9b53-f78094bda8ab-openshift-service-ca\") pod \"maas-api-655f6fc5df-b8pd9\" (UID: \"086dd0e1-6a4f-45dd-9b53-f78094bda8ab\") " pod="odh-ai-gateway-infra/maas-api-655f6fc5df-b8pd9" Nov 01 15:55:05.334254 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:55:05.334219 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/086dd0e1-6a4f-45dd-9b53-f78094bda8ab-openshift-service-ca\") pod \"maas-api-655f6fc5df-b8pd9\" (UID: \"086dd0e1-6a4f-45dd-9b53-f78094bda8ab\") " pod="odh-ai-gateway-infra/maas-api-655f6fc5df-b8pd9" Nov 01 15:55:05.334408 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:55:05.334300 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk2km\" (UniqueName: \"kubernetes.io/projected/086dd0e1-6a4f-45dd-9b53-f78094bda8ab-kube-api-access-kk2km\") pod \"maas-api-655f6fc5df-b8pd9\" (UID: \"086dd0e1-6a4f-45dd-9b53-f78094bda8ab\") " pod="odh-ai-gateway-infra/maas-api-655f6fc5df-b8pd9" Nov 01 15:55:05.334408 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:55:05.334339 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/086dd0e1-6a4f-45dd-9b53-f78094bda8ab-metrics-certs\") pod \"maas-api-655f6fc5df-b8pd9\" (UID: \"086dd0e1-6a4f-45dd-9b53-f78094bda8ab\") " pod="odh-ai-gateway-infra/maas-api-655f6fc5df-b8pd9" Nov 01 15:55:05.334408 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:55:05.334369 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/086dd0e1-6a4f-45dd-9b53-f78094bda8ab-maas-api-tls\") pod \"maas-api-655f6fc5df-b8pd9\" (UID: \"086dd0e1-6a4f-45dd-9b53-f78094bda8ab\") " pod="odh-ai-gateway-infra/maas-api-655f6fc5df-b8pd9" Nov 01 15:55:05.334855 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:55:05.334835 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/086dd0e1-6a4f-45dd-9b53-f78094bda8ab-openshift-service-ca\") pod \"maas-api-655f6fc5df-b8pd9\" (UID: \"086dd0e1-6a4f-45dd-9b53-f78094bda8ab\") " pod="odh-ai-gateway-infra/maas-api-655f6fc5df-b8pd9" Nov 01 15:55:05.336835 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:55:05.336813 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/086dd0e1-6a4f-45dd-9b53-f78094bda8ab-metrics-certs\") pod \"maas-api-655f6fc5df-b8pd9\" (UID: \"086dd0e1-6a4f-45dd-9b53-f78094bda8ab\") " pod="odh-ai-gateway-infra/maas-api-655f6fc5df-b8pd9" Nov 01 15:55:05.336983 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:55:05.336961 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/086dd0e1-6a4f-45dd-9b53-f78094bda8ab-maas-api-tls\") pod \"maas-api-655f6fc5df-b8pd9\" (UID: \"086dd0e1-6a4f-45dd-9b53-f78094bda8ab\") " pod="odh-ai-gateway-infra/maas-api-655f6fc5df-b8pd9" Nov 01 15:55:05.341509 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:55:05.341487 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk2km\" (UniqueName: \"kubernetes.io/projected/086dd0e1-6a4f-45dd-9b53-f78094bda8ab-kube-api-access-kk2km\") pod \"maas-api-655f6fc5df-b8pd9\" (UID: \"086dd0e1-6a4f-45dd-9b53-f78094bda8ab\") " pod="odh-ai-gateway-infra/maas-api-655f6fc5df-b8pd9" Nov 01 15:55:05.480679 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:55:05.480603 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-655f6fc5df-b8pd9" Nov 01 15:55:05.752211 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:55:05.752183 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-655f6fc5df-b8pd9"] Nov 01 15:55:05.754501 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:55:05.754471 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod086dd0e1_6a4f_45dd_9b53_f78094bda8ab.slice/crio-e3c39e19124bb109887ba9c093a0fed505e193289761eb9c1f81224082158cce WatchSource:0}: Error finding container e3c39e19124bb109887ba9c093a0fed505e193289761eb9c1f81224082158cce: Status 404 returned error can't find the container with id e3c39e19124bb109887ba9c093a0fed505e193289761eb9c1f81224082158cce Nov 01 15:55:06.503905 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:55:06.503830 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-655f6fc5df-b8pd9" event={"ID":"086dd0e1-6a4f-45dd-9b53-f78094bda8ab","Type":"ContainerStarted","Data":"e3c39e19124bb109887ba9c093a0fed505e193289761eb9c1f81224082158cce"} Nov 01 15:55:08.514558 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:55:08.514482 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-655f6fc5df-b8pd9" event={"ID":"086dd0e1-6a4f-45dd-9b53-f78094bda8ab","Type":"ContainerStarted","Data":"684d58f509a37ff570fa897e0325c5676ce73e96138ae58aa86bc2ac582b193c"} Nov 01 15:55:08.514914 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:55:08.514632 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-655f6fc5df-b8pd9" Nov 01 15:55:08.531699 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:55:08.531655 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-655f6fc5df-b8pd9" podStartSLOduration=1.173264915 podStartE2EDuration="3.531643245s" totalImagesPullingTime="2.35837833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:55:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:55:08.530051583 +0000 UTC m=+441.861789235" watchObservedRunningTime="2026-11-01 15:55:08.531643245 +0000 UTC m=+441.863380894" Nov 01 15:55:14.524040 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:55:14.523991 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-655f6fc5df-b8pd9" Nov 01 15:55:18.307548 ip-10-0-128-238 kubenswrapper[4363]: E1101 15:55:18.307505 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 15:55:48.307075 ip-10-0-128-238 kubenswrapper[4363]: E1101 15:55:48.307044 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 15:56:18.307071 ip-10-0-128-238 kubenswrapper[4363]: E1101 15:56:18.306959 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 15:56:21.034891 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:21.034855 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-8h5j2"] Nov 01 15:56:21.038492 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:21.038469 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8h5j2" Nov 01 15:56:21.042122 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:21.042101 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 01 15:56:21.043095 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:21.043071 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 01 15:56:21.043199 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:21.043077 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 01 15:56:21.043199 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:21.043079 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ffz56\"" Nov 01 15:56:21.053599 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:21.053563 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-8h5j2"] Nov 01 15:56:21.146928 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:21.146893 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14cafac8-b130-4780-aa9e-e60215541376-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8h5j2\" (UID: \"14cafac8-b130-4780-aa9e-e60215541376\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8h5j2" Nov 01 15:56:21.147120 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:21.146962 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14cafac8-b130-4780-aa9e-e60215541376-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8h5j2\" (UID: \"14cafac8-b130-4780-aa9e-e60215541376\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8h5j2" Nov 01 15:56:21.147120 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:21.147076 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14cafac8-b130-4780-aa9e-e60215541376-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8h5j2\" (UID: \"14cafac8-b130-4780-aa9e-e60215541376\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8h5j2" Nov 01 15:56:21.147120 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:21.147107 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c92qb\" (UniqueName: \"kubernetes.io/projected/14cafac8-b130-4780-aa9e-e60215541376-kube-api-access-c92qb\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8h5j2\" (UID: \"14cafac8-b130-4780-aa9e-e60215541376\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8h5j2" Nov 01 15:56:21.147273 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:21.147136 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14cafac8-b130-4780-aa9e-e60215541376-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8h5j2\" (UID: \"14cafac8-b130-4780-aa9e-e60215541376\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8h5j2" Nov 01 15:56:21.147273 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:21.147227 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14cafac8-b130-4780-aa9e-e60215541376-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8h5j2\" (UID: \"14cafac8-b130-4780-aa9e-e60215541376\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8h5j2" Nov 01 15:56:21.248507 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:21.248452 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14cafac8-b130-4780-aa9e-e60215541376-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8h5j2\" (UID: \"14cafac8-b130-4780-aa9e-e60215541376\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8h5j2" Nov 01 15:56:21.248708 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:21.248528 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14cafac8-b130-4780-aa9e-e60215541376-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8h5j2\" (UID: \"14cafac8-b130-4780-aa9e-e60215541376\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8h5j2" Nov 01 15:56:21.248708 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:21.248558 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14cafac8-b130-4780-aa9e-e60215541376-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8h5j2\" (UID: \"14cafac8-b130-4780-aa9e-e60215541376\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8h5j2" Nov 01 15:56:21.248708 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:21.248599 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14cafac8-b130-4780-aa9e-e60215541376-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8h5j2\" (UID: \"14cafac8-b130-4780-aa9e-e60215541376\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8h5j2" Nov 01 15:56:21.248708 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:21.248617 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14cafac8-b130-4780-aa9e-e60215541376-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8h5j2\" (UID: \"14cafac8-b130-4780-aa9e-e60215541376\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8h5j2" Nov 01 15:56:21.248708 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:21.248635 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c92qb\" (UniqueName: \"kubernetes.io/projected/14cafac8-b130-4780-aa9e-e60215541376-kube-api-access-c92qb\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8h5j2\" (UID: \"14cafac8-b130-4780-aa9e-e60215541376\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8h5j2" Nov 01 15:56:21.248991 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:21.248957 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14cafac8-b130-4780-aa9e-e60215541376-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8h5j2\" (UID: \"14cafac8-b130-4780-aa9e-e60215541376\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8h5j2" Nov 01 15:56:21.249079 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:21.248989 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14cafac8-b130-4780-aa9e-e60215541376-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8h5j2\" (UID: \"14cafac8-b130-4780-aa9e-e60215541376\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8h5j2" Nov 01 15:56:21.249079 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:21.249046 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14cafac8-b130-4780-aa9e-e60215541376-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8h5j2\" (UID: \"14cafac8-b130-4780-aa9e-e60215541376\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8h5j2" Nov 01 15:56:21.250822 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:21.250804 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14cafac8-b130-4780-aa9e-e60215541376-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8h5j2\" (UID: \"14cafac8-b130-4780-aa9e-e60215541376\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8h5j2" Nov 01 15:56:21.251303 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:21.251283 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14cafac8-b130-4780-aa9e-e60215541376-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8h5j2\" (UID: \"14cafac8-b130-4780-aa9e-e60215541376\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8h5j2" Nov 01 15:56:21.256397 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:21.256378 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c92qb\" (UniqueName: \"kubernetes.io/projected/14cafac8-b130-4780-aa9e-e60215541376-kube-api-access-c92qb\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8h5j2\" (UID: \"14cafac8-b130-4780-aa9e-e60215541376\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8h5j2" Nov 01 15:56:21.351718 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:21.351687 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8h5j2" Nov 01 15:56:21.630839 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:21.628308 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-8h5j2"] Nov 01 15:56:21.635165 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:56:21.635132 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14cafac8_b130_4780_aa9e_e60215541376.slice/crio-63b409997e31000a12e553d917fc76c15adf5c183242d664e4c975bf46a7402a WatchSource:0}: Error finding container 63b409997e31000a12e553d917fc76c15adf5c183242d664e4c975bf46a7402a: Status 404 returned error can't find the container with id 63b409997e31000a12e553d917fc76c15adf5c183242d664e4c975bf46a7402a Nov 01 15:56:21.814951 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:21.814920 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8h5j2" event={"ID":"14cafac8-b130-4780-aa9e-e60215541376","Type":"ContainerStarted","Data":"63b409997e31000a12e553d917fc76c15adf5c183242d664e4c975bf46a7402a"} Nov 01 15:56:22.347709 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:22.346876 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k"] Nov 01 15:56:22.353139 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:22.353112 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k" Nov 01 15:56:22.359547 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:22.359479 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 01 15:56:22.363926 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:22.363859 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k"] Nov 01 15:56:22.458400 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:22.458365 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wldz6\" (UniqueName: \"kubernetes.io/projected/336c4cd8-6126-4431-930e-57f9b7ccf09d-kube-api-access-wldz6\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k\" (UID: \"336c4cd8-6126-4431-930e-57f9b7ccf09d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k" Nov 01 15:56:22.458586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:22.458432 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/336c4cd8-6126-4431-930e-57f9b7ccf09d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k\" (UID: \"336c4cd8-6126-4431-930e-57f9b7ccf09d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k" Nov 01 15:56:22.458586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:22.458546 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/336c4cd8-6126-4431-930e-57f9b7ccf09d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k\" (UID: \"336c4cd8-6126-4431-930e-57f9b7ccf09d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k" Nov 01 15:56:22.458586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:22.458576 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/336c4cd8-6126-4431-930e-57f9b7ccf09d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k\" (UID: \"336c4cd8-6126-4431-930e-57f9b7ccf09d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k" Nov 01 15:56:22.458773 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:22.458602 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/336c4cd8-6126-4431-930e-57f9b7ccf09d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k\" (UID: \"336c4cd8-6126-4431-930e-57f9b7ccf09d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k" Nov 01 15:56:22.458773 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:22.458705 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/336c4cd8-6126-4431-930e-57f9b7ccf09d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k\" (UID: \"336c4cd8-6126-4431-930e-57f9b7ccf09d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k" Nov 01 15:56:22.559688 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:22.559654 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wldz6\" (UniqueName: \"kubernetes.io/projected/336c4cd8-6126-4431-930e-57f9b7ccf09d-kube-api-access-wldz6\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k\" (UID: \"336c4cd8-6126-4431-930e-57f9b7ccf09d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k" Nov 01 15:56:22.559860 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:22.559701 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/336c4cd8-6126-4431-930e-57f9b7ccf09d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k\" (UID: \"336c4cd8-6126-4431-930e-57f9b7ccf09d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k" Nov 01 15:56:22.559860 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:22.559737 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/336c4cd8-6126-4431-930e-57f9b7ccf09d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k\" (UID: \"336c4cd8-6126-4431-930e-57f9b7ccf09d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k" Nov 01 15:56:22.559860 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:22.559754 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/336c4cd8-6126-4431-930e-57f9b7ccf09d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k\" (UID: \"336c4cd8-6126-4431-930e-57f9b7ccf09d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k" Nov 01 15:56:22.559860 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:22.559770 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/336c4cd8-6126-4431-930e-57f9b7ccf09d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k\" (UID: \"336c4cd8-6126-4431-930e-57f9b7ccf09d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k" Nov 01 15:56:22.559860 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:22.559793 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/336c4cd8-6126-4431-930e-57f9b7ccf09d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k\" (UID: \"336c4cd8-6126-4431-930e-57f9b7ccf09d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k" Nov 01 15:56:22.560391 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:22.560184 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/336c4cd8-6126-4431-930e-57f9b7ccf09d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k\" (UID: \"336c4cd8-6126-4431-930e-57f9b7ccf09d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k" Nov 01 15:56:22.560391 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:22.560242 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/336c4cd8-6126-4431-930e-57f9b7ccf09d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k\" (UID: \"336c4cd8-6126-4431-930e-57f9b7ccf09d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k" Nov 01 15:56:22.560391 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:22.560274 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/336c4cd8-6126-4431-930e-57f9b7ccf09d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k\" (UID: \"336c4cd8-6126-4431-930e-57f9b7ccf09d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k" Nov 01 15:56:22.562071 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:22.562046 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/336c4cd8-6126-4431-930e-57f9b7ccf09d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k\" (UID: \"336c4cd8-6126-4431-930e-57f9b7ccf09d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k" Nov 01 15:56:22.562332 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:22.562315 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/336c4cd8-6126-4431-930e-57f9b7ccf09d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k\" (UID: \"336c4cd8-6126-4431-930e-57f9b7ccf09d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k" Nov 01 15:56:22.566929 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:22.566905 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wldz6\" (UniqueName: \"kubernetes.io/projected/336c4cd8-6126-4431-930e-57f9b7ccf09d-kube-api-access-wldz6\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k\" (UID: \"336c4cd8-6126-4431-930e-57f9b7ccf09d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k" Nov 01 15:56:22.678033 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:22.677785 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k" Nov 01 15:56:22.965325 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:22.965189 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k"] Nov 01 15:56:22.967769 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:56:22.967741 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod336c4cd8_6126_4431_930e_57f9b7ccf09d.slice/crio-3e8b041d667ef2b3863bb18f5c609a86affe205dc8e8bc5b712806544a6d56c4 WatchSource:0}: Error finding container 3e8b041d667ef2b3863bb18f5c609a86affe205dc8e8bc5b712806544a6d56c4: Status 404 returned error can't find the container with id 3e8b041d667ef2b3863bb18f5c609a86affe205dc8e8bc5b712806544a6d56c4 Nov 01 15:56:23.824233 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:23.824193 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k" event={"ID":"336c4cd8-6126-4431-930e-57f9b7ccf09d","Type":"ContainerStarted","Data":"d194627efce1e64c9247427dbd6b4833eb6e227f22bd2ec32f013765714e8715"} Nov 01 15:56:23.824777 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:23.824256 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k" event={"ID":"336c4cd8-6126-4431-930e-57f9b7ccf09d","Type":"ContainerStarted","Data":"3e8b041d667ef2b3863bb18f5c609a86affe205dc8e8bc5b712806544a6d56c4"} Nov 01 15:56:23.825809 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:23.825779 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8h5j2" event={"ID":"14cafac8-b130-4780-aa9e-e60215541376","Type":"ContainerStarted","Data":"d642aeb4ca5b189f45c156a22fb655dcf6f0857adb5f7459e245349dafc23502"} Nov 01 15:56:23.842743 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:23.842696 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k" podStartSLOduration=1.579173339 podStartE2EDuration="1.842682627s" totalImagesPullingTime="263.509288ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:56:23.840698532 +0000 UTC m=+517.172436182" watchObservedRunningTime="2026-11-01 15:56:23.842682627 +0000 UTC m=+517.174420276" Nov 01 15:56:23.858354 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:23.858310 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8h5j2" podStartSLOduration=1.456999632 podStartE2EDuration="2.858295605s" totalImagesPullingTime="1.401295973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:56:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:56:23.857316752 +0000 UTC m=+517.189054417" watchObservedRunningTime="2026-11-01 15:56:23.858295605 +0000 UTC m=+517.190033255" Nov 01 15:56:24.352404 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:24.352361 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8h5j2" Nov 01 15:56:24.543307 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:24.543272 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-qncjz"] Nov 01 15:56:24.547176 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:24.547155 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-qncjz" Nov 01 15:56:24.550754 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:24.550733 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 01 15:56:24.556690 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:24.556667 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-qncjz"] Nov 01 15:56:24.577256 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:24.577213 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c0c2696-09a8-4726-98ca-d8331d86e830-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-qncjz\" (UID: \"7c0c2696-09a8-4726-98ca-d8331d86e830\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qncjz" Nov 01 15:56:24.577383 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:24.577278 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c0c2696-09a8-4726-98ca-d8331d86e830-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-qncjz\" (UID: \"7c0c2696-09a8-4726-98ca-d8331d86e830\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qncjz" Nov 01 15:56:24.577383 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:24.577301 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c0c2696-09a8-4726-98ca-d8331d86e830-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-qncjz\" (UID: \"7c0c2696-09a8-4726-98ca-d8331d86e830\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qncjz" Nov 01 15:56:24.577383 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:24.577339 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c0c2696-09a8-4726-98ca-d8331d86e830-home\") pod \"echo-simulator-kserve-64b6f8f96c-qncjz\" (UID: \"7c0c2696-09a8-4726-98ca-d8331d86e830\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qncjz" Nov 01 15:56:24.577530 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:24.577394 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c0c2696-09a8-4726-98ca-d8331d86e830-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-qncjz\" (UID: \"7c0c2696-09a8-4726-98ca-d8331d86e830\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qncjz" Nov 01 15:56:24.577530 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:24.577461 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcvns\" (UniqueName: \"kubernetes.io/projected/7c0c2696-09a8-4726-98ca-d8331d86e830-kube-api-access-bcvns\") pod \"echo-simulator-kserve-64b6f8f96c-qncjz\" (UID: \"7c0c2696-09a8-4726-98ca-d8331d86e830\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qncjz" Nov 01 15:56:24.678722 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:24.678611 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k" Nov 01 15:56:24.678873 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:24.678852 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c0c2696-09a8-4726-98ca-d8331d86e830-home\") pod \"echo-simulator-kserve-64b6f8f96c-qncjz\" (UID: \"7c0c2696-09a8-4726-98ca-d8331d86e830\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qncjz" Nov 01 15:56:24.678915 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:24.678880 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c0c2696-09a8-4726-98ca-d8331d86e830-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-qncjz\" (UID: \"7c0c2696-09a8-4726-98ca-d8331d86e830\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qncjz" Nov 01 15:56:24.679064 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:24.679047 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcvns\" (UniqueName: \"kubernetes.io/projected/7c0c2696-09a8-4726-98ca-d8331d86e830-kube-api-access-bcvns\") pod \"echo-simulator-kserve-64b6f8f96c-qncjz\" (UID: \"7c0c2696-09a8-4726-98ca-d8331d86e830\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qncjz" Nov 01 15:56:24.679136 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:24.679120 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c0c2696-09a8-4726-98ca-d8331d86e830-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-qncjz\" (UID: \"7c0c2696-09a8-4726-98ca-d8331d86e830\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qncjz" Nov 01 15:56:24.679187 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:24.679166 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c0c2696-09a8-4726-98ca-d8331d86e830-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-qncjz\" (UID: \"7c0c2696-09a8-4726-98ca-d8331d86e830\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qncjz" Nov 01 15:56:24.679234 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:24.679191 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c0c2696-09a8-4726-98ca-d8331d86e830-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-qncjz\" (UID: \"7c0c2696-09a8-4726-98ca-d8331d86e830\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qncjz" Nov 01 15:56:24.679287 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:24.679268 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c0c2696-09a8-4726-98ca-d8331d86e830-home\") pod \"echo-simulator-kserve-64b6f8f96c-qncjz\" (UID: \"7c0c2696-09a8-4726-98ca-d8331d86e830\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qncjz" Nov 01 15:56:24.679454 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:24.679432 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c0c2696-09a8-4726-98ca-d8331d86e830-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-qncjz\" (UID: \"7c0c2696-09a8-4726-98ca-d8331d86e830\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qncjz" Nov 01 15:56:24.679542 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:24.679440 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c0c2696-09a8-4726-98ca-d8331d86e830-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-qncjz\" (UID: \"7c0c2696-09a8-4726-98ca-d8331d86e830\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qncjz" Nov 01 15:56:24.681598 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:24.681569 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c0c2696-09a8-4726-98ca-d8331d86e830-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-qncjz\" (UID: \"7c0c2696-09a8-4726-98ca-d8331d86e830\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qncjz" Nov 01 15:56:24.681907 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:24.681887 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c0c2696-09a8-4726-98ca-d8331d86e830-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-qncjz\" (UID: \"7c0c2696-09a8-4726-98ca-d8331d86e830\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qncjz" Nov 01 15:56:24.687322 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:24.687304 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcvns\" (UniqueName: \"kubernetes.io/projected/7c0c2696-09a8-4726-98ca-d8331d86e830-kube-api-access-bcvns\") pod \"echo-simulator-kserve-64b6f8f96c-qncjz\" (UID: \"7c0c2696-09a8-4726-98ca-d8331d86e830\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qncjz" Nov 01 15:56:24.861297 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:24.861261 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-qncjz" Nov 01 15:56:25.138189 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:25.138163 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-qncjz"] Nov 01 15:56:25.140921 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:56:25.140893 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c0c2696_09a8_4726_98ca_d8331d86e830.slice/crio-477b7e1a5e00455c54f170678854e83afaf4a0e4cff97b8d25ae38446f4374e7 WatchSource:0}: Error finding container 477b7e1a5e00455c54f170678854e83afaf4a0e4cff97b8d25ae38446f4374e7: Status 404 returned error can't find the container with id 477b7e1a5e00455c54f170678854e83afaf4a0e4cff97b8d25ae38446f4374e7 Nov 01 15:56:25.836083 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:25.836042 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-qncjz" event={"ID":"7c0c2696-09a8-4726-98ca-d8331d86e830","Type":"ContainerStarted","Data":"477b7e1a5e00455c54f170678854e83afaf4a0e4cff97b8d25ae38446f4374e7"} Nov 01 15:56:26.078487 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:26.078450 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-frphn"] Nov 01 15:56:26.083111 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:26.082864 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-frphn" Nov 01 15:56:26.093046 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:26.092927 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-frphn"] Nov 01 15:56:26.093128 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:26.093092 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 01 15:56:26.200665 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:26.200620 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa3886b7-e5e1-4f64-93d4-a097f446a835-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-frphn\" (UID: \"fa3886b7-e5e1-4f64-93d4-a097f446a835\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-frphn" Nov 01 15:56:26.200856 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:26.200672 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa3886b7-e5e1-4f64-93d4-a097f446a835-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-frphn\" (UID: \"fa3886b7-e5e1-4f64-93d4-a097f446a835\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-frphn" Nov 01 15:56:26.200856 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:26.200696 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa3886b7-e5e1-4f64-93d4-a097f446a835-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-frphn\" (UID: \"fa3886b7-e5e1-4f64-93d4-a097f446a835\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-frphn" Nov 01 15:56:26.200856 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:26.200730 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa3886b7-e5e1-4f64-93d4-a097f446a835-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-frphn\" (UID: \"fa3886b7-e5e1-4f64-93d4-a097f446a835\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-frphn" Nov 01 15:56:26.200856 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:26.200785 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa3886b7-e5e1-4f64-93d4-a097f446a835-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-frphn\" (UID: \"fa3886b7-e5e1-4f64-93d4-a097f446a835\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-frphn" Nov 01 15:56:26.200856 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:26.200810 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw8l2\" (UniqueName: \"kubernetes.io/projected/fa3886b7-e5e1-4f64-93d4-a097f446a835-kube-api-access-rw8l2\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-frphn\" (UID: \"fa3886b7-e5e1-4f64-93d4-a097f446a835\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-frphn" Nov 01 15:56:26.302048 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:26.301981 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw8l2\" (UniqueName: \"kubernetes.io/projected/fa3886b7-e5e1-4f64-93d4-a097f446a835-kube-api-access-rw8l2\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-frphn\" (UID: \"fa3886b7-e5e1-4f64-93d4-a097f446a835\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-frphn" Nov 01 15:56:26.302224 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:26.302107 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa3886b7-e5e1-4f64-93d4-a097f446a835-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-frphn\" (UID: \"fa3886b7-e5e1-4f64-93d4-a097f446a835\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-frphn" Nov 01 15:56:26.302224 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:26.302140 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa3886b7-e5e1-4f64-93d4-a097f446a835-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-frphn\" (UID: \"fa3886b7-e5e1-4f64-93d4-a097f446a835\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-frphn" Nov 01 15:56:26.302224 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:26.302163 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa3886b7-e5e1-4f64-93d4-a097f446a835-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-frphn\" (UID: \"fa3886b7-e5e1-4f64-93d4-a097f446a835\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-frphn" Nov 01 15:56:26.302439 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:26.302352 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa3886b7-e5e1-4f64-93d4-a097f446a835-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-frphn\" (UID: \"fa3886b7-e5e1-4f64-93d4-a097f446a835\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-frphn" Nov 01 15:56:26.302439 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:26.302432 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa3886b7-e5e1-4f64-93d4-a097f446a835-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-frphn\" (UID: \"fa3886b7-e5e1-4f64-93d4-a097f446a835\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-frphn" Nov 01 15:56:26.302832 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:26.302597 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa3886b7-e5e1-4f64-93d4-a097f446a835-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-frphn\" (UID: \"fa3886b7-e5e1-4f64-93d4-a097f446a835\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-frphn" Nov 01 15:56:26.302832 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:26.302613 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa3886b7-e5e1-4f64-93d4-a097f446a835-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-frphn\" (UID: \"fa3886b7-e5e1-4f64-93d4-a097f446a835\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-frphn" Nov 01 15:56:26.302832 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:26.302829 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa3886b7-e5e1-4f64-93d4-a097f446a835-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-frphn\" (UID: \"fa3886b7-e5e1-4f64-93d4-a097f446a835\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-frphn" Nov 01 15:56:26.305232 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:26.305207 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa3886b7-e5e1-4f64-93d4-a097f446a835-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-frphn\" (UID: \"fa3886b7-e5e1-4f64-93d4-a097f446a835\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-frphn" Nov 01 15:56:26.305343 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:26.305277 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa3886b7-e5e1-4f64-93d4-a097f446a835-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-frphn\" (UID: \"fa3886b7-e5e1-4f64-93d4-a097f446a835\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-frphn" Nov 01 15:56:26.310181 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:26.310154 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw8l2\" (UniqueName: \"kubernetes.io/projected/fa3886b7-e5e1-4f64-93d4-a097f446a835-kube-api-access-rw8l2\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-frphn\" (UID: \"fa3886b7-e5e1-4f64-93d4-a097f446a835\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-frphn" Nov 01 15:56:26.401423 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:26.401343 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-frphn" Nov 01 15:56:26.679982 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:26.679904 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-frphn"] Nov 01 15:56:26.683695 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:56:26.683665 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa3886b7_e5e1_4f64_93d4_a097f446a835.slice/crio-cf06a9e47995cf29dd336da896a48a3464655f0a695b388013df90873f794a5d WatchSource:0}: Error finding container cf06a9e47995cf29dd336da896a48a3464655f0a695b388013df90873f794a5d: Status 404 returned error can't find the container with id cf06a9e47995cf29dd336da896a48a3464655f0a695b388013df90873f794a5d Nov 01 15:56:26.841997 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:26.841964 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-frphn" event={"ID":"fa3886b7-e5e1-4f64-93d4-a097f446a835","Type":"ContainerStarted","Data":"cf06a9e47995cf29dd336da896a48a3464655f0a695b388013df90873f794a5d"} Nov 01 15:56:26.843458 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:26.843431 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-qncjz" event={"ID":"7c0c2696-09a8-4726-98ca-d8331d86e830","Type":"ContainerStarted","Data":"8a4ba7f9fb8b354b8abeaabf2695435ceb0aff1ffce2135f7b4d02e32c90f7e4"} Nov 01 15:56:26.862186 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:26.862162 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-qncjz" Nov 01 15:56:26.866248 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:26.866208 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-qncjz" podStartSLOduration=1.267751868 podStartE2EDuration="2.866194734s" totalImagesPullingTime="1.598442866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:56:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:56:26.864668556 +0000 UTC m=+520.196406216" watchObservedRunningTime="2026-11-01 15:56:26.866194734 +0000 UTC m=+520.197932383" Nov 01 15:56:27.848804 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:27.848764 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-frphn" event={"ID":"fa3886b7-e5e1-4f64-93d4-a097f446a835","Type":"ContainerStarted","Data":"be3c980971913debb25958dd1ab435cec439228a4437e0b8c6bd43334d264a26"} Nov 01 15:56:27.867703 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:27.867623 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-frphn" podStartSLOduration=1.591263425 podStartE2EDuration="1.867605715s" totalImagesPullingTime="276.34229ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:56:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:56:27.865800047 +0000 UTC m=+521.197537702" watchObservedRunningTime="2026-11-01 15:56:27.867605715 +0000 UTC m=+521.199343365" Nov 01 15:56:28.401575 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:28.401541 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-frphn" Nov 01 15:56:30.054095 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:30.054059 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr"] Nov 01 15:56:30.058609 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:30.058582 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr" Nov 01 15:56:30.067803 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:30.067768 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 01 15:56:30.069989 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:30.069959 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr"] Nov 01 15:56:30.137198 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:30.137161 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/305de5fb-52ed-401a-89fe-b7d802472f5c-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr\" (UID: \"305de5fb-52ed-401a-89fe-b7d802472f5c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr" Nov 01 15:56:30.137198 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:30.137198 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/305de5fb-52ed-401a-89fe-b7d802472f5c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr\" (UID: \"305de5fb-52ed-401a-89fe-b7d802472f5c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr" Nov 01 15:56:30.137388 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:30.137225 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/305de5fb-52ed-401a-89fe-b7d802472f5c-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr\" (UID: \"305de5fb-52ed-401a-89fe-b7d802472f5c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr" Nov 01 15:56:30.137388 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:30.137285 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/305de5fb-52ed-401a-89fe-b7d802472f5c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr\" (UID: \"305de5fb-52ed-401a-89fe-b7d802472f5c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr" Nov 01 15:56:30.137388 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:30.137329 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hggm7\" (UniqueName: \"kubernetes.io/projected/305de5fb-52ed-401a-89fe-b7d802472f5c-kube-api-access-hggm7\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr\" (UID: \"305de5fb-52ed-401a-89fe-b7d802472f5c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr" Nov 01 15:56:30.137485 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:30.137390 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/305de5fb-52ed-401a-89fe-b7d802472f5c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr\" (UID: \"305de5fb-52ed-401a-89fe-b7d802472f5c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr" Nov 01 15:56:30.237902 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:30.237849 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/305de5fb-52ed-401a-89fe-b7d802472f5c-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr\" (UID: \"305de5fb-52ed-401a-89fe-b7d802472f5c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr" Nov 01 15:56:30.237902 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:30.237899 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/305de5fb-52ed-401a-89fe-b7d802472f5c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr\" (UID: \"305de5fb-52ed-401a-89fe-b7d802472f5c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr" Nov 01 15:56:30.238145 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:30.237940 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hggm7\" (UniqueName: \"kubernetes.io/projected/305de5fb-52ed-401a-89fe-b7d802472f5c-kube-api-access-hggm7\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr\" (UID: \"305de5fb-52ed-401a-89fe-b7d802472f5c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr" Nov 01 15:56:30.238145 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:30.237969 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/305de5fb-52ed-401a-89fe-b7d802472f5c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr\" (UID: \"305de5fb-52ed-401a-89fe-b7d802472f5c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr" Nov 01 15:56:30.238145 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:30.238085 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/305de5fb-52ed-401a-89fe-b7d802472f5c-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr\" (UID: \"305de5fb-52ed-401a-89fe-b7d802472f5c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr" Nov 01 15:56:30.238145 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:30.238109 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/305de5fb-52ed-401a-89fe-b7d802472f5c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr\" (UID: \"305de5fb-52ed-401a-89fe-b7d802472f5c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr" Nov 01 15:56:30.238476 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:30.238454 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/305de5fb-52ed-401a-89fe-b7d802472f5c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr\" (UID: \"305de5fb-52ed-401a-89fe-b7d802472f5c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr" Nov 01 15:56:30.238608 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:30.238585 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/305de5fb-52ed-401a-89fe-b7d802472f5c-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr\" (UID: \"305de5fb-52ed-401a-89fe-b7d802472f5c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr" Nov 01 15:56:30.238671 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:30.238617 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/305de5fb-52ed-401a-89fe-b7d802472f5c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr\" (UID: \"305de5fb-52ed-401a-89fe-b7d802472f5c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr" Nov 01 15:56:30.240273 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:30.240250 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/305de5fb-52ed-401a-89fe-b7d802472f5c-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr\" (UID: \"305de5fb-52ed-401a-89fe-b7d802472f5c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr" Nov 01 15:56:30.240662 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:30.240639 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/305de5fb-52ed-401a-89fe-b7d802472f5c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr\" (UID: \"305de5fb-52ed-401a-89fe-b7d802472f5c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr" Nov 01 15:56:30.246187 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:30.246162 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hggm7\" (UniqueName: \"kubernetes.io/projected/305de5fb-52ed-401a-89fe-b7d802472f5c-kube-api-access-hggm7\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr\" (UID: \"305de5fb-52ed-401a-89fe-b7d802472f5c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr" Nov 01 15:56:30.375793 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:30.375760 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr" Nov 01 15:56:30.650477 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:30.650403 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr"] Nov 01 15:56:30.653354 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:56:30.653327 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod305de5fb_52ed_401a_89fe_b7d802472f5c.slice/crio-b22204d5906483c386420d7687af1c77537a83dd9c985de66c881f077cea124d WatchSource:0}: Error finding container b22204d5906483c386420d7687af1c77537a83dd9c985de66c881f077cea124d: Status 404 returned error can't find the container with id b22204d5906483c386420d7687af1c77537a83dd9c985de66c881f077cea124d Nov 01 15:56:30.862399 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:30.862370 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr" event={"ID":"305de5fb-52ed-401a-89fe-b7d802472f5c","Type":"ContainerStarted","Data":"b22204d5906483c386420d7687af1c77537a83dd9c985de66c881f077cea124d"} Nov 01 15:56:31.352468 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:31.352438 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8h5j2" Nov 01 15:56:31.365197 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:31.365177 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8h5j2" Nov 01 15:56:31.868668 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:31.868631 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr" event={"ID":"305de5fb-52ed-401a-89fe-b7d802472f5c","Type":"ContainerStarted","Data":"48984d2f4812707895158093fb8caaa1094342ee240bfc20daffb808ce7395e8"} Nov 01 15:56:31.884817 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:31.884790 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8h5j2" Nov 01 15:56:31.897165 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:31.897115 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr" podStartSLOduration=1.621350303 podStartE2EDuration="1.897101078s" totalImagesPullingTime="275.750775ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:56:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:56:31.896342314 +0000 UTC m=+525.228079965" watchObservedRunningTime="2026-11-01 15:56:31.897101078 +0000 UTC m=+525.228838728" Nov 01 15:56:32.376909 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:32.376871 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr" Nov 01 15:56:32.678067 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:32.677957 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k" Nov 01 15:56:32.690719 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:32.690688 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k" Nov 01 15:56:32.887314 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:32.887279 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k" Nov 01 15:56:34.861574 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:34.861540 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-qncjz" Nov 01 15:56:34.874493 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:34.874453 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-qncjz" Nov 01 15:56:34.898603 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:34.898579 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-qncjz" Nov 01 15:56:36.401628 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:36.401584 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-frphn" Nov 01 15:56:36.414194 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:36.414167 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-frphn" Nov 01 15:56:36.906654 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:36.906626 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-frphn" Nov 01 15:56:40.376182 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:40.376150 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr" Nov 01 15:56:40.388450 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:40.388422 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr" Nov 01 15:56:40.923426 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:40.923399 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr" Nov 01 15:56:46.196841 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:46.196808 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k"] Nov 01 15:56:46.200476 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:46.200446 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k" Nov 01 15:56:46.215067 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:46.214991 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k"] Nov 01 15:56:46.277192 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:46.277153 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d02b9e-d3ce-4b70-9773-301792717809-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8s26k\" (UID: \"e0d02b9e-d3ce-4b70-9773-301792717809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k" Nov 01 15:56:46.277376 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:46.277216 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0d02b9e-d3ce-4b70-9773-301792717809-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8s26k\" (UID: \"e0d02b9e-d3ce-4b70-9773-301792717809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k" Nov 01 15:56:46.277444 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:46.277366 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0d02b9e-d3ce-4b70-9773-301792717809-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8s26k\" (UID: \"e0d02b9e-d3ce-4b70-9773-301792717809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k" Nov 01 15:56:46.277444 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:46.277410 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0d02b9e-d3ce-4b70-9773-301792717809-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8s26k\" (UID: \"e0d02b9e-d3ce-4b70-9773-301792717809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k" Nov 01 15:56:46.277516 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:46.277447 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6hg4\" (UniqueName: \"kubernetes.io/projected/e0d02b9e-d3ce-4b70-9773-301792717809-kube-api-access-m6hg4\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8s26k\" (UID: \"e0d02b9e-d3ce-4b70-9773-301792717809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k" Nov 01 15:56:46.277516 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:46.277489 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d02b9e-d3ce-4b70-9773-301792717809-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8s26k\" (UID: \"e0d02b9e-d3ce-4b70-9773-301792717809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k" Nov 01 15:56:46.277600 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:46.277539 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d02b9e-d3ce-4b70-9773-301792717809-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8s26k\" (UID: \"e0d02b9e-d3ce-4b70-9773-301792717809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k" Nov 01 15:56:46.277600 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:46.277569 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0d02b9e-d3ce-4b70-9773-301792717809-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8s26k\" (UID: \"e0d02b9e-d3ce-4b70-9773-301792717809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k" Nov 01 15:56:46.277600 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:46.277590 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0d02b9e-d3ce-4b70-9773-301792717809-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8s26k\" (UID: \"e0d02b9e-d3ce-4b70-9773-301792717809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k" Nov 01 15:56:46.299234 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:46.299210 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-tkjjf\"" Nov 01 15:56:46.377979 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:46.377946 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0d02b9e-d3ce-4b70-9773-301792717809-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8s26k\" (UID: \"e0d02b9e-d3ce-4b70-9773-301792717809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k" Nov 01 15:56:46.377979 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:46.377977 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6hg4\" (UniqueName: \"kubernetes.io/projected/e0d02b9e-d3ce-4b70-9773-301792717809-kube-api-access-m6hg4\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8s26k\" (UID: \"e0d02b9e-d3ce-4b70-9773-301792717809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k" Nov 01 15:56:46.378239 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:46.378003 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d02b9e-d3ce-4b70-9773-301792717809-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8s26k\" (UID: \"e0d02b9e-d3ce-4b70-9773-301792717809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k" Nov 01 15:56:46.378239 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:46.378046 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d02b9e-d3ce-4b70-9773-301792717809-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8s26k\" (UID: \"e0d02b9e-d3ce-4b70-9773-301792717809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k" Nov 01 15:56:46.378239 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:46.378063 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0d02b9e-d3ce-4b70-9773-301792717809-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8s26k\" (UID: \"e0d02b9e-d3ce-4b70-9773-301792717809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k" Nov 01 15:56:46.378239 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:46.378079 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0d02b9e-d3ce-4b70-9773-301792717809-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8s26k\" (UID: \"e0d02b9e-d3ce-4b70-9773-301792717809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k" Nov 01 15:56:46.378239 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:46.378101 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d02b9e-d3ce-4b70-9773-301792717809-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8s26k\" (UID: \"e0d02b9e-d3ce-4b70-9773-301792717809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k" Nov 01 15:56:46.378239 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:46.378167 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0d02b9e-d3ce-4b70-9773-301792717809-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8s26k\" (UID: \"e0d02b9e-d3ce-4b70-9773-301792717809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k" Nov 01 15:56:46.378239 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:46.378233 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0d02b9e-d3ce-4b70-9773-301792717809-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8s26k\" (UID: \"e0d02b9e-d3ce-4b70-9773-301792717809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k" Nov 01 15:56:46.378582 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:46.378521 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d02b9e-d3ce-4b70-9773-301792717809-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8s26k\" (UID: \"e0d02b9e-d3ce-4b70-9773-301792717809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k" Nov 01 15:56:46.378715 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:46.378691 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d02b9e-d3ce-4b70-9773-301792717809-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8s26k\" (UID: \"e0d02b9e-d3ce-4b70-9773-301792717809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k" Nov 01 15:56:46.378775 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:46.378723 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d02b9e-d3ce-4b70-9773-301792717809-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8s26k\" (UID: \"e0d02b9e-d3ce-4b70-9773-301792717809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k" Nov 01 15:56:46.378829 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:46.378802 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0d02b9e-d3ce-4b70-9773-301792717809-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8s26k\" (UID: \"e0d02b9e-d3ce-4b70-9773-301792717809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k" Nov 01 15:56:46.378893 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:46.378879 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0d02b9e-d3ce-4b70-9773-301792717809-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8s26k\" (UID: \"e0d02b9e-d3ce-4b70-9773-301792717809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k" Nov 01 15:56:46.380305 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:46.380285 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0d02b9e-d3ce-4b70-9773-301792717809-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8s26k\" (UID: \"e0d02b9e-d3ce-4b70-9773-301792717809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k" Nov 01 15:56:46.380687 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:46.380664 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0d02b9e-d3ce-4b70-9773-301792717809-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8s26k\" (UID: \"e0d02b9e-d3ce-4b70-9773-301792717809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k" Nov 01 15:56:46.385929 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:46.385901 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0d02b9e-d3ce-4b70-9773-301792717809-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8s26k\" (UID: \"e0d02b9e-d3ce-4b70-9773-301792717809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k" Nov 01 15:56:46.386570 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:46.386553 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6hg4\" (UniqueName: \"kubernetes.io/projected/e0d02b9e-d3ce-4b70-9773-301792717809-kube-api-access-m6hg4\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8s26k\" (UID: \"e0d02b9e-d3ce-4b70-9773-301792717809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k" Nov 01 15:56:46.516973 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:46.516905 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k" Nov 01 15:56:46.795106 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:56:46.795075 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0d02b9e_d3ce_4b70_9773_301792717809.slice/crio-2187291293b2319f80ac966e02829809140a9a9d14c07fc439d5b474af9ac122 WatchSource:0}: Error finding container 2187291293b2319f80ac966e02829809140a9a9d14c07fc439d5b474af9ac122: Status 404 returned error can't find the container with id 2187291293b2319f80ac966e02829809140a9a9d14c07fc439d5b474af9ac122 Nov 01 15:56:46.797469 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:46.797447 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k"] Nov 01 15:56:46.938888 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:46.938860 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k" event={"ID":"e0d02b9e-d3ce-4b70-9773-301792717809","Type":"ContainerStarted","Data":"2187291293b2319f80ac966e02829809140a9a9d14c07fc439d5b474af9ac122"} Nov 01 15:56:48.311452 ip-10-0-128-238 kubenswrapper[4363]: E1101 15:56:48.311411 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 15:56:50.022164 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:50.022134 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:56:50.022421 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:50.022201 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:56:50.022421 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:50.022230 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:56:50.957514 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:50.957472 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k" event={"ID":"e0d02b9e-d3ce-4b70-9773-301792717809","Type":"ContainerStarted","Data":"ebf2ba8422b0376eb066c9a9a33504594c511012e0886e618f4b8413054325db"} Nov 01 15:56:50.976881 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:50.976837 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k" podStartSLOduration=1.75291753 podStartE2EDuration="4.976822834s" totalImagesPullingTime="3.223905304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:56:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:56:50.975464951 +0000 UTC m=+544.307202612" watchObservedRunningTime="2026-11-01 15:56:50.976822834 +0000 UTC m=+544.308560484" Nov 01 15:56:51.518093 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:51.518052 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k" Nov 01 15:56:51.519423 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:51.519396 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8s26k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Nov 01 15:56:51.519519 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:51.519455 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k" podUID="e0d02b9e-d3ce-4b70-9773-301792717809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Nov 01 15:56:52.517881 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:52.517844 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8s26k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Nov 01 15:56:52.518077 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:52.517922 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k" podUID="e0d02b9e-d3ce-4b70-9773-301792717809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Nov 01 15:56:53.517814 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:53.517777 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8s26k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Nov 01 15:56:53.518226 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:53.517834 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k" podUID="e0d02b9e-d3ce-4b70-9773-301792717809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Nov 01 15:56:54.518267 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:54.518228 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8s26k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Nov 01 15:56:54.518675 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:54.518288 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k" podUID="e0d02b9e-d3ce-4b70-9773-301792717809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Nov 01 15:56:55.517373 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:55.517339 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8s26k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Nov 01 15:56:55.517619 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:55.517396 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k" podUID="e0d02b9e-d3ce-4b70-9773-301792717809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Nov 01 15:56:56.518096 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:56.518056 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8s26k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Nov 01 15:56:56.518503 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:56.518126 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k" podUID="e0d02b9e-d3ce-4b70-9773-301792717809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Nov 01 15:56:58.219517 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:58.219487 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k" Nov 01 15:56:58.220164 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:58.220137 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k" Nov 01 15:56:58.220983 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:56:58.220960 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k" Nov 01 15:57:01.210963 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.210932 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4"] Nov 01 15:57:01.219212 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.219187 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" Nov 01 15:57:01.234154 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.234127 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4"] Nov 01 15:57:01.249028 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.248991 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84"] Nov 01 15:57:01.253702 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.253679 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" Nov 01 15:57:01.267955 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.267934 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84"] Nov 01 15:57:01.411357 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.411326 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dh7c4\" (UID: \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" Nov 01 15:57:01.411357 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.411357 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd20bf6-a51f-4006-85a9-1db75176c696-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r5m84\" (UID: \"afd20bf6-a51f-4006-85a9-1db75176c696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" Nov 01 15:57:01.411601 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.411380 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dh7c4\" (UID: \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" Nov 01 15:57:01.411601 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.411398 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd20bf6-a51f-4006-85a9-1db75176c696-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r5m84\" (UID: \"afd20bf6-a51f-4006-85a9-1db75176c696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" Nov 01 15:57:01.411601 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.411529 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dh7c4\" (UID: \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" Nov 01 15:57:01.411601 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.411574 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd20bf6-a51f-4006-85a9-1db75176c696-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r5m84\" (UID: \"afd20bf6-a51f-4006-85a9-1db75176c696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" Nov 01 15:57:01.411817 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.411663 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd20bf6-a51f-4006-85a9-1db75176c696-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r5m84\" (UID: \"afd20bf6-a51f-4006-85a9-1db75176c696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" Nov 01 15:57:01.411817 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.411716 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dh7c4\" (UID: \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" Nov 01 15:57:01.411817 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.411750 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z4ck\" (UniqueName: \"kubernetes.io/projected/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-kube-api-access-7z4ck\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dh7c4\" (UID: \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" Nov 01 15:57:01.411928 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.411826 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd20bf6-a51f-4006-85a9-1db75176c696-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r5m84\" (UID: \"afd20bf6-a51f-4006-85a9-1db75176c696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" Nov 01 15:57:01.411928 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.411844 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd20bf6-a51f-4006-85a9-1db75176c696-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r5m84\" (UID: \"afd20bf6-a51f-4006-85a9-1db75176c696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" Nov 01 15:57:01.411996 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.411921 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dh7c4\" (UID: \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" Nov 01 15:57:01.411996 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.411946 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dh7c4\" (UID: \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" Nov 01 15:57:01.411996 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.411987 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dh7c4\" (UID: \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" Nov 01 15:57:01.412135 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.412005 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd20bf6-a51f-4006-85a9-1db75176c696-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r5m84\" (UID: \"afd20bf6-a51f-4006-85a9-1db75176c696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" Nov 01 15:57:01.412135 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.412058 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngncl\" (UniqueName: \"kubernetes.io/projected/afd20bf6-a51f-4006-85a9-1db75176c696-kube-api-access-ngncl\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r5m84\" (UID: \"afd20bf6-a51f-4006-85a9-1db75176c696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" Nov 01 15:57:01.412135 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.412108 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd20bf6-a51f-4006-85a9-1db75176c696-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r5m84\" (UID: \"afd20bf6-a51f-4006-85a9-1db75176c696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" Nov 01 15:57:01.412230 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.412148 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dh7c4\" (UID: \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" Nov 01 15:57:01.513206 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.513135 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dh7c4\" (UID: \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" Nov 01 15:57:01.513206 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.513183 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd20bf6-a51f-4006-85a9-1db75176c696-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r5m84\" (UID: \"afd20bf6-a51f-4006-85a9-1db75176c696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" Nov 01 15:57:01.513371 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.513296 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd20bf6-a51f-4006-85a9-1db75176c696-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r5m84\" (UID: \"afd20bf6-a51f-4006-85a9-1db75176c696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" Nov 01 15:57:01.513371 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.513349 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dh7c4\" (UID: \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" Nov 01 15:57:01.513478 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.513385 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z4ck\" (UniqueName: \"kubernetes.io/projected/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-kube-api-access-7z4ck\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dh7c4\" (UID: \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" Nov 01 15:57:01.513478 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.513415 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd20bf6-a51f-4006-85a9-1db75176c696-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r5m84\" (UID: \"afd20bf6-a51f-4006-85a9-1db75176c696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" Nov 01 15:57:01.513478 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.513440 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd20bf6-a51f-4006-85a9-1db75176c696-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r5m84\" (UID: \"afd20bf6-a51f-4006-85a9-1db75176c696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" Nov 01 15:57:01.513631 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.513487 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dh7c4\" (UID: \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" Nov 01 15:57:01.513631 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.513515 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dh7c4\" (UID: \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" Nov 01 15:57:01.513631 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.513521 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dh7c4\" (UID: \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" Nov 01 15:57:01.513631 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.513589 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dh7c4\" (UID: \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" Nov 01 15:57:01.513631 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.513619 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd20bf6-a51f-4006-85a9-1db75176c696-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r5m84\" (UID: \"afd20bf6-a51f-4006-85a9-1db75176c696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" Nov 01 15:57:01.513876 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.513642 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngncl\" (UniqueName: \"kubernetes.io/projected/afd20bf6-a51f-4006-85a9-1db75176c696-kube-api-access-ngncl\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r5m84\" (UID: \"afd20bf6-a51f-4006-85a9-1db75176c696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" Nov 01 15:57:01.513876 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.513672 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd20bf6-a51f-4006-85a9-1db75176c696-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r5m84\" (UID: \"afd20bf6-a51f-4006-85a9-1db75176c696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" Nov 01 15:57:01.513876 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.513716 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dh7c4\" (UID: \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" Nov 01 15:57:01.513876 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.513724 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd20bf6-a51f-4006-85a9-1db75176c696-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r5m84\" (UID: \"afd20bf6-a51f-4006-85a9-1db75176c696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" Nov 01 15:57:01.513876 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.513734 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dh7c4\" (UID: \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" Nov 01 15:57:01.513876 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.513798 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dh7c4\" (UID: \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" Nov 01 15:57:01.513876 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.513832 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd20bf6-a51f-4006-85a9-1db75176c696-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r5m84\" (UID: \"afd20bf6-a51f-4006-85a9-1db75176c696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" Nov 01 15:57:01.513876 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.513866 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dh7c4\" (UID: \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" Nov 01 15:57:01.514316 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.513896 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd20bf6-a51f-4006-85a9-1db75176c696-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r5m84\" (UID: \"afd20bf6-a51f-4006-85a9-1db75176c696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" Nov 01 15:57:01.514316 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.514048 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd20bf6-a51f-4006-85a9-1db75176c696-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r5m84\" (UID: \"afd20bf6-a51f-4006-85a9-1db75176c696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" Nov 01 15:57:01.514316 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.514216 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd20bf6-a51f-4006-85a9-1db75176c696-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r5m84\" (UID: \"afd20bf6-a51f-4006-85a9-1db75176c696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" Nov 01 15:57:01.514316 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.514253 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dh7c4\" (UID: \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" Nov 01 15:57:01.514523 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.514328 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd20bf6-a51f-4006-85a9-1db75176c696-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r5m84\" (UID: \"afd20bf6-a51f-4006-85a9-1db75176c696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" Nov 01 15:57:01.514523 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.514351 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dh7c4\" (UID: \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" Nov 01 15:57:01.514632 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.514599 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd20bf6-a51f-4006-85a9-1db75176c696-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r5m84\" (UID: \"afd20bf6-a51f-4006-85a9-1db75176c696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" Nov 01 15:57:01.514751 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.514731 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dh7c4\" (UID: \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" Nov 01 15:57:01.516105 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.516070 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd20bf6-a51f-4006-85a9-1db75176c696-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r5m84\" (UID: \"afd20bf6-a51f-4006-85a9-1db75176c696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" Nov 01 15:57:01.516217 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.516127 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dh7c4\" (UID: \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" Nov 01 15:57:01.516294 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.516278 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dh7c4\" (UID: \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" Nov 01 15:57:01.516835 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.516814 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd20bf6-a51f-4006-85a9-1db75176c696-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r5m84\" (UID: \"afd20bf6-a51f-4006-85a9-1db75176c696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" Nov 01 15:57:01.521773 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.521744 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z4ck\" (UniqueName: \"kubernetes.io/projected/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-kube-api-access-7z4ck\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dh7c4\" (UID: \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" Nov 01 15:57:01.521913 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.521894 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dh7c4\" (UID: \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" Nov 01 15:57:01.522033 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.522001 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngncl\" (UniqueName: \"kubernetes.io/projected/afd20bf6-a51f-4006-85a9-1db75176c696-kube-api-access-ngncl\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r5m84\" (UID: \"afd20bf6-a51f-4006-85a9-1db75176c696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" Nov 01 15:57:01.522308 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.522292 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd20bf6-a51f-4006-85a9-1db75176c696-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r5m84\" (UID: \"afd20bf6-a51f-4006-85a9-1db75176c696\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" Nov 01 15:57:01.532874 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.532854 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" Nov 01 15:57:01.571999 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.571968 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" Nov 01 15:57:01.813831 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.813802 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4"] Nov 01 15:57:01.816365 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:57:01.816335 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eb560c5_c5f0_497b_a30d_fa3a44caae3d.slice/crio-bd121101c47340a0393e00cc2ca7fa796b702e6f9f1a1b643af9646f97032a3c WatchSource:0}: Error finding container bd121101c47340a0393e00cc2ca7fa796b702e6f9f1a1b643af9646f97032a3c: Status 404 returned error can't find the container with id bd121101c47340a0393e00cc2ca7fa796b702e6f9f1a1b643af9646f97032a3c Nov 01 15:57:01.819916 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.819885 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:57:01.820041 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.819949 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:57:01.820041 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.819976 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:57:01.859610 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.859588 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84"] Nov 01 15:57:01.862957 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:57:01.862908 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafd20bf6_a51f_4006_85a9_1db75176c696.slice/crio-21e884374ad19f6540e3973ca8ae4be5ba1e74637b9b583871c733c7cfbf3a17 WatchSource:0}: Error finding container 21e884374ad19f6540e3973ca8ae4be5ba1e74637b9b583871c733c7cfbf3a17: Status 404 returned error can't find the container with id 21e884374ad19f6540e3973ca8ae4be5ba1e74637b9b583871c733c7cfbf3a17 Nov 01 15:57:01.866515 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.866485 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:57:01.866611 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.866556 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:57:01.866611 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:01.866582 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:57:02.024808 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:02.024761 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" event={"ID":"5eb560c5-c5f0-497b-a30d-fa3a44caae3d","Type":"ContainerStarted","Data":"35f9a6e42cb1d896a5c0ad69e76af50c30844e172d884b5b0e44037c6e61a4f9"} Nov 01 15:57:02.024998 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:02.024822 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" event={"ID":"5eb560c5-c5f0-497b-a30d-fa3a44caae3d","Type":"ContainerStarted","Data":"bd121101c47340a0393e00cc2ca7fa796b702e6f9f1a1b643af9646f97032a3c"} Nov 01 15:57:02.029047 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:02.027382 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" event={"ID":"afd20bf6-a51f-4006-85a9-1db75176c696","Type":"ContainerStarted","Data":"b8dc44b4544b118f285de5dbcc15bf3cefde3bed81a0bf0e20d02d96b2c5ece0"} Nov 01 15:57:02.029047 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:02.027430 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" event={"ID":"afd20bf6-a51f-4006-85a9-1db75176c696","Type":"ContainerStarted","Data":"21e884374ad19f6540e3973ca8ae4be5ba1e74637b9b583871c733c7cfbf3a17"} Nov 01 15:57:02.047038 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:02.046962 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" podStartSLOduration=1.046948255 podStartE2EDuration="1.046948255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:57:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:57:02.043053269 +0000 UTC m=+555.374790919" watchObservedRunningTime="2026-11-01 15:57:02.046948255 +0000 UTC m=+555.378685905" Nov 01 15:57:02.063029 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:02.062922 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" podStartSLOduration=1.062905028 podStartE2EDuration="1.062905028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:57:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:57:02.060852487 +0000 UTC m=+555.392590138" watchObservedRunningTime="2026-11-01 15:57:02.062905028 +0000 UTC m=+555.394642681" Nov 01 15:57:02.534459 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:02.534340 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" Nov 01 15:57:02.535862 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:02.535820 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dh7c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 01 15:57:02.536040 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:02.535883 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" podUID="5eb560c5-c5f0-497b-a30d-fa3a44caae3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 01 15:57:02.572369 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:02.572334 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" Nov 01 15:57:02.573647 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:02.573621 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-r5m84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 01 15:57:02.573769 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:02.573671 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" podUID="afd20bf6-a51f-4006-85a9-1db75176c696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 01 15:57:03.534042 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:03.533640 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dh7c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 01 15:57:03.534042 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:03.533717 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" podUID="5eb560c5-c5f0-497b-a30d-fa3a44caae3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 01 15:57:03.573232 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:03.573200 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-r5m84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 01 15:57:03.573594 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:03.573262 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" podUID="afd20bf6-a51f-4006-85a9-1db75176c696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 01 15:57:04.533702 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:04.533666 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dh7c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 01 15:57:04.533880 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:04.533721 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" podUID="5eb560c5-c5f0-497b-a30d-fa3a44caae3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 01 15:57:04.572419 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:04.572381 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-r5m84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 01 15:57:04.572586 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:04.572443 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" podUID="afd20bf6-a51f-4006-85a9-1db75176c696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 01 15:57:05.534432 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:05.534389 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dh7c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 01 15:57:05.534819 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:05.534448 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" podUID="5eb560c5-c5f0-497b-a30d-fa3a44caae3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 01 15:57:05.573201 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:05.573169 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-r5m84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 01 15:57:05.573334 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:05.573213 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" podUID="afd20bf6-a51f-4006-85a9-1db75176c696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 01 15:57:06.533583 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:06.533539 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dh7c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 01 15:57:06.533771 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:06.533609 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" podUID="5eb560c5-c5f0-497b-a30d-fa3a44caae3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 01 15:57:06.573170 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:06.573139 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-r5m84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 01 15:57:06.573531 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:06.573181 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" podUID="afd20bf6-a51f-4006-85a9-1db75176c696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 01 15:57:07.533552 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:07.533514 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dh7c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 01 15:57:07.533731 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:07.533572 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" podUID="5eb560c5-c5f0-497b-a30d-fa3a44caae3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 01 15:57:07.573326 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:07.573299 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-r5m84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 01 15:57:07.573639 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:07.573343 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" podUID="afd20bf6-a51f-4006-85a9-1db75176c696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 01 15:57:08.533236 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:08.533201 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dh7c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 01 15:57:08.533414 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:08.533249 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" podUID="5eb560c5-c5f0-497b-a30d-fa3a44caae3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 01 15:57:08.572507 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:08.572465 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-r5m84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 01 15:57:08.572696 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:08.572524 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" podUID="afd20bf6-a51f-4006-85a9-1db75176c696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 01 15:57:10.185248 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:10.185212 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" Nov 01 15:57:10.186057 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:10.186010 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" Nov 01 15:57:10.186887 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:10.186869 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" Nov 01 15:57:10.189369 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:10.189348 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" Nov 01 15:57:10.189649 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:10.189626 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" Nov 01 15:57:10.190298 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:10.190278 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" Nov 01 15:57:18.310838 ip-10-0-128-238 kubenswrapper[4363]: E1101 15:57:18.310733 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 15:57:22.831011 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:22.830975 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-l5qnm"] Nov 01 15:57:22.841631 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:22.841606 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-l5qnm" Nov 01 15:57:22.843319 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:22.843295 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-l5qnm"] Nov 01 15:57:22.907325 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:22.907297 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 01 15:57:22.907563 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:22.907322 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-mkx9l\"" Nov 01 15:57:23.018427 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:23.018384 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dae26a45-6a59-4c32-a565-704e50c5f271-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-l5qnm\" (UID: \"dae26a45-6a59-4c32-a565-704e50c5f271\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-l5qnm" Nov 01 15:57:23.018427 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:23.018432 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkj6l\" (UniqueName: \"kubernetes.io/projected/dae26a45-6a59-4c32-a565-704e50c5f271-kube-api-access-nkj6l\") pod \"payload-processing-55fbbbf9dc-l5qnm\" (UID: \"dae26a45-6a59-4c32-a565-704e50c5f271\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-l5qnm" Nov 01 15:57:23.119870 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:23.119834 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dae26a45-6a59-4c32-a565-704e50c5f271-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-l5qnm\" (UID: \"dae26a45-6a59-4c32-a565-704e50c5f271\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-l5qnm" Nov 01 15:57:23.120054 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:23.119876 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkj6l\" (UniqueName: \"kubernetes.io/projected/dae26a45-6a59-4c32-a565-704e50c5f271-kube-api-access-nkj6l\") pod \"payload-processing-55fbbbf9dc-l5qnm\" (UID: \"dae26a45-6a59-4c32-a565-704e50c5f271\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-l5qnm" Nov 01 15:57:23.120876 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:23.120848 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dae26a45-6a59-4c32-a565-704e50c5f271-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-l5qnm\" (UID: \"dae26a45-6a59-4c32-a565-704e50c5f271\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-l5qnm" Nov 01 15:57:23.127375 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:23.127350 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkj6l\" (UniqueName: \"kubernetes.io/projected/dae26a45-6a59-4c32-a565-704e50c5f271-kube-api-access-nkj6l\") pod \"payload-processing-55fbbbf9dc-l5qnm\" (UID: \"dae26a45-6a59-4c32-a565-704e50c5f271\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-l5qnm" Nov 01 15:57:23.157567 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:23.157542 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-l5qnm" Nov 01 15:57:23.428137 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:23.428111 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-l5qnm"] Nov 01 15:57:23.430052 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:57:23.430001 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddae26a45_6a59_4c32_a565_704e50c5f271.slice/crio-0f368dbab3baddf7279707a42d15ca726b318c6785ef2bf084fb0480d37666a3 WatchSource:0}: Error finding container 0f368dbab3baddf7279707a42d15ca726b318c6785ef2bf084fb0480d37666a3: Status 404 returned error can't find the container with id 0f368dbab3baddf7279707a42d15ca726b318c6785ef2bf084fb0480d37666a3 Nov 01 15:57:24.156141 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:24.156096 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-l5qnm" event={"ID":"dae26a45-6a59-4c32-a565-704e50c5f271","Type":"ContainerStarted","Data":"0f368dbab3baddf7279707a42d15ca726b318c6785ef2bf084fb0480d37666a3"} Nov 01 15:57:26.173539 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:26.173494 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-l5qnm" event={"ID":"dae26a45-6a59-4c32-a565-704e50c5f271","Type":"ContainerStarted","Data":"940c0d24e582dab83bcc774bc20d2a3c06e22a037b92eac775af61812d2a2e00"} Nov 01 15:57:26.173937 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:26.173676 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-l5qnm" Nov 01 15:57:26.190420 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:26.190369 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-l5qnm" podStartSLOduration=2.051099222 podStartE2EDuration="4.190353689s" totalImagesPullingTime="2.139254467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:57:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:57:26.189132117 +0000 UTC m=+579.520869766" watchObservedRunningTime="2026-11-01 15:57:26.190353689 +0000 UTC m=+579.522091338" Nov 01 15:57:37.181748 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:37.181671 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-l5qnm" Nov 01 15:57:46.274994 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:46.274959 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x"] Nov 01 15:57:46.278589 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:46.278572 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" Nov 01 15:57:46.290547 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:46.290521 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x"] Nov 01 15:57:46.340740 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:46.340702 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4387988a-aa6e-4596-b718-09b8b6de4ac9-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q757x\" (UID: \"4387988a-aa6e-4596-b718-09b8b6de4ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" Nov 01 15:57:46.340893 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:46.340758 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4387988a-aa6e-4596-b718-09b8b6de4ac9-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q757x\" (UID: \"4387988a-aa6e-4596-b718-09b8b6de4ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" Nov 01 15:57:46.340893 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:46.340855 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4387988a-aa6e-4596-b718-09b8b6de4ac9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q757x\" (UID: \"4387988a-aa6e-4596-b718-09b8b6de4ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" Nov 01 15:57:46.340893 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:46.340889 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcjwp\" (UniqueName: \"kubernetes.io/projected/4387988a-aa6e-4596-b718-09b8b6de4ac9-kube-api-access-hcjwp\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q757x\" (UID: \"4387988a-aa6e-4596-b718-09b8b6de4ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" Nov 01 15:57:46.341067 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:46.340927 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4387988a-aa6e-4596-b718-09b8b6de4ac9-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q757x\" (UID: \"4387988a-aa6e-4596-b718-09b8b6de4ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" Nov 01 15:57:46.341067 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:46.341031 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4387988a-aa6e-4596-b718-09b8b6de4ac9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q757x\" (UID: \"4387988a-aa6e-4596-b718-09b8b6de4ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" Nov 01 15:57:46.341067 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:46.341051 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4387988a-aa6e-4596-b718-09b8b6de4ac9-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q757x\" (UID: \"4387988a-aa6e-4596-b718-09b8b6de4ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" Nov 01 15:57:46.341228 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:46.341071 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4387988a-aa6e-4596-b718-09b8b6de4ac9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q757x\" (UID: \"4387988a-aa6e-4596-b718-09b8b6de4ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" Nov 01 15:57:46.341228 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:46.341174 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4387988a-aa6e-4596-b718-09b8b6de4ac9-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q757x\" (UID: \"4387988a-aa6e-4596-b718-09b8b6de4ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" Nov 01 15:57:46.441858 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:46.441827 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4387988a-aa6e-4596-b718-09b8b6de4ac9-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q757x\" (UID: \"4387988a-aa6e-4596-b718-09b8b6de4ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" Nov 01 15:57:46.442051 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:46.441872 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4387988a-aa6e-4596-b718-09b8b6de4ac9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q757x\" (UID: \"4387988a-aa6e-4596-b718-09b8b6de4ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" Nov 01 15:57:46.442051 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:46.441888 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4387988a-aa6e-4596-b718-09b8b6de4ac9-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q757x\" (UID: \"4387988a-aa6e-4596-b718-09b8b6de4ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" Nov 01 15:57:46.442051 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:46.441903 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4387988a-aa6e-4596-b718-09b8b6de4ac9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q757x\" (UID: \"4387988a-aa6e-4596-b718-09b8b6de4ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" Nov 01 15:57:46.442051 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:46.441924 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4387988a-aa6e-4596-b718-09b8b6de4ac9-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q757x\" (UID: \"4387988a-aa6e-4596-b718-09b8b6de4ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" Nov 01 15:57:46.442051 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:46.441961 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4387988a-aa6e-4596-b718-09b8b6de4ac9-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q757x\" (UID: \"4387988a-aa6e-4596-b718-09b8b6de4ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" Nov 01 15:57:46.442051 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:46.441999 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4387988a-aa6e-4596-b718-09b8b6de4ac9-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q757x\" (UID: \"4387988a-aa6e-4596-b718-09b8b6de4ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" Nov 01 15:57:46.442051 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:46.442051 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4387988a-aa6e-4596-b718-09b8b6de4ac9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q757x\" (UID: \"4387988a-aa6e-4596-b718-09b8b6de4ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" Nov 01 15:57:46.442406 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:46.442081 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcjwp\" (UniqueName: \"kubernetes.io/projected/4387988a-aa6e-4596-b718-09b8b6de4ac9-kube-api-access-hcjwp\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q757x\" (UID: \"4387988a-aa6e-4596-b718-09b8b6de4ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" Nov 01 15:57:46.442406 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:46.442299 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4387988a-aa6e-4596-b718-09b8b6de4ac9-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q757x\" (UID: \"4387988a-aa6e-4596-b718-09b8b6de4ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" Nov 01 15:57:46.442545 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:46.442525 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4387988a-aa6e-4596-b718-09b8b6de4ac9-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q757x\" (UID: \"4387988a-aa6e-4596-b718-09b8b6de4ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" Nov 01 15:57:46.442617 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:46.442596 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4387988a-aa6e-4596-b718-09b8b6de4ac9-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q757x\" (UID: \"4387988a-aa6e-4596-b718-09b8b6de4ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" Nov 01 15:57:46.442651 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:46.442607 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4387988a-aa6e-4596-b718-09b8b6de4ac9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q757x\" (UID: \"4387988a-aa6e-4596-b718-09b8b6de4ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" Nov 01 15:57:46.442700 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:46.442683 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4387988a-aa6e-4596-b718-09b8b6de4ac9-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q757x\" (UID: \"4387988a-aa6e-4596-b718-09b8b6de4ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" Nov 01 15:57:46.444500 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:46.444476 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4387988a-aa6e-4596-b718-09b8b6de4ac9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q757x\" (UID: \"4387988a-aa6e-4596-b718-09b8b6de4ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" Nov 01 15:57:46.444590 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:46.444574 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4387988a-aa6e-4596-b718-09b8b6de4ac9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q757x\" (UID: \"4387988a-aa6e-4596-b718-09b8b6de4ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" Nov 01 15:57:46.449735 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:46.449712 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4387988a-aa6e-4596-b718-09b8b6de4ac9-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q757x\" (UID: \"4387988a-aa6e-4596-b718-09b8b6de4ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" Nov 01 15:57:46.449819 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:46.449793 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcjwp\" (UniqueName: \"kubernetes.io/projected/4387988a-aa6e-4596-b718-09b8b6de4ac9-kube-api-access-hcjwp\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q757x\" (UID: \"4387988a-aa6e-4596-b718-09b8b6de4ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" Nov 01 15:57:46.592847 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:46.592819 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" Nov 01 15:57:46.833336 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:46.833310 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtwhw_8254aafa-dd5b-4150-87e4-d165dab6bcb5/ovn-acl-logging/0.log" Nov 01 15:57:46.836530 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:46.836510 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtwhw_8254aafa-dd5b-4150-87e4-d165dab6bcb5/ovn-acl-logging/0.log" Nov 01 15:57:46.870812 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:46.870792 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x"] Nov 01 15:57:46.872970 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:57:46.872945 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4387988a_aa6e_4596_b718_09b8b6de4ac9.slice/crio-7f358b106c41ca958190eba96818488509d6f3fb3cab53668686835419e18f13 WatchSource:0}: Error finding container 7f358b106c41ca958190eba96818488509d6f3fb3cab53668686835419e18f13: Status 404 returned error can't find the container with id 7f358b106c41ca958190eba96818488509d6f3fb3cab53668686835419e18f13 Nov 01 15:57:46.875658 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:46.875630 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:57:46.875745 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:46.875690 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:57:46.875745 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:46.875731 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:57:47.275564 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:47.275473 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" event={"ID":"4387988a-aa6e-4596-b718-09b8b6de4ac9","Type":"ContainerStarted","Data":"ab87e1f3aca756b5f373a56bd47328eed33ade55968ec662b6b2dce42ca51600"} Nov 01 15:57:47.275564 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:47.275528 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" event={"ID":"4387988a-aa6e-4596-b718-09b8b6de4ac9","Type":"ContainerStarted","Data":"7f358b106c41ca958190eba96818488509d6f3fb3cab53668686835419e18f13"} Nov 01 15:57:47.303644 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:47.303592 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" podStartSLOduration=1.303577427 podStartE2EDuration="1.303577427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:57:47.302565278 +0000 UTC m=+600.634302963" watchObservedRunningTime="2026-11-01 15:57:47.303577427 +0000 UTC m=+600.635315074" Nov 01 15:57:47.602041 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:47.596439 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" Nov 01 15:57:47.602041 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:47.599188 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-q757x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 01 15:57:47.602041 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:47.599241 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" podUID="4387988a-aa6e-4596-b718-09b8b6de4ac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 01 15:57:48.327539 ip-10-0-128-238 kubenswrapper[4363]: E1101 15:57:48.327504 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 15:57:48.602385 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:48.602349 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-q757x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 01 15:57:48.602546 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:48.602403 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" podUID="4387988a-aa6e-4596-b718-09b8b6de4ac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 01 15:57:49.600040 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:49.595370 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-q757x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 01 15:57:49.600040 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:49.595439 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" podUID="4387988a-aa6e-4596-b718-09b8b6de4ac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 01 15:57:50.602147 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:50.602109 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-q757x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 01 15:57:50.602578 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:50.602174 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" podUID="4387988a-aa6e-4596-b718-09b8b6de4ac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 01 15:57:51.596393 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:51.596358 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-q757x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 01 15:57:51.596785 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:51.596751 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" podUID="4387988a-aa6e-4596-b718-09b8b6de4ac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 01 15:57:52.596430 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:52.596390 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-q757x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 01 15:57:52.597041 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:52.596982 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" podUID="4387988a-aa6e-4596-b718-09b8b6de4ac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 01 15:57:53.593825 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:53.593777 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-q757x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 01 15:57:53.594049 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:53.593845 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" podUID="4387988a-aa6e-4596-b718-09b8b6de4ac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 01 15:57:54.995488 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:54.995454 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" Nov 01 15:57:54.996093 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:54.996067 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" Nov 01 15:57:54.996903 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:57:54.996882 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" Nov 01 15:58:13.558007 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.557971 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf"] Nov 01 15:58:13.561517 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.561500 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" Nov 01 15:58:13.586835 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.586804 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf"] Nov 01 15:58:13.628615 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.628578 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-istio-envoy\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf\" (UID: \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" Nov 01 15:58:13.628615 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.628588 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-dfff59-openshift-default-dockercfg-bzb6m\"" Nov 01 15:58:13.628823 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.628653 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twwp8\" (UniqueName: \"kubernetes.io/projected/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-kube-api-access-twwp8\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf\" (UID: \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" Nov 01 15:58:13.628823 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.628702 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-workload-certs\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf\" (UID: \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" Nov 01 15:58:13.628823 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.628727 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-workload-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf\" (UID: \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" Nov 01 15:58:13.628823 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.628746 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-istio-podinfo\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf\" (UID: \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" Nov 01 15:58:13.629029 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.628883 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-credential-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf\" (UID: \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" Nov 01 15:58:13.629029 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.628910 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-istiod-ca-cert\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf\" (UID: \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" Nov 01 15:58:13.629029 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.628970 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-istio-data\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf\" (UID: \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" Nov 01 15:58:13.629029 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.629000 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-istio-token\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf\" (UID: \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" Nov 01 15:58:13.710920 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.710888 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8"] Nov 01 15:58:13.715579 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.715558 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" Nov 01 15:58:13.725633 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.725607 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-9f4dc9-openshift-default-dockercfg-tmp6r\"" Nov 01 15:58:13.729785 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.729757 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-workload-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf\" (UID: \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" Nov 01 15:58:13.729923 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.729801 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-istio-podinfo\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf\" (UID: \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" Nov 01 15:58:13.729923 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.729866 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-credential-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf\" (UID: \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" Nov 01 15:58:13.729923 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.729902 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-istiod-ca-cert\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf\" (UID: \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" Nov 01 15:58:13.730108 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.729945 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-istio-data\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf\" (UID: \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" Nov 01 15:58:13.730108 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.729984 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-istio-token\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf\" (UID: \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" Nov 01 15:58:13.730108 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.730095 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-istio-envoy\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf\" (UID: \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" Nov 01 15:58:13.730265 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.730133 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twwp8\" (UniqueName: \"kubernetes.io/projected/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-kube-api-access-twwp8\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf\" (UID: \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" Nov 01 15:58:13.730265 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.730191 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-workload-certs\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf\" (UID: \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" Nov 01 15:58:13.730599 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.730573 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-workload-certs\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf\" (UID: \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" Nov 01 15:58:13.730734 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.730709 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-workload-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf\" (UID: \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" Nov 01 15:58:13.731650 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.731628 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-credential-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf\" (UID: \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" Nov 01 15:58:13.731976 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.731954 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-istiod-ca-cert\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf\" (UID: \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" Nov 01 15:58:13.733228 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.733192 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8"] Nov 01 15:58:13.733339 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.733249 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-istio-data\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf\" (UID: \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" Nov 01 15:58:13.733663 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.733638 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-istio-envoy\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf\" (UID: \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" Nov 01 15:58:13.733931 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.733914 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-istio-podinfo\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf\" (UID: \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" Nov 01 15:58:13.744253 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.744231 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-istio-token\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf\" (UID: \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" Nov 01 15:58:13.744372 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.744348 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twwp8\" (UniqueName: \"kubernetes.io/projected/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-kube-api-access-twwp8\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf\" (UID: \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" Nov 01 15:58:13.831588 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.831500 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69f7030b-22ab-4f75-8705-af57a56c29f5-istio-envoy\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8\" (UID: \"69f7030b-22ab-4f75-8705-af57a56c29f5\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" Nov 01 15:58:13.831588 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.831542 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9xdq\" (UniqueName: \"kubernetes.io/projected/69f7030b-22ab-4f75-8705-af57a56c29f5-kube-api-access-q9xdq\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8\" (UID: \"69f7030b-22ab-4f75-8705-af57a56c29f5\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" Nov 01 15:58:13.831588 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.831571 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69f7030b-22ab-4f75-8705-af57a56c29f5-istio-data\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8\" (UID: \"69f7030b-22ab-4f75-8705-af57a56c29f5\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" Nov 01 15:58:13.831854 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.831684 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69f7030b-22ab-4f75-8705-af57a56c29f5-workload-certs\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8\" (UID: \"69f7030b-22ab-4f75-8705-af57a56c29f5\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" Nov 01 15:58:13.831854 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.831730 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69f7030b-22ab-4f75-8705-af57a56c29f5-workload-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8\" (UID: \"69f7030b-22ab-4f75-8705-af57a56c29f5\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" Nov 01 15:58:13.831854 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.831755 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69f7030b-22ab-4f75-8705-af57a56c29f5-credential-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8\" (UID: \"69f7030b-22ab-4f75-8705-af57a56c29f5\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" Nov 01 15:58:13.831854 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.831793 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69f7030b-22ab-4f75-8705-af57a56c29f5-istio-token\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8\" (UID: \"69f7030b-22ab-4f75-8705-af57a56c29f5\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" Nov 01 15:58:13.831854 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.831845 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69f7030b-22ab-4f75-8705-af57a56c29f5-istio-podinfo\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8\" (UID: \"69f7030b-22ab-4f75-8705-af57a56c29f5\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" Nov 01 15:58:13.832619 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.831884 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69f7030b-22ab-4f75-8705-af57a56c29f5-istiod-ca-cert\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8\" (UID: \"69f7030b-22ab-4f75-8705-af57a56c29f5\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" Nov 01 15:58:13.875257 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.875207 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" Nov 01 15:58:13.932771 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.932739 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69f7030b-22ab-4f75-8705-af57a56c29f5-workload-certs\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8\" (UID: \"69f7030b-22ab-4f75-8705-af57a56c29f5\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" Nov 01 15:58:13.933028 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.932987 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69f7030b-22ab-4f75-8705-af57a56c29f5-workload-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8\" (UID: \"69f7030b-22ab-4f75-8705-af57a56c29f5\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" Nov 01 15:58:13.933157 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.933075 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69f7030b-22ab-4f75-8705-af57a56c29f5-credential-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8\" (UID: \"69f7030b-22ab-4f75-8705-af57a56c29f5\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" Nov 01 15:58:13.933157 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.933096 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69f7030b-22ab-4f75-8705-af57a56c29f5-workload-certs\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8\" (UID: \"69f7030b-22ab-4f75-8705-af57a56c29f5\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" Nov 01 15:58:13.933505 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.933463 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69f7030b-22ab-4f75-8705-af57a56c29f5-credential-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8\" (UID: \"69f7030b-22ab-4f75-8705-af57a56c29f5\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" Nov 01 15:58:13.933505 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.933456 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69f7030b-22ab-4f75-8705-af57a56c29f5-workload-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8\" (UID: \"69f7030b-22ab-4f75-8705-af57a56c29f5\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" Nov 01 15:58:13.933645 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.933533 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69f7030b-22ab-4f75-8705-af57a56c29f5-istio-token\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8\" (UID: \"69f7030b-22ab-4f75-8705-af57a56c29f5\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" Nov 01 15:58:13.933645 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.933611 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69f7030b-22ab-4f75-8705-af57a56c29f5-istio-podinfo\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8\" (UID: \"69f7030b-22ab-4f75-8705-af57a56c29f5\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" Nov 01 15:58:13.934686 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.934659 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69f7030b-22ab-4f75-8705-af57a56c29f5-istiod-ca-cert\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8\" (UID: \"69f7030b-22ab-4f75-8705-af57a56c29f5\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" Nov 01 15:58:13.934830 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.933648 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69f7030b-22ab-4f75-8705-af57a56c29f5-istiod-ca-cert\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8\" (UID: \"69f7030b-22ab-4f75-8705-af57a56c29f5\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" Nov 01 15:58:13.935058 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.934977 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69f7030b-22ab-4f75-8705-af57a56c29f5-istio-envoy\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8\" (UID: \"69f7030b-22ab-4f75-8705-af57a56c29f5\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" Nov 01 15:58:13.935164 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.935064 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9xdq\" (UniqueName: \"kubernetes.io/projected/69f7030b-22ab-4f75-8705-af57a56c29f5-kube-api-access-q9xdq\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8\" (UID: \"69f7030b-22ab-4f75-8705-af57a56c29f5\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" Nov 01 15:58:13.935164 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.935089 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69f7030b-22ab-4f75-8705-af57a56c29f5-istio-data\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8\" (UID: \"69f7030b-22ab-4f75-8705-af57a56c29f5\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" Nov 01 15:58:13.935640 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.935428 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69f7030b-22ab-4f75-8705-af57a56c29f5-istio-data\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8\" (UID: \"69f7030b-22ab-4f75-8705-af57a56c29f5\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" Nov 01 15:58:13.937774 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.937748 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69f7030b-22ab-4f75-8705-af57a56c29f5-istio-envoy\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8\" (UID: \"69f7030b-22ab-4f75-8705-af57a56c29f5\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" Nov 01 15:58:13.939210 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.939186 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69f7030b-22ab-4f75-8705-af57a56c29f5-istio-podinfo\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8\" (UID: \"69f7030b-22ab-4f75-8705-af57a56c29f5\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" Nov 01 15:58:13.942902 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.942833 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69f7030b-22ab-4f75-8705-af57a56c29f5-istio-token\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8\" (UID: \"69f7030b-22ab-4f75-8705-af57a56c29f5\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" Nov 01 15:58:13.945091 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:13.944779 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9xdq\" (UniqueName: \"kubernetes.io/projected/69f7030b-22ab-4f75-8705-af57a56c29f5-kube-api-access-q9xdq\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8\" (UID: \"69f7030b-22ab-4f75-8705-af57a56c29f5\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" Nov 01 15:58:14.029427 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:14.029388 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" Nov 01 15:58:14.195150 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:14.195122 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf"] Nov 01 15:58:14.197309 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:58:14.197272 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e7a1c82_7d0f_4304_b4dc_5c36b0f6ab12.slice/crio-ee0ad12683ed464798009188d69966759e24d3ddf75244201b01515070c310c7 WatchSource:0}: Error finding container ee0ad12683ed464798009188d69966759e24d3ddf75244201b01515070c310c7: Status 404 returned error can't find the container with id ee0ad12683ed464798009188d69966759e24d3ddf75244201b01515070c310c7 Nov 01 15:58:14.199399 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:14.199361 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:58:14.199497 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:14.199450 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:58:14.199558 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:14.199494 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:58:14.376722 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:14.376603 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8"] Nov 01 15:58:14.380055 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:58:14.380009 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69f7030b_22ab_4f75_8705_af57a56c29f5.slice/crio-e0b7c77b7346e5143be3e28b90579858554586f2a6b9d6dc33728260dfe13ede WatchSource:0}: Error finding container e0b7c77b7346e5143be3e28b90579858554586f2a6b9d6dc33728260dfe13ede: Status 404 returned error can't find the container with id e0b7c77b7346e5143be3e28b90579858554586f2a6b9d6dc33728260dfe13ede Nov 01 15:58:14.382570 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:14.382538 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:58:14.382679 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:14.382600 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:58:14.382679 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:14.382630 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:58:14.448390 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:14.448354 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" event={"ID":"69f7030b-22ab-4f75-8705-af57a56c29f5","Type":"ContainerStarted","Data":"e0b7c77b7346e5143be3e28b90579858554586f2a6b9d6dc33728260dfe13ede"} Nov 01 15:58:14.450752 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:14.450725 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" event={"ID":"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12","Type":"ContainerStarted","Data":"b39a2cceb2e7f80ace969b2f8ef8e83f046c29be469d309a114e4c8f25098b49"} Nov 01 15:58:14.450889 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:14.450771 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" event={"ID":"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12","Type":"ContainerStarted","Data":"ee0ad12683ed464798009188d69966759e24d3ddf75244201b01515070c310c7"} Nov 01 15:58:14.471560 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:14.471510 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" podStartSLOduration=1.471492016 podStartE2EDuration="1.471492016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:58:14.469273963 +0000 UTC m=+627.801011614" watchObservedRunningTime="2026-11-01 15:58:14.471492016 +0000 UTC m=+627.803229668" Nov 01 15:58:14.875471 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:14.875442 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" Nov 01 15:58:15.458091 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:15.458054 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" event={"ID":"69f7030b-22ab-4f75-8705-af57a56c29f5","Type":"ContainerStarted","Data":"600118d493d6e661c338eef2b89235dfcee2bc96d4c1467a14e048f0881400f1"} Nov 01 15:58:15.477929 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:15.477874 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" podStartSLOduration=2.477861014 podStartE2EDuration="2.477861014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:58:15.4764337 +0000 UTC m=+628.808171350" watchObservedRunningTime="2026-11-01 15:58:15.477861014 +0000 UTC m=+628.809598665" Nov 01 15:58:15.882764 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:15.882728 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" Nov 01 15:58:16.030131 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:16.030095 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" Nov 01 15:58:16.035346 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:16.035318 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" Nov 01 15:58:16.463704 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:16.463677 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" Nov 01 15:58:16.464078 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:16.464048 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" Nov 01 15:58:16.464482 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:16.464461 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" Nov 01 15:58:16.464698 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:16.464677 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" Nov 01 15:58:17.843264 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:17.843230 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-43fac8-84b47867b7-4gzrb"] Nov 01 15:58:17.847339 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:17.847313 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-43fac8-84b47867b7-4gzrb" Nov 01 15:58:17.855752 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:17.855729 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-43fac8\"" Nov 01 15:58:17.855870 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:17.855849 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-43fac8-dockercfg-t8qgw\"" Nov 01 15:58:17.857673 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:17.857610 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-43fac8-84b47867b7-4gzrb"] Nov 01 15:58:17.879972 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:17.879571 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb1a7676-ecc1-451e-ba91-83f91fdf1bed-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-43fac8-84b47867b7-4gzrb\" (UID: \"bb1a7676-ecc1-451e-ba91-83f91fdf1bed\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-43fac8-84b47867b7-4gzrb" Nov 01 15:58:17.879972 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:17.879740 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55hbk\" (UniqueName: \"kubernetes.io/projected/bb1a7676-ecc1-451e-ba91-83f91fdf1bed-kube-api-access-55hbk\") pod \"payload-pre-processing-e2e-worker-w6-43fac8-84b47867b7-4gzrb\" (UID: \"bb1a7676-ecc1-451e-ba91-83f91fdf1bed\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-43fac8-84b47867b7-4gzrb" Nov 01 15:58:17.985620 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:17.985575 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb1a7676-ecc1-451e-ba91-83f91fdf1bed-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-43fac8-84b47867b7-4gzrb\" (UID: \"bb1a7676-ecc1-451e-ba91-83f91fdf1bed\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-43fac8-84b47867b7-4gzrb" Nov 01 15:58:17.986895 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:17.986859 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55hbk\" (UniqueName: \"kubernetes.io/projected/bb1a7676-ecc1-451e-ba91-83f91fdf1bed-kube-api-access-55hbk\") pod \"payload-pre-processing-e2e-worker-w6-43fac8-84b47867b7-4gzrb\" (UID: \"bb1a7676-ecc1-451e-ba91-83f91fdf1bed\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-43fac8-84b47867b7-4gzrb" Nov 01 15:58:17.987637 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:17.986776 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb1a7676-ecc1-451e-ba91-83f91fdf1bed-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-43fac8-84b47867b7-4gzrb\" (UID: \"bb1a7676-ecc1-451e-ba91-83f91fdf1bed\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-43fac8-84b47867b7-4gzrb" Nov 01 15:58:17.999923 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:17.999889 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55hbk\" (UniqueName: \"kubernetes.io/projected/bb1a7676-ecc1-451e-ba91-83f91fdf1bed-kube-api-access-55hbk\") pod \"payload-pre-processing-e2e-worker-w6-43fac8-84b47867b7-4gzrb\" (UID: \"bb1a7676-ecc1-451e-ba91-83f91fdf1bed\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-43fac8-84b47867b7-4gzrb" Nov 01 15:58:18.162543 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:18.162456 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-43fac8-84b47867b7-4gzrb" Nov 01 15:58:18.309311 ip-10-0-128-238 kubenswrapper[4363]: E1101 15:58:18.309274 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 15:58:18.442909 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:18.442879 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-43fac8-84b47867b7-4gzrb"] Nov 01 15:58:18.445622 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:58:18.445592 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb1a7676_ecc1_451e_ba91_83f91fdf1bed.slice/crio-a967126e78bb6239e89bc977ef99c3fb11cb0c57052311b4ceeddcea4c5e751f WatchSource:0}: Error finding container a967126e78bb6239e89bc977ef99c3fb11cb0c57052311b4ceeddcea4c5e751f: Status 404 returned error can't find the container with id a967126e78bb6239e89bc977ef99c3fb11cb0c57052311b4ceeddcea4c5e751f Nov 01 15:58:18.475939 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:18.475912 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-43fac8-84b47867b7-4gzrb" event={"ID":"bb1a7676-ecc1-451e-ba91-83f91fdf1bed","Type":"ContainerStarted","Data":"a967126e78bb6239e89bc977ef99c3fb11cb0c57052311b4ceeddcea4c5e751f"} Nov 01 15:58:18.487880 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:18.487854 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4a083d-84c576c9b7-jlscm"] Nov 01 15:58:18.493225 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:18.493207 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-4a083d-84c576c9b7-jlscm" Nov 01 15:58:18.503003 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:18.501462 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4a083d-84c576c9b7-jlscm"] Nov 01 15:58:18.503003 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:18.501595 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-4a083d\"" Nov 01 15:58:18.503003 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:18.501960 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-4a083d-dockercfg-lpws5\"" Nov 01 15:58:18.594173 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:18.594140 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ptws\" (UniqueName: \"kubernetes.io/projected/a1a8daa3-0ee1-4164-9dab-1fdba6d81073-kube-api-access-7ptws\") pod \"payload-processing-e2e-worker-w0-4a083d-84c576c9b7-jlscm\" (UID: \"a1a8daa3-0ee1-4164-9dab-1fdba6d81073\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4a083d-84c576c9b7-jlscm" Nov 01 15:58:18.594387 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:18.594363 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a8daa3-0ee1-4164-9dab-1fdba6d81073-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-4a083d-84c576c9b7-jlscm\" (UID: \"a1a8daa3-0ee1-4164-9dab-1fdba6d81073\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4a083d-84c576c9b7-jlscm" Nov 01 15:58:18.695352 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:18.695266 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ptws\" (UniqueName: \"kubernetes.io/projected/a1a8daa3-0ee1-4164-9dab-1fdba6d81073-kube-api-access-7ptws\") pod \"payload-processing-e2e-worker-w0-4a083d-84c576c9b7-jlscm\" (UID: \"a1a8daa3-0ee1-4164-9dab-1fdba6d81073\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4a083d-84c576c9b7-jlscm" Nov 01 15:58:18.695352 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:18.695337 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a8daa3-0ee1-4164-9dab-1fdba6d81073-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-4a083d-84c576c9b7-jlscm\" (UID: \"a1a8daa3-0ee1-4164-9dab-1fdba6d81073\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4a083d-84c576c9b7-jlscm" Nov 01 15:58:18.696027 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:18.696001 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a8daa3-0ee1-4164-9dab-1fdba6d81073-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-4a083d-84c576c9b7-jlscm\" (UID: \"a1a8daa3-0ee1-4164-9dab-1fdba6d81073\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4a083d-84c576c9b7-jlscm" Nov 01 15:58:18.703284 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:18.703262 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ptws\" (UniqueName: \"kubernetes.io/projected/a1a8daa3-0ee1-4164-9dab-1fdba6d81073-kube-api-access-7ptws\") pod \"payload-processing-e2e-worker-w0-4a083d-84c576c9b7-jlscm\" (UID: \"a1a8daa3-0ee1-4164-9dab-1fdba6d81073\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4a083d-84c576c9b7-jlscm" Nov 01 15:58:18.809968 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:18.809935 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-4a083d-84c576c9b7-jlscm" Nov 01 15:58:19.114002 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:19.113770 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4a083d-84c576c9b7-jlscm"] Nov 01 15:58:19.119402 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:58:19.119223 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a8daa3_0ee1_4164_9dab_1fdba6d81073.slice/crio-1a61272de52eb30083f5e818c2578cc11a9f3a414bb14db4ae236e48ccf8491f WatchSource:0}: Error finding container 1a61272de52eb30083f5e818c2578cc11a9f3a414bb14db4ae236e48ccf8491f: Status 404 returned error can't find the container with id 1a61272de52eb30083f5e818c2578cc11a9f3a414bb14db4ae236e48ccf8491f Nov 01 15:58:19.334565 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:19.334534 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-43fac8-ff59b5dfb-r8zjf"] Nov 01 15:58:19.339165 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:19.339139 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-43fac8-ff59b5dfb-r8zjf" Nov 01 15:58:19.348361 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:19.348317 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-43fac8-serving-cert\"" Nov 01 15:58:19.351034 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:19.350996 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-43fac8-ff59b5dfb-r8zjf"] Nov 01 15:58:19.402897 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:19.402822 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rzq2\" (UniqueName: \"kubernetes.io/projected/6be52ed0-203a-4a91-abee-9420e0dff7dd-kube-api-access-2rzq2\") pod \"maas-api-e2e-worker-w6-43fac8-ff59b5dfb-r8zjf\" (UID: \"6be52ed0-203a-4a91-abee-9420e0dff7dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-43fac8-ff59b5dfb-r8zjf" Nov 01 15:58:19.403078 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:19.402899 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6be52ed0-203a-4a91-abee-9420e0dff7dd-metrics-certs\") pod \"maas-api-e2e-worker-w6-43fac8-ff59b5dfb-r8zjf\" (UID: \"6be52ed0-203a-4a91-abee-9420e0dff7dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-43fac8-ff59b5dfb-r8zjf" Nov 01 15:58:19.403078 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:19.402926 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6be52ed0-203a-4a91-abee-9420e0dff7dd-maas-api-tls\") pod \"maas-api-e2e-worker-w6-43fac8-ff59b5dfb-r8zjf\" (UID: \"6be52ed0-203a-4a91-abee-9420e0dff7dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-43fac8-ff59b5dfb-r8zjf" Nov 01 15:58:19.403078 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:19.402941 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6be52ed0-203a-4a91-abee-9420e0dff7dd-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-43fac8-ff59b5dfb-r8zjf\" (UID: \"6be52ed0-203a-4a91-abee-9420e0dff7dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-43fac8-ff59b5dfb-r8zjf" Nov 01 15:58:19.482370 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:19.482332 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-43fac8-84b47867b7-4gzrb" event={"ID":"bb1a7676-ecc1-451e-ba91-83f91fdf1bed","Type":"ContainerStarted","Data":"2e2108f9def8588cd4c86249d485464e54f2df8836971eb7c1a8b872c24ab362"} Nov 01 15:58:19.482543 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:19.482401 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-43fac8-84b47867b7-4gzrb" Nov 01 15:58:19.483853 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:19.483825 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-4a083d-84c576c9b7-jlscm" event={"ID":"a1a8daa3-0ee1-4164-9dab-1fdba6d81073","Type":"ContainerStarted","Data":"f2fe23e94e94984fa2d17809d5fad34c4e7fe6cf3bfc2fc5bf17b5d2bb0df0c0"} Nov 01 15:58:19.483954 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:19.483874 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-4a083d-84c576c9b7-jlscm" event={"ID":"a1a8daa3-0ee1-4164-9dab-1fdba6d81073","Type":"ContainerStarted","Data":"1a61272de52eb30083f5e818c2578cc11a9f3a414bb14db4ae236e48ccf8491f"} Nov 01 15:58:19.483954 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:19.483928 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-4a083d-84c576c9b7-jlscm" Nov 01 15:58:19.500344 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:19.500299 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-43fac8-84b47867b7-4gzrb" podStartSLOduration=2.500285904 podStartE2EDuration="2.500285904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:58:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:58:19.498206638 +0000 UTC m=+632.829944288" watchObservedRunningTime="2026-11-01 15:58:19.500285904 +0000 UTC m=+632.832023554" Nov 01 15:58:19.504375 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:19.504355 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rzq2\" (UniqueName: \"kubernetes.io/projected/6be52ed0-203a-4a91-abee-9420e0dff7dd-kube-api-access-2rzq2\") pod \"maas-api-e2e-worker-w6-43fac8-ff59b5dfb-r8zjf\" (UID: \"6be52ed0-203a-4a91-abee-9420e0dff7dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-43fac8-ff59b5dfb-r8zjf" Nov 01 15:58:19.504440 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:19.504412 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6be52ed0-203a-4a91-abee-9420e0dff7dd-metrics-certs\") pod \"maas-api-e2e-worker-w6-43fac8-ff59b5dfb-r8zjf\" (UID: \"6be52ed0-203a-4a91-abee-9420e0dff7dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-43fac8-ff59b5dfb-r8zjf" Nov 01 15:58:19.504440 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:19.504429 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6be52ed0-203a-4a91-abee-9420e0dff7dd-maas-api-tls\") pod \"maas-api-e2e-worker-w6-43fac8-ff59b5dfb-r8zjf\" (UID: \"6be52ed0-203a-4a91-abee-9420e0dff7dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-43fac8-ff59b5dfb-r8zjf" Nov 01 15:58:19.504514 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:19.504447 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6be52ed0-203a-4a91-abee-9420e0dff7dd-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-43fac8-ff59b5dfb-r8zjf\" (UID: \"6be52ed0-203a-4a91-abee-9420e0dff7dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-43fac8-ff59b5dfb-r8zjf" Nov 01 15:58:19.505095 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:19.505070 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6be52ed0-203a-4a91-abee-9420e0dff7dd-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-43fac8-ff59b5dfb-r8zjf\" (UID: \"6be52ed0-203a-4a91-abee-9420e0dff7dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-43fac8-ff59b5dfb-r8zjf" Nov 01 15:58:19.507032 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:19.506992 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6be52ed0-203a-4a91-abee-9420e0dff7dd-maas-api-tls\") pod \"maas-api-e2e-worker-w6-43fac8-ff59b5dfb-r8zjf\" (UID: \"6be52ed0-203a-4a91-abee-9420e0dff7dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-43fac8-ff59b5dfb-r8zjf" Nov 01 15:58:19.507222 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:19.507204 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6be52ed0-203a-4a91-abee-9420e0dff7dd-metrics-certs\") pod \"maas-api-e2e-worker-w6-43fac8-ff59b5dfb-r8zjf\" (UID: \"6be52ed0-203a-4a91-abee-9420e0dff7dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-43fac8-ff59b5dfb-r8zjf" Nov 01 15:58:19.512228 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:19.512208 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rzq2\" (UniqueName: \"kubernetes.io/projected/6be52ed0-203a-4a91-abee-9420e0dff7dd-kube-api-access-2rzq2\") pod \"maas-api-e2e-worker-w6-43fac8-ff59b5dfb-r8zjf\" (UID: \"6be52ed0-203a-4a91-abee-9420e0dff7dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-43fac8-ff59b5dfb-r8zjf" Nov 01 15:58:19.524047 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:19.522322 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-4a083d-84c576c9b7-jlscm" podStartSLOduration=1.522306479 podStartE2EDuration="1.522306479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:58:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:58:19.52149269 +0000 UTC m=+632.853230370" watchObservedRunningTime="2026-11-01 15:58:19.522306479 +0000 UTC m=+632.854044131" Nov 01 15:58:19.661398 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:19.660294 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-43fac8-ff59b5dfb-r8zjf" Nov 01 15:58:19.947043 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:19.946999 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-43fac8-ff59b5dfb-r8zjf"] Nov 01 15:58:19.948486 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:58:19.948454 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6be52ed0_203a_4a91_abee_9420e0dff7dd.slice/crio-f99d66d888075aa206496af3a504e2d73c1c3be25fff2a348a259e2dfec3b17a WatchSource:0}: Error finding container f99d66d888075aa206496af3a504e2d73c1c3be25fff2a348a259e2dfec3b17a: Status 404 returned error can't find the container with id f99d66d888075aa206496af3a504e2d73c1c3be25fff2a348a259e2dfec3b17a Nov 01 15:58:19.949941 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:19.949920 4363 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 15:58:20.141417 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:20.141377 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-dfff59-688dc9df89-4vk2r"] Nov 01 15:58:20.146710 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:20.146688 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-dfff59-688dc9df89-4vk2r" Nov 01 15:58:20.154007 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:20.153983 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-dfff59\"" Nov 01 15:58:20.154557 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:20.154517 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-dfff59-dockercfg-kkdbp\"" Nov 01 15:58:20.156825 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:20.156796 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-dfff59-688dc9df89-4vk2r"] Nov 01 15:58:20.241089 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:20.240514 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/247d3008-5891-46e2-843b-da07c4329f01-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-dfff59-688dc9df89-4vk2r\" (UID: \"247d3008-5891-46e2-843b-da07c4329f01\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-dfff59-688dc9df89-4vk2r" Nov 01 15:58:20.241089 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:20.240661 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx858\" (UniqueName: \"kubernetes.io/projected/247d3008-5891-46e2-843b-da07c4329f01-kube-api-access-tx858\") pod \"payload-processing-e2e-worker-w1-dfff59-688dc9df89-4vk2r\" (UID: \"247d3008-5891-46e2-843b-da07c4329f01\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-dfff59-688dc9df89-4vk2r" Nov 01 15:58:20.341919 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:20.341885 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/247d3008-5891-46e2-843b-da07c4329f01-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-dfff59-688dc9df89-4vk2r\" (UID: \"247d3008-5891-46e2-843b-da07c4329f01\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-dfff59-688dc9df89-4vk2r" Nov 01 15:58:20.342135 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:20.341973 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx858\" (UniqueName: \"kubernetes.io/projected/247d3008-5891-46e2-843b-da07c4329f01-kube-api-access-tx858\") pod \"payload-processing-e2e-worker-w1-dfff59-688dc9df89-4vk2r\" (UID: \"247d3008-5891-46e2-843b-da07c4329f01\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-dfff59-688dc9df89-4vk2r" Nov 01 15:58:20.342692 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:20.342662 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/247d3008-5891-46e2-843b-da07c4329f01-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-dfff59-688dc9df89-4vk2r\" (UID: \"247d3008-5891-46e2-843b-da07c4329f01\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-dfff59-688dc9df89-4vk2r" Nov 01 15:58:20.350868 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:20.350814 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx858\" (UniqueName: \"kubernetes.io/projected/247d3008-5891-46e2-843b-da07c4329f01-kube-api-access-tx858\") pod \"payload-processing-e2e-worker-w1-dfff59-688dc9df89-4vk2r\" (UID: \"247d3008-5891-46e2-843b-da07c4329f01\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-dfff59-688dc9df89-4vk2r" Nov 01 15:58:20.395807 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:20.395773 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-bd831a-56899656f5-jhvtb"] Nov 01 15:58:20.399664 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:20.399640 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bd831a-56899656f5-jhvtb" Nov 01 15:58:20.411633 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:20.411605 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-bd831a-56899656f5-jhvtb"] Nov 01 15:58:20.461943 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:20.461849 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-dfff59-688dc9df89-4vk2r" Nov 01 15:58:20.482060 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:20.482034 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-bd831a\"" Nov 01 15:58:20.482641 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:20.482615 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-bd831a-dockercfg-whrbm\"" Nov 01 15:58:20.523368 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:20.509783 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-43fac8-ff59b5dfb-r8zjf" event={"ID":"6be52ed0-203a-4a91-abee-9420e0dff7dd","Type":"ContainerStarted","Data":"f99d66d888075aa206496af3a504e2d73c1c3be25fff2a348a259e2dfec3b17a"} Nov 01 15:58:20.544551 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:20.544298 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g7pr\" (UniqueName: \"kubernetes.io/projected/bc5eb74e-c5d9-4210-a626-94e8279c3a36-kube-api-access-8g7pr\") pod \"payload-pre-processing-e2e-worker-w3-bd831a-56899656f5-jhvtb\" (UID: \"bc5eb74e-c5d9-4210-a626-94e8279c3a36\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bd831a-56899656f5-jhvtb" Nov 01 15:58:20.544551 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:20.544491 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc5eb74e-c5d9-4210-a626-94e8279c3a36-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-bd831a-56899656f5-jhvtb\" (UID: \"bc5eb74e-c5d9-4210-a626-94e8279c3a36\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bd831a-56899656f5-jhvtb" Nov 01 15:58:20.645917 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:20.645164 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g7pr\" (UniqueName: \"kubernetes.io/projected/bc5eb74e-c5d9-4210-a626-94e8279c3a36-kube-api-access-8g7pr\") pod \"payload-pre-processing-e2e-worker-w3-bd831a-56899656f5-jhvtb\" (UID: \"bc5eb74e-c5d9-4210-a626-94e8279c3a36\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bd831a-56899656f5-jhvtb" Nov 01 15:58:20.645917 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:20.645342 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc5eb74e-c5d9-4210-a626-94e8279c3a36-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-bd831a-56899656f5-jhvtb\" (UID: \"bc5eb74e-c5d9-4210-a626-94e8279c3a36\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bd831a-56899656f5-jhvtb" Nov 01 15:58:20.647331 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:20.647297 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc5eb74e-c5d9-4210-a626-94e8279c3a36-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-bd831a-56899656f5-jhvtb\" (UID: \"bc5eb74e-c5d9-4210-a626-94e8279c3a36\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bd831a-56899656f5-jhvtb" Nov 01 15:58:20.652877 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:20.652854 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g7pr\" (UniqueName: \"kubernetes.io/projected/bc5eb74e-c5d9-4210-a626-94e8279c3a36-kube-api-access-8g7pr\") pod \"payload-pre-processing-e2e-worker-w3-bd831a-56899656f5-jhvtb\" (UID: \"bc5eb74e-c5d9-4210-a626-94e8279c3a36\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bd831a-56899656f5-jhvtb" Nov 01 15:58:20.715883 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:20.715851 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bd831a-56899656f5-jhvtb" Nov 01 15:58:20.815623 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:20.815581 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-dfff59-688dc9df89-4vk2r"] Nov 01 15:58:21.004322 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:21.004294 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-bd831a-56899656f5-jhvtb"] Nov 01 15:58:21.005394 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:58:21.005368 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc5eb74e_c5d9_4210_a626_94e8279c3a36.slice/crio-cf89c24952a1e8f1e5f1b9519862de32539e69da91c7a168986c724c4b5169dc WatchSource:0}: Error finding container cf89c24952a1e8f1e5f1b9519862de32539e69da91c7a168986c724c4b5169dc: Status 404 returned error can't find the container with id cf89c24952a1e8f1e5f1b9519862de32539e69da91c7a168986c724c4b5169dc Nov 01 15:58:21.533040 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:21.523273 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-dfff59-688dc9df89-4vk2r" event={"ID":"247d3008-5891-46e2-843b-da07c4329f01","Type":"ContainerStarted","Data":"396c50fadb78b5123057932d296478c829edabd3df4fbb67afcf7e0a79780955"} Nov 01 15:58:21.533040 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:21.523340 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-dfff59-688dc9df89-4vk2r" event={"ID":"247d3008-5891-46e2-843b-da07c4329f01","Type":"ContainerStarted","Data":"f55db40440ffe4a416df9ec52b31580f8c5ae44abe14ed76dfefe549c5c31b7a"} Nov 01 15:58:21.533040 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:21.524231 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-dfff59-688dc9df89-4vk2r" Nov 01 15:58:21.533040 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:21.527964 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-43fac8-ff59b5dfb-r8zjf" event={"ID":"6be52ed0-203a-4a91-abee-9420e0dff7dd","Type":"ContainerStarted","Data":"f61adfd659bef00b44fed69eb071f6d14300a9b8e907db26172a4c7b964273db"} Nov 01 15:58:21.533040 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:21.528634 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-43fac8-ff59b5dfb-r8zjf" Nov 01 15:58:21.533040 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:21.531857 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bd831a-56899656f5-jhvtb" event={"ID":"bc5eb74e-c5d9-4210-a626-94e8279c3a36","Type":"ContainerStarted","Data":"f1b7825cb139ffad509aba5ec68f675537e4175cd6a5b8bbd89b67c7316cebe0"} Nov 01 15:58:21.533040 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:21.531897 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bd831a-56899656f5-jhvtb" event={"ID":"bc5eb74e-c5d9-4210-a626-94e8279c3a36","Type":"ContainerStarted","Data":"cf89c24952a1e8f1e5f1b9519862de32539e69da91c7a168986c724c4b5169dc"} Nov 01 15:58:21.533040 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:21.532463 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bd831a-56899656f5-jhvtb" Nov 01 15:58:21.569638 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:21.569570 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-dfff59-688dc9df89-4vk2r" podStartSLOduration=1.569550316 podStartE2EDuration="1.569550316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:58:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:58:21.567926997 +0000 UTC m=+634.899664660" watchObservedRunningTime="2026-11-01 15:58:21.569550316 +0000 UTC m=+634.901287970" Nov 01 15:58:21.584272 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:21.584225 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bd831a-56899656f5-jhvtb" podStartSLOduration=1.58421234 podStartE2EDuration="1.58421234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:58:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:58:21.582590226 +0000 UTC m=+634.914327876" watchObservedRunningTime="2026-11-01 15:58:21.58421234 +0000 UTC m=+634.915949990" Nov 01 15:58:21.604283 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:21.604234 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-43fac8-ff59b5dfb-r8zjf" podStartSLOduration=2.092327526 podStartE2EDuration="2.604219555s" totalImagesPullingTime="511.892029ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:58:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:58:21.603186641 +0000 UTC m=+634.934924291" watchObservedRunningTime="2026-11-01 15:58:21.604219555 +0000 UTC m=+634.935957204" Nov 01 15:58:22.120535 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:22.120497 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-164db6-5746fd596c-rmbpv"] Nov 01 15:58:22.126412 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:22.126385 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-164db6-5746fd596c-rmbpv" Nov 01 15:58:22.136335 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:22.136308 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-164db6-serving-cert\"" Nov 01 15:58:22.138244 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:22.138215 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-164db6-5746fd596c-rmbpv"] Nov 01 15:58:22.275876 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:22.275844 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7q9q\" (UniqueName: \"kubernetes.io/projected/c6e54fd2-84cf-4043-b891-ba574ddef23b-kube-api-access-j7q9q\") pod \"maas-api-e2e-worker-w5-164db6-5746fd596c-rmbpv\" (UID: \"c6e54fd2-84cf-4043-b891-ba574ddef23b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-164db6-5746fd596c-rmbpv" Nov 01 15:58:22.276086 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:22.275993 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6e54fd2-84cf-4043-b891-ba574ddef23b-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-164db6-5746fd596c-rmbpv\" (UID: \"c6e54fd2-84cf-4043-b891-ba574ddef23b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-164db6-5746fd596c-rmbpv" Nov 01 15:58:22.276086 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:22.276064 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6e54fd2-84cf-4043-b891-ba574ddef23b-metrics-certs\") pod \"maas-api-e2e-worker-w5-164db6-5746fd596c-rmbpv\" (UID: \"c6e54fd2-84cf-4043-b891-ba574ddef23b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-164db6-5746fd596c-rmbpv" Nov 01 15:58:22.276186 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:22.276146 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6e54fd2-84cf-4043-b891-ba574ddef23b-maas-api-tls\") pod \"maas-api-e2e-worker-w5-164db6-5746fd596c-rmbpv\" (UID: \"c6e54fd2-84cf-4043-b891-ba574ddef23b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-164db6-5746fd596c-rmbpv" Nov 01 15:58:22.377086 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:22.376984 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6e54fd2-84cf-4043-b891-ba574ddef23b-metrics-certs\") pod \"maas-api-e2e-worker-w5-164db6-5746fd596c-rmbpv\" (UID: \"c6e54fd2-84cf-4043-b891-ba574ddef23b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-164db6-5746fd596c-rmbpv" Nov 01 15:58:22.377243 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:22.377091 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6e54fd2-84cf-4043-b891-ba574ddef23b-maas-api-tls\") pod \"maas-api-e2e-worker-w5-164db6-5746fd596c-rmbpv\" (UID: \"c6e54fd2-84cf-4043-b891-ba574ddef23b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-164db6-5746fd596c-rmbpv" Nov 01 15:58:22.377243 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:22.377153 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7q9q\" (UniqueName: \"kubernetes.io/projected/c6e54fd2-84cf-4043-b891-ba574ddef23b-kube-api-access-j7q9q\") pod \"maas-api-e2e-worker-w5-164db6-5746fd596c-rmbpv\" (UID: \"c6e54fd2-84cf-4043-b891-ba574ddef23b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-164db6-5746fd596c-rmbpv" Nov 01 15:58:22.377362 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:22.377273 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6e54fd2-84cf-4043-b891-ba574ddef23b-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-164db6-5746fd596c-rmbpv\" (UID: \"c6e54fd2-84cf-4043-b891-ba574ddef23b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-164db6-5746fd596c-rmbpv" Nov 01 15:58:22.378103 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:22.378072 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6e54fd2-84cf-4043-b891-ba574ddef23b-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-164db6-5746fd596c-rmbpv\" (UID: \"c6e54fd2-84cf-4043-b891-ba574ddef23b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-164db6-5746fd596c-rmbpv" Nov 01 15:58:22.379692 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:22.379660 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6e54fd2-84cf-4043-b891-ba574ddef23b-metrics-certs\") pod \"maas-api-e2e-worker-w5-164db6-5746fd596c-rmbpv\" (UID: \"c6e54fd2-84cf-4043-b891-ba574ddef23b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-164db6-5746fd596c-rmbpv" Nov 01 15:58:22.379818 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:22.379742 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6e54fd2-84cf-4043-b891-ba574ddef23b-maas-api-tls\") pod \"maas-api-e2e-worker-w5-164db6-5746fd596c-rmbpv\" (UID: \"c6e54fd2-84cf-4043-b891-ba574ddef23b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-164db6-5746fd596c-rmbpv" Nov 01 15:58:22.384728 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:22.384705 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7q9q\" (UniqueName: \"kubernetes.io/projected/c6e54fd2-84cf-4043-b891-ba574ddef23b-kube-api-access-j7q9q\") pod \"maas-api-e2e-worker-w5-164db6-5746fd596c-rmbpv\" (UID: \"c6e54fd2-84cf-4043-b891-ba574ddef23b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-164db6-5746fd596c-rmbpv" Nov 01 15:58:22.444153 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:22.444124 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-164db6-5746fd596c-rmbpv" Nov 01 15:58:22.821352 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:22.821200 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-164db6-5746fd596c-rmbpv"] Nov 01 15:58:22.825474 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:58:22.825442 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6e54fd2_84cf_4043_b891_ba574ddef23b.slice/crio-a55ab93182a831be377ea7e9b55b309406723a92dbfb37650fa7431e621e0edc WatchSource:0}: Error finding container a55ab93182a831be377ea7e9b55b309406723a92dbfb37650fa7431e621e0edc: Status 404 returned error can't find the container with id a55ab93182a831be377ea7e9b55b309406723a92dbfb37650fa7431e621e0edc Nov 01 15:58:23.547459 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:23.547376 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-164db6-5746fd596c-rmbpv" event={"ID":"c6e54fd2-84cf-4043-b891-ba574ddef23b","Type":"ContainerStarted","Data":"829758803f3353e8fcd3c0df278d7563debfbfdfbc34035d86bba6e723eafc70"} Nov 01 15:58:23.547971 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:23.547949 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-164db6-5746fd596c-rmbpv" Nov 01 15:58:23.548075 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:23.548002 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-164db6-5746fd596c-rmbpv" event={"ID":"c6e54fd2-84cf-4043-b891-ba574ddef23b","Type":"ContainerStarted","Data":"a55ab93182a831be377ea7e9b55b309406723a92dbfb37650fa7431e621e0edc"} Nov 01 15:58:24.824619 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:24.824557 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-164db6-5746fd596c-rmbpv" podStartSLOduration=2.408534915 podStartE2EDuration="2.82452928s" totalImagesPullingTime="415.994365ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:58:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:58:23.602339699 +0000 UTC m=+636.934077360" watchObservedRunningTime="2026-11-01 15:58:24.82452928 +0000 UTC m=+638.156266934" Nov 01 15:58:24.825652 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:24.825624 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b"] Nov 01 15:58:24.829978 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:24.829952 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b" Nov 01 15:58:24.836630 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:24.836601 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-6b0e0f-serving-cert\"" Nov 01 15:58:24.841100 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:24.841073 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b"] Nov 01 15:58:24.919074 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:24.919039 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2-maas-api-tls\") pod \"maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b\" (UID: \"45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b" Nov 01 15:58:24.919229 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:24.919092 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjpbw\" (UniqueName: \"kubernetes.io/projected/45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2-kube-api-access-tjpbw\") pod \"maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b\" (UID: \"45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b" Nov 01 15:58:24.919229 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:24.919213 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b\" (UID: \"45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b" Nov 01 15:58:24.919322 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:24.919262 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2-metrics-certs\") pod \"maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b\" (UID: \"45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b" Nov 01 15:58:25.020659 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:25.020619 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjpbw\" (UniqueName: \"kubernetes.io/projected/45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2-kube-api-access-tjpbw\") pod \"maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b\" (UID: \"45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b" Nov 01 15:58:25.020802 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:25.020751 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b\" (UID: \"45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b" Nov 01 15:58:25.020857 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:25.020817 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2-metrics-certs\") pod \"maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b\" (UID: \"45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b" Nov 01 15:58:25.020898 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:25.020882 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2-maas-api-tls\") pod \"maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b\" (UID: \"45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b" Nov 01 15:58:25.021078 ip-10-0-128-238 kubenswrapper[4363]: E1101 15:58:25.021038 4363 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-6b0e0f-serving-cert: secret "maas-api-e2e-worker-w2-6b0e0f-serving-cert" not found Nov 01 15:58:25.021202 ip-10-0-128-238 kubenswrapper[4363]: E1101 15:58:25.021133 4363 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2-maas-api-tls podName:45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2 nodeName:}" failed. No retries permitted until 2026-11-01 15:58:25.521112978 +0000 UTC m=+638.852850623 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2-maas-api-tls") pod "maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b" (UID: "45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2") : secret "maas-api-e2e-worker-w2-6b0e0f-serving-cert" not found Nov 01 15:58:25.021745 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:25.021722 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b\" (UID: \"45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b" Nov 01 15:58:25.023687 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:25.023666 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2-metrics-certs\") pod \"maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b\" (UID: \"45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b" Nov 01 15:58:25.029231 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:25.029207 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjpbw\" (UniqueName: \"kubernetes.io/projected/45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2-kube-api-access-tjpbw\") pod \"maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b\" (UID: \"45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b" Nov 01 15:58:25.528615 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:25.528578 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2-maas-api-tls\") pod \"maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b\" (UID: \"45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b" Nov 01 15:58:25.530954 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:25.530926 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2-maas-api-tls\") pod \"maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b\" (UID: \"45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b" Nov 01 15:58:25.747702 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:25.747673 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b" Nov 01 15:58:26.028622 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:26.028590 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b"] Nov 01 15:58:26.031656 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:58:26.031630 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45637fe1_f7c2_4ce9_aee1_79d93fc3d2b2.slice/crio-d9dddc497211791ce9c723c6141ce2a18c76db6158c0604ecb1d600b35ed0b77 WatchSource:0}: Error finding container d9dddc497211791ce9c723c6141ce2a18c76db6158c0604ecb1d600b35ed0b77: Status 404 returned error can't find the container with id d9dddc497211791ce9c723c6141ce2a18c76db6158c0604ecb1d600b35ed0b77 Nov 01 15:58:26.567450 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:26.567395 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b" event={"ID":"45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2","Type":"ContainerStarted","Data":"ba010ce47ece69eb44f07c3e8f73bf3cc29e0d41cc2dc4077f1c5e3e6e31d26d"} Nov 01 15:58:26.567606 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:26.567559 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b" Nov 01 15:58:26.567606 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:26.567577 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b" event={"ID":"45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2","Type":"ContainerStarted","Data":"d9dddc497211791ce9c723c6141ce2a18c76db6158c0604ecb1d600b35ed0b77"} Nov 01 15:58:26.586342 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:26.586297 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b" podStartSLOduration=2.169395826 podStartE2EDuration="2.586283572s" totalImagesPullingTime="416.887746ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:58:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:58:26.585073217 +0000 UTC m=+639.916810864" watchObservedRunningTime="2026-11-01 15:58:26.586283572 +0000 UTC m=+639.918021222" Nov 01 15:58:28.553295 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:28.553263 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-43fac8-ff59b5dfb-r8zjf" Nov 01 15:58:30.513087 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:30.513051 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-43fac8-84b47867b7-4gzrb" Nov 01 15:58:30.513471 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:30.513123 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-4a083d-84c576c9b7-jlscm" Nov 01 15:58:30.567007 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:30.566978 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-164db6-5746fd596c-rmbpv" Nov 01 15:58:32.591236 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:32.591202 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b" Nov 01 15:58:33.553070 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:33.552992 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-dfff59-688dc9df89-4vk2r" Nov 01 15:58:33.553560 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:33.553540 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bd831a-56899656f5-jhvtb" Nov 01 15:58:40.740696 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:40.740660 4363 generic.go:356] "Generic (PLEG): container finished" podID="1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12" containerID="b39a2cceb2e7f80ace969b2f8ef8e83f046c29be469d309a114e4c8f25098b49" exitCode=0 Nov 01 15:58:40.741162 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:40.740734 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" event={"ID":"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12","Type":"ContainerDied","Data":"b39a2cceb2e7f80ace969b2f8ef8e83f046c29be469d309a114e4c8f25098b49"} Nov 01 15:58:40.741222 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:40.741204 4363 scope.go:122] "RemoveContainer" containerID="b39a2cceb2e7f80ace969b2f8ef8e83f046c29be469d309a114e4c8f25098b49" Nov 01 15:58:40.742464 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:40.742441 4363 generic.go:356] "Generic (PLEG): container finished" podID="69f7030b-22ab-4f75-8705-af57a56c29f5" containerID="600118d493d6e661c338eef2b89235dfcee2bc96d4c1467a14e048f0881400f1" exitCode=0 Nov 01 15:58:40.742538 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:40.742472 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" event={"ID":"69f7030b-22ab-4f75-8705-af57a56c29f5","Type":"ContainerDied","Data":"600118d493d6e661c338eef2b89235dfcee2bc96d4c1467a14e048f0881400f1"} Nov 01 15:58:40.742640 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:40.742616 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:58:40.742694 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:40.742681 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:58:40.742732 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:40.742715 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:58:40.742792 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:40.742778 4363 scope.go:122] "RemoveContainer" containerID="600118d493d6e661c338eef2b89235dfcee2bc96d4c1467a14e048f0881400f1" Nov 01 15:58:40.743851 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:40.743820 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:58:40.743937 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:40.743895 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:58:40.743982 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:40.743951 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:58:41.756066 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:41.752221 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" event={"ID":"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12","Type":"ContainerStarted","Data":"7c5e178d9f75b1517a8dd2d804127ff4a9ac088a5337cb37341ca8a3c29446f1"} Nov 01 15:58:41.762885 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:41.762671 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" event={"ID":"69f7030b-22ab-4f75-8705-af57a56c29f5","Type":"ContainerStarted","Data":"2936c71b4370d557c4253153a44fdc51748064e8a380ddce3c305747cba26b50"} Nov 01 15:58:41.875481 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:41.875448 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" Nov 01 15:58:41.876467 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:41.876446 4363 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 01 15:58:41.876550 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:41.876488 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" podUID="1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 01 15:58:42.038121 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:42.037194 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" Nov 01 15:58:42.041175 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:42.041139 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 01 15:58:42.041443 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:42.041414 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" podUID="69f7030b-22ab-4f75-8705-af57a56c29f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 01 15:58:42.886043 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:42.875712 4363 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 01 15:58:42.886043 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:42.875774 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" podUID="1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 01 15:58:43.047297 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:43.047188 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 01 15:58:43.047590 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:43.047521 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" podUID="69f7030b-22ab-4f75-8705-af57a56c29f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 01 15:58:43.876508 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:43.876217 4363 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 01 15:58:43.876508 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:43.876294 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" podUID="1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 01 15:58:44.033048 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:44.032539 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 01 15:58:44.033048 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:44.032615 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" podUID="69f7030b-22ab-4f75-8705-af57a56c29f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 01 15:58:44.939042 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:44.938909 4363 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 01 15:58:44.939042 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:44.938968 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" podUID="1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 01 15:58:45.057503 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:45.057443 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 01 15:58:45.058060 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:45.057522 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" podUID="69f7030b-22ab-4f75-8705-af57a56c29f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 01 15:58:45.876236 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:45.876196 4363 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 01 15:58:45.876454 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:45.876263 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" podUID="1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 01 15:58:46.033454 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:46.033415 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 01 15:58:46.033735 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:46.033706 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" podUID="69f7030b-22ab-4f75-8705-af57a56c29f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 01 15:58:46.464270 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:46.464233 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" Nov 01 15:58:46.464668 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:46.464328 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" Nov 01 15:58:46.875878 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:46.875846 4363 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 01 15:58:46.876090 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:46.875897 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" podUID="1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 01 15:58:47.079124 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:47.079090 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" Nov 01 15:58:47.081048 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:47.080994 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" Nov 01 15:58:47.881085 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:47.881055 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" Nov 01 15:58:47.882305 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:47.882281 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" Nov 01 15:58:48.309779 ip-10-0-128-238 kubenswrapper[4363]: E1101 15:58:48.309696 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 15:58:55.240359 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:55.240315 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6"] Nov 01 15:58:55.245002 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:55.244981 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 15:58:55.254162 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:55.254140 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6"] Nov 01 15:58:55.254778 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:55.254752 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-9f4dc9\"/\"kube-root-ca.crt\"" Nov 01 15:58:55.256009 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:55.255985 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9f4dc9\"/\"default-dockercfg-n49qh\"" Nov 01 15:58:55.256009 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:55.255991 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-9f4dc9\"/\"openshift-service-ca.crt\"" Nov 01 15:58:55.256226 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:55.256091 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9f4dc9\"/\"simulator-w4-9f4dc9-kserve-self-signed-certs\"" Nov 01 15:58:55.341650 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:55.341614 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8261b28f-6536-4ba3-88f6-0713370eb6d3-home\") pod \"simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6\" (UID: \"8261b28f-6536-4ba3-88f6-0713370eb6d3\") " pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 15:58:55.341650 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:55.341656 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f782x\" (UniqueName: \"kubernetes.io/projected/8261b28f-6536-4ba3-88f6-0713370eb6d3-kube-api-access-f782x\") pod \"simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6\" (UID: \"8261b28f-6536-4ba3-88f6-0713370eb6d3\") " pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 15:58:55.341843 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:55.341758 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8261b28f-6536-4ba3-88f6-0713370eb6d3-tls-certs\") pod \"simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6\" (UID: \"8261b28f-6536-4ba3-88f6-0713370eb6d3\") " pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 15:58:55.341843 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:55.341820 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8261b28f-6536-4ba3-88f6-0713370eb6d3-tmp-dir\") pod \"simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6\" (UID: \"8261b28f-6536-4ba3-88f6-0713370eb6d3\") " pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 15:58:55.341843 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:55.341841 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8261b28f-6536-4ba3-88f6-0713370eb6d3-model-cache\") pod \"simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6\" (UID: \"8261b28f-6536-4ba3-88f6-0713370eb6d3\") " pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 15:58:55.341941 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:55.341911 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8261b28f-6536-4ba3-88f6-0713370eb6d3-dshm\") pod \"simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6\" (UID: \"8261b28f-6536-4ba3-88f6-0713370eb6d3\") " pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 15:58:55.443246 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:55.443214 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8261b28f-6536-4ba3-88f6-0713370eb6d3-home\") pod \"simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6\" (UID: \"8261b28f-6536-4ba3-88f6-0713370eb6d3\") " pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 15:58:55.443430 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:55.443252 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f782x\" (UniqueName: \"kubernetes.io/projected/8261b28f-6536-4ba3-88f6-0713370eb6d3-kube-api-access-f782x\") pod \"simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6\" (UID: \"8261b28f-6536-4ba3-88f6-0713370eb6d3\") " pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 15:58:55.443430 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:55.443294 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8261b28f-6536-4ba3-88f6-0713370eb6d3-tls-certs\") pod \"simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6\" (UID: \"8261b28f-6536-4ba3-88f6-0713370eb6d3\") " pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 15:58:55.443430 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:55.443351 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8261b28f-6536-4ba3-88f6-0713370eb6d3-tmp-dir\") pod \"simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6\" (UID: \"8261b28f-6536-4ba3-88f6-0713370eb6d3\") " pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 15:58:55.443430 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:55.443384 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8261b28f-6536-4ba3-88f6-0713370eb6d3-model-cache\") pod \"simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6\" (UID: \"8261b28f-6536-4ba3-88f6-0713370eb6d3\") " pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 15:58:55.443430 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:55.443416 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8261b28f-6536-4ba3-88f6-0713370eb6d3-dshm\") pod \"simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6\" (UID: \"8261b28f-6536-4ba3-88f6-0713370eb6d3\") " pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 15:58:55.443765 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:55.443734 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8261b28f-6536-4ba3-88f6-0713370eb6d3-tmp-dir\") pod \"simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6\" (UID: \"8261b28f-6536-4ba3-88f6-0713370eb6d3\") " pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 15:58:55.443837 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:55.443751 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8261b28f-6536-4ba3-88f6-0713370eb6d3-home\") pod \"simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6\" (UID: \"8261b28f-6536-4ba3-88f6-0713370eb6d3\") " pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 15:58:55.443837 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:55.443816 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8261b28f-6536-4ba3-88f6-0713370eb6d3-model-cache\") pod \"simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6\" (UID: \"8261b28f-6536-4ba3-88f6-0713370eb6d3\") " pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 15:58:55.445654 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:55.445634 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8261b28f-6536-4ba3-88f6-0713370eb6d3-dshm\") pod \"simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6\" (UID: \"8261b28f-6536-4ba3-88f6-0713370eb6d3\") " pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 15:58:55.445902 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:55.445886 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8261b28f-6536-4ba3-88f6-0713370eb6d3-tls-certs\") pod \"simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6\" (UID: \"8261b28f-6536-4ba3-88f6-0713370eb6d3\") " pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 15:58:55.450977 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:55.450955 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f782x\" (UniqueName: \"kubernetes.io/projected/8261b28f-6536-4ba3-88f6-0713370eb6d3-kube-api-access-f782x\") pod \"simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6\" (UID: \"8261b28f-6536-4ba3-88f6-0713370eb6d3\") " pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 15:58:55.566689 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:55.566604 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 15:58:55.839858 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:55.839834 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6"] Nov 01 15:58:55.841829 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:58:55.841798 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8261b28f_6536_4ba3_88f6_0713370eb6d3.slice/crio-4d67939ffbb7519b4b95b9232190a3b0704277350ee140ffc0be824eed6671bd WatchSource:0}: Error finding container 4d67939ffbb7519b4b95b9232190a3b0704277350ee140ffc0be824eed6671bd: Status 404 returned error can't find the container with id 4d67939ffbb7519b4b95b9232190a3b0704277350ee140ffc0be824eed6671bd Nov 01 15:58:56.026102 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:56.026054 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" event={"ID":"8261b28f-6536-4ba3-88f6-0713370eb6d3","Type":"ContainerStarted","Data":"80fad3514371311b623cb629207f77dbfa1c75ae0e3450e815d375c94a31b156"} Nov 01 15:58:56.026295 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:56.026110 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" event={"ID":"8261b28f-6536-4ba3-88f6-0713370eb6d3","Type":"ContainerStarted","Data":"4d67939ffbb7519b4b95b9232190a3b0704277350ee140ffc0be824eed6671bd"} Nov 01 15:58:56.051899 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:56.051845 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podStartSLOduration=1.051830535 podStartE2EDuration="1.051830535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:58:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:58:56.049810574 +0000 UTC m=+669.381548225" watchObservedRunningTime="2026-11-01 15:58:56.051830535 +0000 UTC m=+669.383568185" Nov 01 15:58:56.567491 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:58:56.567459 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 15:59:05.567412 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:05.567308 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 15:59:05.580301 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:05.580273 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 15:59:06.090853 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:06.090822 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 15:59:09.090203 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:09.090169 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx"] Nov 01 15:59:09.093984 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:09.093961 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 15:59:09.100302 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:09.100269 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-164db6\"/\"default-dockercfg-l8c2c\"" Nov 01 15:59:09.101240 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:09.101213 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-164db6\"/\"premium-w5-164db6-kserve-self-signed-certs\"" Nov 01 15:59:09.102765 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:09.102743 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-164db6\"/\"kube-root-ca.crt\"" Nov 01 15:59:09.102765 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:09.102755 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-164db6\"/\"openshift-service-ca.crt\"" Nov 01 15:59:09.110588 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:09.110565 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx"] Nov 01 15:59:09.220362 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:09.220326 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed53e8bc-2f4c-426d-8088-ccaac93847ff-dshm\") pod \"premium-w5-164db6-kserve-68b84d5f96-nbgxx\" (UID: \"ed53e8bc-2f4c-426d-8088-ccaac93847ff\") " pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 15:59:09.220362 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:09.220377 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed53e8bc-2f4c-426d-8088-ccaac93847ff-home\") pod \"premium-w5-164db6-kserve-68b84d5f96-nbgxx\" (UID: \"ed53e8bc-2f4c-426d-8088-ccaac93847ff\") " pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 15:59:09.220598 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:09.220393 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed53e8bc-2f4c-426d-8088-ccaac93847ff-model-cache\") pod \"premium-w5-164db6-kserve-68b84d5f96-nbgxx\" (UID: \"ed53e8bc-2f4c-426d-8088-ccaac93847ff\") " pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 15:59:09.220598 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:09.220537 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed53e8bc-2f4c-426d-8088-ccaac93847ff-tls-certs\") pod \"premium-w5-164db6-kserve-68b84d5f96-nbgxx\" (UID: \"ed53e8bc-2f4c-426d-8088-ccaac93847ff\") " pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 15:59:09.220598 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:09.220591 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed53e8bc-2f4c-426d-8088-ccaac93847ff-tmp-dir\") pod \"premium-w5-164db6-kserve-68b84d5f96-nbgxx\" (UID: \"ed53e8bc-2f4c-426d-8088-ccaac93847ff\") " pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 15:59:09.220709 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:09.220637 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r79xb\" (UniqueName: \"kubernetes.io/projected/ed53e8bc-2f4c-426d-8088-ccaac93847ff-kube-api-access-r79xb\") pod \"premium-w5-164db6-kserve-68b84d5f96-nbgxx\" (UID: \"ed53e8bc-2f4c-426d-8088-ccaac93847ff\") " pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 15:59:09.321638 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:09.321585 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed53e8bc-2f4c-426d-8088-ccaac93847ff-dshm\") pod \"premium-w5-164db6-kserve-68b84d5f96-nbgxx\" (UID: \"ed53e8bc-2f4c-426d-8088-ccaac93847ff\") " pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 15:59:09.321837 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:09.321670 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed53e8bc-2f4c-426d-8088-ccaac93847ff-home\") pod \"premium-w5-164db6-kserve-68b84d5f96-nbgxx\" (UID: \"ed53e8bc-2f4c-426d-8088-ccaac93847ff\") " pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 15:59:09.321837 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:09.321694 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed53e8bc-2f4c-426d-8088-ccaac93847ff-model-cache\") pod \"premium-w5-164db6-kserve-68b84d5f96-nbgxx\" (UID: \"ed53e8bc-2f4c-426d-8088-ccaac93847ff\") " pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 15:59:09.321837 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:09.321746 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed53e8bc-2f4c-426d-8088-ccaac93847ff-tls-certs\") pod \"premium-w5-164db6-kserve-68b84d5f96-nbgxx\" (UID: \"ed53e8bc-2f4c-426d-8088-ccaac93847ff\") " pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 15:59:09.321837 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:09.321793 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed53e8bc-2f4c-426d-8088-ccaac93847ff-tmp-dir\") pod \"premium-w5-164db6-kserve-68b84d5f96-nbgxx\" (UID: \"ed53e8bc-2f4c-426d-8088-ccaac93847ff\") " pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 15:59:09.321837 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:09.321817 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r79xb\" (UniqueName: \"kubernetes.io/projected/ed53e8bc-2f4c-426d-8088-ccaac93847ff-kube-api-access-r79xb\") pod \"premium-w5-164db6-kserve-68b84d5f96-nbgxx\" (UID: \"ed53e8bc-2f4c-426d-8088-ccaac93847ff\") " pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 15:59:09.323043 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:09.322998 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed53e8bc-2f4c-426d-8088-ccaac93847ff-model-cache\") pod \"premium-w5-164db6-kserve-68b84d5f96-nbgxx\" (UID: \"ed53e8bc-2f4c-426d-8088-ccaac93847ff\") " pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 15:59:09.328072 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:09.323455 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed53e8bc-2f4c-426d-8088-ccaac93847ff-home\") pod \"premium-w5-164db6-kserve-68b84d5f96-nbgxx\" (UID: \"ed53e8bc-2f4c-426d-8088-ccaac93847ff\") " pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 15:59:09.328267 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:09.325463 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed53e8bc-2f4c-426d-8088-ccaac93847ff-tmp-dir\") pod \"premium-w5-164db6-kserve-68b84d5f96-nbgxx\" (UID: \"ed53e8bc-2f4c-426d-8088-ccaac93847ff\") " pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 15:59:09.329775 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:09.329748 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed53e8bc-2f4c-426d-8088-ccaac93847ff-tls-certs\") pod \"premium-w5-164db6-kserve-68b84d5f96-nbgxx\" (UID: \"ed53e8bc-2f4c-426d-8088-ccaac93847ff\") " pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 15:59:09.331764 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:09.331743 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r79xb\" (UniqueName: \"kubernetes.io/projected/ed53e8bc-2f4c-426d-8088-ccaac93847ff-kube-api-access-r79xb\") pod \"premium-w5-164db6-kserve-68b84d5f96-nbgxx\" (UID: \"ed53e8bc-2f4c-426d-8088-ccaac93847ff\") " pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 15:59:09.332407 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:09.332386 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed53e8bc-2f4c-426d-8088-ccaac93847ff-dshm\") pod \"premium-w5-164db6-kserve-68b84d5f96-nbgxx\" (UID: \"ed53e8bc-2f4c-426d-8088-ccaac93847ff\") " pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 15:59:09.414581 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:09.414547 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 15:59:09.700424 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:09.700388 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx"] Nov 01 15:59:09.700757 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:59:09.700735 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded53e8bc_2f4c_426d_8088_ccaac93847ff.slice/crio-4791c407044df2b2d1b85a2525f5c28f2a8cd76d7cc3d10613076272a3569fd7 WatchSource:0}: Error finding container 4791c407044df2b2d1b85a2525f5c28f2a8cd76d7cc3d10613076272a3569fd7: Status 404 returned error can't find the container with id 4791c407044df2b2d1b85a2525f5c28f2a8cd76d7cc3d10613076272a3569fd7 Nov 01 15:59:10.105482 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:10.105447 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" event={"ID":"ed53e8bc-2f4c-426d-8088-ccaac93847ff","Type":"ContainerStarted","Data":"6a0250be15169c8396b6a52f815f873c677123510048d991cafa4b288a205fa0"} Nov 01 15:59:10.105959 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:10.105497 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" event={"ID":"ed53e8bc-2f4c-426d-8088-ccaac93847ff","Type":"ContainerStarted","Data":"4791c407044df2b2d1b85a2525f5c28f2a8cd76d7cc3d10613076272a3569fd7"} Nov 01 15:59:10.136644 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:10.136583 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podStartSLOduration=1.136561175 podStartE2EDuration="1.136561175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:59:10.125480809 +0000 UTC m=+683.457218483" watchObservedRunningTime="2026-11-01 15:59:10.136561175 +0000 UTC m=+683.468298828" Nov 01 15:59:10.415046 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:10.414919 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 15:59:18.310480 ip-10-0-128-238 kubenswrapper[4363]: E1101 15:59:18.310438 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 15:59:19.415006 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:19.414968 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 15:59:19.432787 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:19.432762 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 15:59:20.211420 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:20.211384 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 15:59:23.622672 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:23.622638 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr"] Nov 01 15:59:23.628455 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:23.628432 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 15:59:23.636853 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:23.636824 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9f4dc9\"/\"premium-w4-9f4dc9-kserve-self-signed-certs\"" Nov 01 15:59:23.636853 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:23.636842 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr"] Nov 01 15:59:23.712873 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:23.712834 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6476aee-e7a4-497f-84f0-ef0d63ede742-dshm\") pod \"premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr\" (UID: \"a6476aee-e7a4-497f-84f0-ef0d63ede742\") " pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 15:59:23.713074 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:23.712883 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6476aee-e7a4-497f-84f0-ef0d63ede742-tmp-dir\") pod \"premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr\" (UID: \"a6476aee-e7a4-497f-84f0-ef0d63ede742\") " pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 15:59:23.713074 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:23.712975 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6476aee-e7a4-497f-84f0-ef0d63ede742-tls-certs\") pod \"premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr\" (UID: \"a6476aee-e7a4-497f-84f0-ef0d63ede742\") " pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 15:59:23.713074 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:23.713045 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6476aee-e7a4-497f-84f0-ef0d63ede742-model-cache\") pod \"premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr\" (UID: \"a6476aee-e7a4-497f-84f0-ef0d63ede742\") " pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 15:59:23.713200 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:23.713087 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v97s9\" (UniqueName: \"kubernetes.io/projected/a6476aee-e7a4-497f-84f0-ef0d63ede742-kube-api-access-v97s9\") pod \"premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr\" (UID: \"a6476aee-e7a4-497f-84f0-ef0d63ede742\") " pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 15:59:23.713200 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:23.713185 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6476aee-e7a4-497f-84f0-ef0d63ede742-home\") pod \"premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr\" (UID: \"a6476aee-e7a4-497f-84f0-ef0d63ede742\") " pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 15:59:23.813977 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:23.813941 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6476aee-e7a4-497f-84f0-ef0d63ede742-tls-certs\") pod \"premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr\" (UID: \"a6476aee-e7a4-497f-84f0-ef0d63ede742\") " pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 15:59:23.813977 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:23.813980 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6476aee-e7a4-497f-84f0-ef0d63ede742-model-cache\") pod \"premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr\" (UID: \"a6476aee-e7a4-497f-84f0-ef0d63ede742\") " pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 15:59:23.814217 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:23.814045 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v97s9\" (UniqueName: \"kubernetes.io/projected/a6476aee-e7a4-497f-84f0-ef0d63ede742-kube-api-access-v97s9\") pod \"premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr\" (UID: \"a6476aee-e7a4-497f-84f0-ef0d63ede742\") " pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 15:59:23.814217 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:23.814094 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6476aee-e7a4-497f-84f0-ef0d63ede742-home\") pod \"premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr\" (UID: \"a6476aee-e7a4-497f-84f0-ef0d63ede742\") " pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 15:59:23.814217 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:23.814186 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6476aee-e7a4-497f-84f0-ef0d63ede742-dshm\") pod \"premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr\" (UID: \"a6476aee-e7a4-497f-84f0-ef0d63ede742\") " pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 15:59:23.814364 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:23.814227 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6476aee-e7a4-497f-84f0-ef0d63ede742-tmp-dir\") pod \"premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr\" (UID: \"a6476aee-e7a4-497f-84f0-ef0d63ede742\") " pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 15:59:23.814510 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:23.814488 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6476aee-e7a4-497f-84f0-ef0d63ede742-model-cache\") pod \"premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr\" (UID: \"a6476aee-e7a4-497f-84f0-ef0d63ede742\") " pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 15:59:23.814585 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:23.814566 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6476aee-e7a4-497f-84f0-ef0d63ede742-home\") pod \"premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr\" (UID: \"a6476aee-e7a4-497f-84f0-ef0d63ede742\") " pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 15:59:23.814632 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:23.814585 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6476aee-e7a4-497f-84f0-ef0d63ede742-tmp-dir\") pod \"premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr\" (UID: \"a6476aee-e7a4-497f-84f0-ef0d63ede742\") " pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 15:59:23.816536 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:23.816499 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6476aee-e7a4-497f-84f0-ef0d63ede742-dshm\") pod \"premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr\" (UID: \"a6476aee-e7a4-497f-84f0-ef0d63ede742\") " pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 15:59:23.816841 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:23.816821 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6476aee-e7a4-497f-84f0-ef0d63ede742-tls-certs\") pod \"premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr\" (UID: \"a6476aee-e7a4-497f-84f0-ef0d63ede742\") " pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 15:59:23.821881 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:23.821858 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v97s9\" (UniqueName: \"kubernetes.io/projected/a6476aee-e7a4-497f-84f0-ef0d63ede742-kube-api-access-v97s9\") pod \"premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr\" (UID: \"a6476aee-e7a4-497f-84f0-ef0d63ede742\") " pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 15:59:23.942374 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:23.942287 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 15:59:24.223248 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:24.223222 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr"] Nov 01 15:59:24.225317 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:59:24.225292 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6476aee_e7a4_497f_84f0_ef0d63ede742.slice/crio-350ce28ca5f3e8a2d56c7117ef5725810a9a9379f14c93a453f43d8dbff2398b WatchSource:0}: Error finding container 350ce28ca5f3e8a2d56c7117ef5725810a9a9379f14c93a453f43d8dbff2398b: Status 404 returned error can't find the container with id 350ce28ca5f3e8a2d56c7117ef5725810a9a9379f14c93a453f43d8dbff2398b Nov 01 15:59:25.228300 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:25.228263 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" event={"ID":"a6476aee-e7a4-497f-84f0-ef0d63ede742","Type":"ContainerStarted","Data":"8ba72283f27f9372064628a2ecedd172c117b3952b1ffc4144a9a6c2fa1b32ce"} Nov 01 15:59:25.228300 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:25.228299 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" event={"ID":"a6476aee-e7a4-497f-84f0-ef0d63ede742","Type":"ContainerStarted","Data":"350ce28ca5f3e8a2d56c7117ef5725810a9a9379f14c93a453f43d8dbff2398b"} Nov 01 15:59:25.247650 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:25.247579 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podStartSLOduration=2.247564483 podStartE2EDuration="2.247564483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:59:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:59:25.246720926 +0000 UTC m=+698.578458576" watchObservedRunningTime="2026-11-01 15:59:25.247564483 +0000 UTC m=+698.579302133" Nov 01 15:59:25.949047 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:25.947722 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 15:59:28.381050 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.380999 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw"] Nov 01 15:59:28.385203 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.385181 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" Nov 01 15:59:28.397430 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.393961 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-164db6-openshift-default-dockercfg-mt9cn\"" Nov 01 15:59:28.397430 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.396821 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw"] Nov 01 15:59:28.523504 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.523463 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf"] Nov 01 15:59:28.527839 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.527816 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" Nov 01 15:59:28.534096 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.533851 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-4a083d-openshift-default-dockercfg-qxpcr\"" Nov 01 15:59:28.536880 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.536858 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf"] Nov 01 15:59:28.582468 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.582431 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-istio-envoy\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf\" (UID: \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" Nov 01 15:59:28.582769 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.582752 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/674b7965-e091-422f-b128-35635859883f-workload-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw\" (UID: \"674b7965-e091-422f-b128-35635859883f\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" Nov 01 15:59:28.582925 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.582911 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/674b7965-e091-422f-b128-35635859883f-workload-certs\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw\" (UID: \"674b7965-e091-422f-b128-35635859883f\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" Nov 01 15:59:28.583067 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.583053 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/674b7965-e091-422f-b128-35635859883f-istiod-ca-cert\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw\" (UID: \"674b7965-e091-422f-b128-35635859883f\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" Nov 01 15:59:28.583224 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.583210 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnpfs\" (UniqueName: \"kubernetes.io/projected/674b7965-e091-422f-b128-35635859883f-kube-api-access-hnpfs\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw\" (UID: \"674b7965-e091-422f-b128-35635859883f\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" Nov 01 15:59:28.583345 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.583332 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgqwd\" (UniqueName: \"kubernetes.io/projected/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-kube-api-access-xgqwd\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf\" (UID: \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" Nov 01 15:59:28.583470 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.583457 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-istio-data\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf\" (UID: \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" Nov 01 15:59:28.583597 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.583584 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/674b7965-e091-422f-b128-35635859883f-credential-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw\" (UID: \"674b7965-e091-422f-b128-35635859883f\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" Nov 01 15:59:28.583718 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.583706 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/674b7965-e091-422f-b128-35635859883f-istio-data\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw\" (UID: \"674b7965-e091-422f-b128-35635859883f\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" Nov 01 15:59:28.583836 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.583823 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-workload-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf\" (UID: \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" Nov 01 15:59:28.583983 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.583971 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-istiod-ca-cert\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf\" (UID: \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" Nov 01 15:59:28.584125 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.584113 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/674b7965-e091-422f-b128-35635859883f-istio-podinfo\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw\" (UID: \"674b7965-e091-422f-b128-35635859883f\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" Nov 01 15:59:28.584261 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.584248 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-istio-podinfo\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf\" (UID: \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" Nov 01 15:59:28.584412 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.584396 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/674b7965-e091-422f-b128-35635859883f-istio-envoy\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw\" (UID: \"674b7965-e091-422f-b128-35635859883f\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" Nov 01 15:59:28.584590 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.584570 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-workload-certs\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf\" (UID: \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" Nov 01 15:59:28.584752 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.584732 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/674b7965-e091-422f-b128-35635859883f-istio-token\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw\" (UID: \"674b7965-e091-422f-b128-35635859883f\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" Nov 01 15:59:28.584908 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.584802 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-credential-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf\" (UID: \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" Nov 01 15:59:28.584908 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.584870 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-istio-token\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf\" (UID: \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" Nov 01 15:59:28.686666 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.686583 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-credential-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf\" (UID: \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" Nov 01 15:59:28.686666 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.686630 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-istio-token\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf\" (UID: \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" Nov 01 15:59:28.686920 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.686689 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-istio-envoy\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf\" (UID: \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" Nov 01 15:59:28.686920 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.686818 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/674b7965-e091-422f-b128-35635859883f-workload-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw\" (UID: \"674b7965-e091-422f-b128-35635859883f\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" Nov 01 15:59:28.686920 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.686866 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/674b7965-e091-422f-b128-35635859883f-workload-certs\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw\" (UID: \"674b7965-e091-422f-b128-35635859883f\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" Nov 01 15:59:28.686920 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.686897 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/674b7965-e091-422f-b128-35635859883f-istiod-ca-cert\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw\" (UID: \"674b7965-e091-422f-b128-35635859883f\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" Nov 01 15:59:28.687207 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.686947 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnpfs\" (UniqueName: \"kubernetes.io/projected/674b7965-e091-422f-b128-35635859883f-kube-api-access-hnpfs\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw\" (UID: \"674b7965-e091-422f-b128-35635859883f\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" Nov 01 15:59:28.687207 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.686972 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgqwd\" (UniqueName: \"kubernetes.io/projected/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-kube-api-access-xgqwd\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf\" (UID: \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" Nov 01 15:59:28.687207 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.686996 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-istio-data\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf\" (UID: \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" Nov 01 15:59:28.687207 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.687056 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/674b7965-e091-422f-b128-35635859883f-credential-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw\" (UID: \"674b7965-e091-422f-b128-35635859883f\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" Nov 01 15:59:28.687207 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.687090 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-credential-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf\" (UID: \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" Nov 01 15:59:28.687207 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.687094 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/674b7965-e091-422f-b128-35635859883f-istio-data\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw\" (UID: \"674b7965-e091-422f-b128-35635859883f\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" Nov 01 15:59:28.687207 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.687119 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-workload-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf\" (UID: \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" Nov 01 15:59:28.687207 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.687197 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-istiod-ca-cert\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf\" (UID: \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" Nov 01 15:59:28.687639 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.687229 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/674b7965-e091-422f-b128-35635859883f-istio-podinfo\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw\" (UID: \"674b7965-e091-422f-b128-35635859883f\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" Nov 01 15:59:28.687639 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.687262 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-istio-podinfo\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf\" (UID: \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" Nov 01 15:59:28.687639 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.687296 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/674b7965-e091-422f-b128-35635859883f-workload-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw\" (UID: \"674b7965-e091-422f-b128-35635859883f\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" Nov 01 15:59:28.687639 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.687318 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/674b7965-e091-422f-b128-35635859883f-istio-envoy\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw\" (UID: \"674b7965-e091-422f-b128-35635859883f\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" Nov 01 15:59:28.687639 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.687343 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-istio-data\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf\" (UID: \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" Nov 01 15:59:28.687639 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.687354 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-workload-certs\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf\" (UID: \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" Nov 01 15:59:28.687639 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.687389 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/674b7965-e091-422f-b128-35635859883f-istio-token\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw\" (UID: \"674b7965-e091-422f-b128-35635859883f\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" Nov 01 15:59:28.687639 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.687407 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/674b7965-e091-422f-b128-35635859883f-workload-certs\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw\" (UID: \"674b7965-e091-422f-b128-35635859883f\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" Nov 01 15:59:28.687639 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.687544 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-workload-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf\" (UID: \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" Nov 01 15:59:28.688169 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.687675 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/674b7965-e091-422f-b128-35635859883f-istiod-ca-cert\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw\" (UID: \"674b7965-e091-422f-b128-35635859883f\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" Nov 01 15:59:28.688169 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.687740 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/674b7965-e091-422f-b128-35635859883f-credential-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw\" (UID: \"674b7965-e091-422f-b128-35635859883f\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" Nov 01 15:59:28.688169 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.687776 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-workload-certs\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf\" (UID: \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" Nov 01 15:59:28.688169 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.687809 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/674b7965-e091-422f-b128-35635859883f-istio-data\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw\" (UID: \"674b7965-e091-422f-b128-35635859883f\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" Nov 01 15:59:28.688384 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.688283 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-istiod-ca-cert\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf\" (UID: \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" Nov 01 15:59:28.690236 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.690191 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-istio-envoy\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf\" (UID: \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" Nov 01 15:59:28.690357 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.690315 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/674b7965-e091-422f-b128-35635859883f-istio-envoy\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw\" (UID: \"674b7965-e091-422f-b128-35635859883f\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" Nov 01 15:59:28.690618 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.690589 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-istio-podinfo\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf\" (UID: \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" Nov 01 15:59:28.691069 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.691045 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/674b7965-e091-422f-b128-35635859883f-istio-podinfo\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw\" (UID: \"674b7965-e091-422f-b128-35635859883f\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" Nov 01 15:59:28.694890 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.694863 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-istio-token\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf\" (UID: \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" Nov 01 15:59:28.695403 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.695378 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgqwd\" (UniqueName: \"kubernetes.io/projected/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-kube-api-access-xgqwd\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf\" (UID: \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" Nov 01 15:59:28.695490 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.695381 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnpfs\" (UniqueName: \"kubernetes.io/projected/674b7965-e091-422f-b128-35635859883f-kube-api-access-hnpfs\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw\" (UID: \"674b7965-e091-422f-b128-35635859883f\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" Nov 01 15:59:28.695734 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.695713 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/674b7965-e091-422f-b128-35635859883f-istio-token\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw\" (UID: \"674b7965-e091-422f-b128-35635859883f\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" Nov 01 15:59:28.704625 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.704604 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" Nov 01 15:59:28.844937 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.844903 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" Nov 01 15:59:28.987951 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.987918 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw"] Nov 01 15:59:28.990664 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:59:28.990639 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod674b7965_e091_422f_b128_35635859883f.slice/crio-aca1e4213d6cc5c2d45444dfaae7b14625584c8ca1aee41e6503bd49063eafae WatchSource:0}: Error finding container aca1e4213d6cc5c2d45444dfaae7b14625584c8ca1aee41e6503bd49063eafae: Status 404 returned error can't find the container with id aca1e4213d6cc5c2d45444dfaae7b14625584c8ca1aee41e6503bd49063eafae Nov 01 15:59:28.993711 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.993676 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:59:28.993813 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.993757 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:59:28.993813 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:28.993801 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:59:29.133064 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:29.133039 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf"] Nov 01 15:59:29.135881 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:59:29.135838 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e6063b9_04b2_42f5_ac6b_cea0dc65007a.slice/crio-76a1d3d52b46623aa7d39ff0aa07d325d0c237fd888aab0a88f3f7c8f57cc8b9 WatchSource:0}: Error finding container 76a1d3d52b46623aa7d39ff0aa07d325d0c237fd888aab0a88f3f7c8f57cc8b9: Status 404 returned error can't find the container with id 76a1d3d52b46623aa7d39ff0aa07d325d0c237fd888aab0a88f3f7c8f57cc8b9 Nov 01 15:59:29.138988 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:29.138952 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:59:29.139090 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:29.139050 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:59:29.139130 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:29.139091 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:59:29.254482 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:29.254392 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" event={"ID":"674b7965-e091-422f-b128-35635859883f","Type":"ContainerStarted","Data":"bf72c11e6f0a4c21909cb35bc0e5f34e511c723707e46605498c6d3fc98224c0"} Nov 01 15:59:29.254482 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:29.254456 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" event={"ID":"674b7965-e091-422f-b128-35635859883f","Type":"ContainerStarted","Data":"aca1e4213d6cc5c2d45444dfaae7b14625584c8ca1aee41e6503bd49063eafae"} Nov 01 15:59:29.256685 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:29.256653 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" event={"ID":"0e6063b9-04b2-42f5-ac6b-cea0dc65007a","Type":"ContainerStarted","Data":"1fbc16b99a880843de096ac7d24277b648d33dd2c94140dc3dbd165272de4977"} Nov 01 15:59:29.256825 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:29.256713 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" event={"ID":"0e6063b9-04b2-42f5-ac6b-cea0dc65007a","Type":"ContainerStarted","Data":"76a1d3d52b46623aa7d39ff0aa07d325d0c237fd888aab0a88f3f7c8f57cc8b9"} Nov 01 15:59:29.274496 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:29.274422 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" podStartSLOduration=1.274403919 podStartE2EDuration="1.274403919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:59:29.272659853 +0000 UTC m=+702.604397504" watchObservedRunningTime="2026-11-01 15:59:29.274403919 +0000 UTC m=+702.606141570" Nov 01 15:59:29.293295 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:29.293248 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" podStartSLOduration=1.293234383 podStartE2EDuration="1.293234383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:59:29.292203644 +0000 UTC m=+702.623941295" watchObservedRunningTime="2026-11-01 15:59:29.293234383 +0000 UTC m=+702.624972112" Nov 01 15:59:29.709044 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:29.705145 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" Nov 01 15:59:29.845253 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:29.845192 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" Nov 01 15:59:30.709319 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:30.709281 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 01 15:59:30.709949 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:30.709903 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" podUID="674b7965-e091-422f-b128-35635859883f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 01 15:59:30.853754 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:30.853718 4363 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 01 15:59:30.854056 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:30.853992 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" podUID="0e6063b9-04b2-42f5-ac6b-cea0dc65007a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 01 15:59:31.712508 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:31.712470 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 01 15:59:31.713158 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:31.713119 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" podUID="674b7965-e091-422f-b128-35635859883f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 01 15:59:31.845513 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:31.845453 4363 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 01 15:59:31.845719 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:31.845535 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" podUID="0e6063b9-04b2-42f5-ac6b-cea0dc65007a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 01 15:59:32.719062 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:32.716595 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 01 15:59:32.719062 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:32.716652 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" podUID="674b7965-e091-422f-b128-35635859883f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 01 15:59:32.878614 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:32.850356 4363 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 01 15:59:32.878614 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:32.850429 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" podUID="0e6063b9-04b2-42f5-ac6b-cea0dc65007a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 01 15:59:33.727044 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:33.726588 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 01 15:59:33.727044 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:33.726668 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" podUID="674b7965-e091-422f-b128-35635859883f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 01 15:59:33.847424 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:33.847385 4363 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 01 15:59:33.847692 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:33.847662 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" podUID="0e6063b9-04b2-42f5-ac6b-cea0dc65007a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 01 15:59:33.950041 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:33.947064 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 15:59:34.000102 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:33.999423 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 15:59:34.347589 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:34.347564 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 15:59:34.708896 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:34.708781 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 01 15:59:34.709220 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:34.709155 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" podUID="674b7965-e091-422f-b128-35635859883f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 01 15:59:34.845518 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:34.845482 4363 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 01 15:59:34.845974 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:34.845950 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" podUID="0e6063b9-04b2-42f5-ac6b-cea0dc65007a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 01 15:59:35.705292 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:35.705252 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 01 15:59:35.705505 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:35.705321 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" podUID="674b7965-e091-422f-b128-35635859883f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 01 15:59:35.846118 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:35.846078 4363 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 01 15:59:35.846522 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:35.846145 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" podUID="0e6063b9-04b2-42f5-ac6b-cea0dc65007a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 01 15:59:36.954008 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:36.953973 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" Nov 01 15:59:36.954715 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:36.954690 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" Nov 01 15:59:36.956732 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:36.956711 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" Nov 01 15:59:37.253974 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:37.253884 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" Nov 01 15:59:37.254284 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:37.254258 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" Nov 01 15:59:37.255238 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:37.255217 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" Nov 01 15:59:43.187727 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:43.187690 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs"] Nov 01 15:59:43.191832 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:43.191811 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" Nov 01 15:59:43.200953 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:43.200930 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-43fac8-openshift-default-dockercfg-4dq2m\"" Nov 01 15:59:43.209909 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:43.209545 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs"] Nov 01 15:59:43.288664 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:43.288620 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-istio-envoy\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs\" (UID: \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" Nov 01 15:59:43.288839 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:43.288674 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-istio-podinfo\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs\" (UID: \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" Nov 01 15:59:43.288839 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:43.288764 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-istio-data\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs\" (UID: \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" Nov 01 15:59:43.288839 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:43.288827 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-workload-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs\" (UID: \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" Nov 01 15:59:43.289000 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:43.288859 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-istiod-ca-cert\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs\" (UID: \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" Nov 01 15:59:43.289000 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:43.288964 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-credential-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs\" (UID: \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" Nov 01 15:59:43.289000 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:43.288983 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-istio-token\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs\" (UID: \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" Nov 01 15:59:43.289178 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:43.289116 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-workload-certs\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs\" (UID: \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" Nov 01 15:59:43.289178 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:43.289149 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmglj\" (UniqueName: \"kubernetes.io/projected/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-kube-api-access-pmglj\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs\" (UID: \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" Nov 01 15:59:43.389754 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:43.389723 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-istio-podinfo\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs\" (UID: \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" Nov 01 15:59:43.389930 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:43.389772 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-istio-data\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs\" (UID: \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" Nov 01 15:59:43.389930 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:43.389810 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-workload-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs\" (UID: \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" Nov 01 15:59:43.389930 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:43.389829 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-istiod-ca-cert\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs\" (UID: \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" Nov 01 15:59:43.389930 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:43.389880 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-credential-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs\" (UID: \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" Nov 01 15:59:43.389930 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:43.389906 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-istio-token\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs\" (UID: \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" Nov 01 15:59:43.390333 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:43.390303 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-istio-data\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs\" (UID: \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" Nov 01 15:59:43.390333 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:43.390330 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-credential-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs\" (UID: \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" Nov 01 15:59:43.390524 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:43.390349 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-workload-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs\" (UID: \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" Nov 01 15:59:43.390524 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:43.390301 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-workload-certs\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs\" (UID: \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" Nov 01 15:59:43.390524 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:43.390411 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmglj\" (UniqueName: \"kubernetes.io/projected/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-kube-api-access-pmglj\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs\" (UID: \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" Nov 01 15:59:43.390524 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:43.390480 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-istio-envoy\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs\" (UID: \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" Nov 01 15:59:43.390742 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:43.390556 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-workload-certs\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs\" (UID: \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" Nov 01 15:59:43.391303 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:43.391277 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-istiod-ca-cert\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs\" (UID: \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" Nov 01 15:59:43.392781 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:43.392759 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-istio-envoy\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs\" (UID: \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" Nov 01 15:59:43.392892 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:43.392766 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-istio-podinfo\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs\" (UID: \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" Nov 01 15:59:43.397133 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:43.397112 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-istio-token\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs\" (UID: \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" Nov 01 15:59:43.397424 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:43.397405 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmglj\" (UniqueName: \"kubernetes.io/projected/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-kube-api-access-pmglj\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs\" (UID: \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" Nov 01 15:59:43.511556 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:43.511476 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" Nov 01 15:59:43.800554 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:43.800521 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs"] Nov 01 15:59:43.801109 ip-10-0-128-238 kubenswrapper[4363]: W1101 15:59:43.801084 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff3d8641_74e7_496e_8b67_fd5db4a94e8e.slice/crio-719c331401c8aa6489decbea4f9839c08a44409641d2d0d92e528347693ab868 WatchSource:0}: Error finding container 719c331401c8aa6489decbea4f9839c08a44409641d2d0d92e528347693ab868: Status 404 returned error can't find the container with id 719c331401c8aa6489decbea4f9839c08a44409641d2d0d92e528347693ab868 Nov 01 15:59:43.803651 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:43.803617 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:59:43.803747 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:43.803694 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:59:43.803810 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:43.803752 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:59:44.395788 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:44.395749 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" event={"ID":"ff3d8641-74e7-496e-8b67-fd5db4a94e8e","Type":"ContainerStarted","Data":"047032a99321f97e085c9eaf28c06b2be9413fcee7a10ac837a0d72e404b5659"} Nov 01 15:59:44.396232 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:44.395816 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" event={"ID":"ff3d8641-74e7-496e-8b67-fd5db4a94e8e","Type":"ContainerStarted","Data":"719c331401c8aa6489decbea4f9839c08a44409641d2d0d92e528347693ab868"} Nov 01 15:59:44.427333 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:44.427260 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" podStartSLOduration=1.427242892 podStartE2EDuration="1.427242892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:59:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:59:44.425832458 +0000 UTC m=+717.757570131" watchObservedRunningTime="2026-11-01 15:59:44.427242892 +0000 UTC m=+717.758980544" Nov 01 15:59:44.511998 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:44.511967 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" Nov 01 15:59:44.513232 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:44.513208 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 01 15:59:44.513342 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:44.513255 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" podUID="ff3d8641-74e7-496e-8b67-fd5db4a94e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 01 15:59:45.512358 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:45.512317 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 01 15:59:45.512860 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:45.512396 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" podUID="ff3d8641-74e7-496e-8b67-fd5db4a94e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 01 15:59:46.512686 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:46.512649 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 01 15:59:46.513196 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:46.512725 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" podUID="ff3d8641-74e7-496e-8b67-fd5db4a94e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 01 15:59:47.522039 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:47.521460 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 01 15:59:47.522039 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:47.521521 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" podUID="ff3d8641-74e7-496e-8b67-fd5db4a94e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 01 15:59:48.326159 ip-10-0-128-238 kubenswrapper[4363]: E1101 15:59:48.326125 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 15:59:48.512111 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:48.512072 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 01 15:59:48.512292 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:48.512126 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" podUID="ff3d8641-74e7-496e-8b67-fd5db4a94e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 01 15:59:49.511955 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:49.511904 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 01 15:59:49.512432 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:49.511967 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" podUID="ff3d8641-74e7-496e-8b67-fd5db4a94e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 01 15:59:50.511784 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:50.511749 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 01 15:59:50.511992 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:50.511809 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" podUID="ff3d8641-74e7-496e-8b67-fd5db4a94e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 01 15:59:51.793856 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:51.793826 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" Nov 01 15:59:51.794377 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:51.794354 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" Nov 01 15:59:51.795557 ip-10-0-128-238 kubenswrapper[4363]: I1101 15:59:51.795532 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" Nov 01 16:00:00.142254 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:00.142219 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892480-xxx82"] Nov 01 16:00:00.147684 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:00.147661 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892480-xxx82" Nov 01 16:00:00.157505 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:00.157463 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:00:00.158376 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:00.158354 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892480-xxx82"] Nov 01 16:00:00.180187 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:00.180155 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892480-55q9q"] Nov 01 16:00:00.184832 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:00.184808 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892480-55q9q" Nov 01 16:00:00.185390 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:00.185361 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892480-55q9q"] Nov 01 16:00:00.198006 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:00.197976 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15abbe29-c47b-419f-890d-3dd383439df4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892480-xxx82\" (UID: \"15abbe29-c47b-419f-890d-3dd383439df4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892480-xxx82" Nov 01 16:00:00.198235 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:00.198212 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8gwd\" (UniqueName: \"kubernetes.io/projected/15abbe29-c47b-419f-890d-3dd383439df4-kube-api-access-c8gwd\") pod \"maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892480-xxx82\" (UID: \"15abbe29-c47b-419f-890d-3dd383439df4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892480-xxx82" Nov 01 16:00:00.299117 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:00.299079 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15abbe29-c47b-419f-890d-3dd383439df4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892480-xxx82\" (UID: \"15abbe29-c47b-419f-890d-3dd383439df4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892480-xxx82" Nov 01 16:00:00.299117 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:00.299124 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd8e7f50-405f-4af6-ade0-2eeb086d43af-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892480-55q9q\" (UID: \"dd8e7f50-405f-4af6-ade0-2eeb086d43af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892480-55q9q" Nov 01 16:00:00.299382 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:00.299180 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26th2\" (UniqueName: \"kubernetes.io/projected/dd8e7f50-405f-4af6-ade0-2eeb086d43af-kube-api-access-26th2\") pod \"maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892480-55q9q\" (UID: \"dd8e7f50-405f-4af6-ade0-2eeb086d43af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892480-55q9q" Nov 01 16:00:00.299382 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:00.299359 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8gwd\" (UniqueName: \"kubernetes.io/projected/15abbe29-c47b-419f-890d-3dd383439df4-kube-api-access-c8gwd\") pod \"maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892480-xxx82\" (UID: \"15abbe29-c47b-419f-890d-3dd383439df4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892480-xxx82" Nov 01 16:00:00.299812 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:00.299789 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15abbe29-c47b-419f-890d-3dd383439df4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892480-xxx82\" (UID: \"15abbe29-c47b-419f-890d-3dd383439df4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892480-xxx82" Nov 01 16:00:00.307486 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:00.307458 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8gwd\" (UniqueName: \"kubernetes.io/projected/15abbe29-c47b-419f-890d-3dd383439df4-kube-api-access-c8gwd\") pod \"maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892480-xxx82\" (UID: \"15abbe29-c47b-419f-890d-3dd383439df4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892480-xxx82" Nov 01 16:00:00.400197 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:00.400118 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd8e7f50-405f-4af6-ade0-2eeb086d43af-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892480-55q9q\" (UID: \"dd8e7f50-405f-4af6-ade0-2eeb086d43af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892480-55q9q" Nov 01 16:00:00.400197 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:00.400189 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26th2\" (UniqueName: \"kubernetes.io/projected/dd8e7f50-405f-4af6-ade0-2eeb086d43af-kube-api-access-26th2\") pod \"maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892480-55q9q\" (UID: \"dd8e7f50-405f-4af6-ade0-2eeb086d43af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892480-55q9q" Nov 01 16:00:00.400776 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:00.400754 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd8e7f50-405f-4af6-ade0-2eeb086d43af-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892480-55q9q\" (UID: \"dd8e7f50-405f-4af6-ade0-2eeb086d43af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892480-55q9q" Nov 01 16:00:00.408735 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:00.408705 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26th2\" (UniqueName: \"kubernetes.io/projected/dd8e7f50-405f-4af6-ade0-2eeb086d43af-kube-api-access-26th2\") pod \"maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892480-55q9q\" (UID: \"dd8e7f50-405f-4af6-ade0-2eeb086d43af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892480-55q9q" Nov 01 16:00:00.463116 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:00.463086 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892480-xxx82" Nov 01 16:00:00.506121 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:00.506086 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892480-55q9q" Nov 01 16:00:00.750266 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:00.750241 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892480-xxx82"] Nov 01 16:00:00.751923 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:00:00.751900 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15abbe29_c47b_419f_890d_3dd383439df4.slice/crio-866429aaa131904a127255011eb52cb45d8cbfb02a47169a4b03ba67d1b92573 WatchSource:0}: Error finding container 866429aaa131904a127255011eb52cb45d8cbfb02a47169a4b03ba67d1b92573: Status 404 returned error can't find the container with id 866429aaa131904a127255011eb52cb45d8cbfb02a47169a4b03ba67d1b92573 Nov 01 16:00:00.799502 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:00:00.799458 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd8e7f50_405f_4af6_ade0_2eeb086d43af.slice/crio-b43d01f811b532341d53bad494a58f86d5a6f046589628696f43a96b4d193060 WatchSource:0}: Error finding container b43d01f811b532341d53bad494a58f86d5a6f046589628696f43a96b4d193060: Status 404 returned error can't find the container with id b43d01f811b532341d53bad494a58f86d5a6f046589628696f43a96b4d193060 Nov 01 16:00:00.799502 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:00.799471 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892480-55q9q"] Nov 01 16:00:01.541930 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:01.541892 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892480-xxx82" event={"ID":"15abbe29-c47b-419f-890d-3dd383439df4","Type":"ContainerStarted","Data":"866429aaa131904a127255011eb52cb45d8cbfb02a47169a4b03ba67d1b92573"} Nov 01 16:00:01.543070 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:01.543043 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892480-55q9q" event={"ID":"dd8e7f50-405f-4af6-ade0-2eeb086d43af","Type":"ContainerStarted","Data":"b43d01f811b532341d53bad494a58f86d5a6f046589628696f43a96b4d193060"} Nov 01 16:00:04.617485 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:04.616549 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892480-55q9q" event={"ID":"dd8e7f50-405f-4af6-ade0-2eeb086d43af","Type":"ContainerStarted","Data":"99b6f75f3d5282055a7112e9c05471e3bffa1f491eec9aaf6b9ecd471f5221ef"} Nov 01 16:00:04.635664 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:04.635612 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892480-55q9q" podStartSLOduration=1.343253685 podStartE2EDuration="4.635597983s" totalImagesPullingTime="3.292344298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:00:04.634008534 +0000 UTC m=+737.965746186" watchObservedRunningTime="2026-11-01 16:00:04.635597983 +0000 UTC m=+737.967335633" Nov 01 16:00:05.624070 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:05.624028 4363 generic.go:356] "Generic (PLEG): container finished" podID="15abbe29-c47b-419f-890d-3dd383439df4" containerID="8432d7220627347801252e22d72caa8bd18c67c6df8e445b494e0b9ace313f51" exitCode=60 Nov 01 16:00:05.624582 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:05.624117 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892480-xxx82" event={"ID":"15abbe29-c47b-419f-890d-3dd383439df4","Type":"ContainerDied","Data":"8432d7220627347801252e22d72caa8bd18c67c6df8e445b494e0b9ace313f51"} Nov 01 16:00:05.624582 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:05.624429 4363 scope.go:122] "RemoveContainer" containerID="8432d7220627347801252e22d72caa8bd18c67c6df8e445b494e0b9ace313f51" Nov 01 16:00:05.625961 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:05.625935 4363 generic.go:356] "Generic (PLEG): container finished" podID="dd8e7f50-405f-4af6-ade0-2eeb086d43af" containerID="99b6f75f3d5282055a7112e9c05471e3bffa1f491eec9aaf6b9ecd471f5221ef" exitCode=60 Nov 01 16:00:05.626099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:05.626030 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892480-55q9q" event={"ID":"dd8e7f50-405f-4af6-ade0-2eeb086d43af","Type":"ContainerDied","Data":"99b6f75f3d5282055a7112e9c05471e3bffa1f491eec9aaf6b9ecd471f5221ef"} Nov 01 16:00:05.626437 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:05.626409 4363 scope.go:122] "RemoveContainer" containerID="99b6f75f3d5282055a7112e9c05471e3bffa1f491eec9aaf6b9ecd471f5221ef" Nov 01 16:00:06.634479 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:06.634429 4363 generic.go:356] "Generic (PLEG): container finished" podID="15abbe29-c47b-419f-890d-3dd383439df4" containerID="c9a8508308d5de9ece82d019adca1f07a07e72f7f880eec825b74de869d3cc0a" exitCode=60 Nov 01 16:00:06.634968 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:06.634474 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892480-xxx82" event={"ID":"15abbe29-c47b-419f-890d-3dd383439df4","Type":"ContainerDied","Data":"c9a8508308d5de9ece82d019adca1f07a07e72f7f880eec825b74de869d3cc0a"} Nov 01 16:00:06.634968 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:06.634572 4363 scope.go:122] "RemoveContainer" containerID="8432d7220627347801252e22d72caa8bd18c67c6df8e445b494e0b9ace313f51" Nov 01 16:00:06.634968 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:06.634807 4363 scope.go:122] "RemoveContainer" containerID="c9a8508308d5de9ece82d019adca1f07a07e72f7f880eec825b74de869d3cc0a" Nov 01 16:00:06.635146 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:00:06.635080 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892480-xxx82_odh-ai-gateway-infra(15abbe29-c47b-419f-890d-3dd383439df4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892480-xxx82" podUID="15abbe29-c47b-419f-890d-3dd383439df4" Nov 01 16:00:06.637395 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:06.637367 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892480-55q9q" event={"ID":"dd8e7f50-405f-4af6-ade0-2eeb086d43af","Type":"ContainerStarted","Data":"bcc38dc4f049447d1c00731e40b3a17481f641077bb8cd50a50e00ac686040d2"} Nov 01 16:00:07.644093 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:07.644060 4363 scope.go:122] "RemoveContainer" containerID="c9a8508308d5de9ece82d019adca1f07a07e72f7f880eec825b74de869d3cc0a" Nov 01 16:00:07.644572 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:00:07.644376 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892480-xxx82_odh-ai-gateway-infra(15abbe29-c47b-419f-890d-3dd383439df4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892480-xxx82" podUID="15abbe29-c47b-419f-890d-3dd383439df4" Nov 01 16:00:07.645687 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:07.645665 4363 generic.go:356] "Generic (PLEG): container finished" podID="dd8e7f50-405f-4af6-ade0-2eeb086d43af" containerID="bcc38dc4f049447d1c00731e40b3a17481f641077bb8cd50a50e00ac686040d2" exitCode=60 Nov 01 16:00:07.645767 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:07.645736 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892480-55q9q" event={"ID":"dd8e7f50-405f-4af6-ade0-2eeb086d43af","Type":"ContainerDied","Data":"bcc38dc4f049447d1c00731e40b3a17481f641077bb8cd50a50e00ac686040d2"} Nov 01 16:00:07.645832 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:07.645778 4363 scope.go:122] "RemoveContainer" containerID="99b6f75f3d5282055a7112e9c05471e3bffa1f491eec9aaf6b9ecd471f5221ef" Nov 01 16:00:07.646041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:07.646010 4363 scope.go:122] "RemoveContainer" containerID="bcc38dc4f049447d1c00731e40b3a17481f641077bb8cd50a50e00ac686040d2" Nov 01 16:00:07.646304 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:00:07.646285 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892480-55q9q_odh-ai-gateway-infra(dd8e7f50-405f-4af6-ade0-2eeb086d43af)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892480-55q9q" podUID="dd8e7f50-405f-4af6-ade0-2eeb086d43af" Nov 01 16:00:07.933117 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:07.933036 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02433184-qzbf4"] Nov 01 16:00:07.938077 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:07.938059 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02433184-qzbf4" Nov 01 16:00:07.954111 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:07.954078 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02433184-qzbf4"] Nov 01 16:00:07.957503 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:07.957477 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:00:08.052894 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:08.052862 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba7a1d02-36e4-4fc8-9cc6-75606b0427bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02433184-qzbf4\" (UID: \"ba7a1d02-36e4-4fc8-9cc6-75606b0427bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02433184-qzbf4" Nov 01 16:00:08.053100 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:08.052919 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvkkb\" (UniqueName: \"kubernetes.io/projected/ba7a1d02-36e4-4fc8-9cc6-75606b0427bc-kube-api-access-vvkkb\") pod \"maas-api-revoke-sub-02433184-qzbf4\" (UID: \"ba7a1d02-36e4-4fc8-9cc6-75606b0427bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02433184-qzbf4" Nov 01 16:00:08.053161 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:08.053137 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba7a1d02-36e4-4fc8-9cc6-75606b0427bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02433184-qzbf4\" (UID: \"ba7a1d02-36e4-4fc8-9cc6-75606b0427bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02433184-qzbf4" Nov 01 16:00:08.158134 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:08.155151 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba7a1d02-36e4-4fc8-9cc6-75606b0427bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02433184-qzbf4\" (UID: \"ba7a1d02-36e4-4fc8-9cc6-75606b0427bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02433184-qzbf4" Nov 01 16:00:08.158134 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:08.155287 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba7a1d02-36e4-4fc8-9cc6-75606b0427bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02433184-qzbf4\" (UID: \"ba7a1d02-36e4-4fc8-9cc6-75606b0427bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02433184-qzbf4" Nov 01 16:00:08.158134 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:08.155334 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvkkb\" (UniqueName: \"kubernetes.io/projected/ba7a1d02-36e4-4fc8-9cc6-75606b0427bc-kube-api-access-vvkkb\") pod \"maas-api-revoke-sub-02433184-qzbf4\" (UID: \"ba7a1d02-36e4-4fc8-9cc6-75606b0427bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02433184-qzbf4" Nov 01 16:00:08.158134 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:08.156360 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba7a1d02-36e4-4fc8-9cc6-75606b0427bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02433184-qzbf4\" (UID: \"ba7a1d02-36e4-4fc8-9cc6-75606b0427bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02433184-qzbf4" Nov 01 16:00:08.174828 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:08.174770 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba7a1d02-36e4-4fc8-9cc6-75606b0427bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02433184-qzbf4\" (UID: \"ba7a1d02-36e4-4fc8-9cc6-75606b0427bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02433184-qzbf4" Nov 01 16:00:08.178570 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:08.178546 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvkkb\" (UniqueName: \"kubernetes.io/projected/ba7a1d02-36e4-4fc8-9cc6-75606b0427bc-kube-api-access-vvkkb\") pod \"maas-api-revoke-sub-02433184-qzbf4\" (UID: \"ba7a1d02-36e4-4fc8-9cc6-75606b0427bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02433184-qzbf4" Nov 01 16:00:08.250880 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:08.250802 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02433184-qzbf4" Nov 01 16:00:08.534793 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:08.534766 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02433184-qzbf4"] Nov 01 16:00:08.536716 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:00:08.536688 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba7a1d02_36e4_4fc8_9cc6_75606b0427bc.slice/crio-6740b614a795fa7457f594aef4983e55904ac82013a1e039874e0e8998249cfa WatchSource:0}: Error finding container 6740b614a795fa7457f594aef4983e55904ac82013a1e039874e0e8998249cfa: Status 404 returned error can't find the container with id 6740b614a795fa7457f594aef4983e55904ac82013a1e039874e0e8998249cfa Nov 01 16:00:08.653273 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:08.653237 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02433184-qzbf4" event={"ID":"ba7a1d02-36e4-4fc8-9cc6-75606b0427bc","Type":"ContainerStarted","Data":"6740b614a795fa7457f594aef4983e55904ac82013a1e039874e0e8998249cfa"} Nov 01 16:00:08.655064 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:08.655043 4363 scope.go:122] "RemoveContainer" containerID="bcc38dc4f049447d1c00731e40b3a17481f641077bb8cd50a50e00ac686040d2" Nov 01 16:00:08.655335 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:00:08.655316 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892480-55q9q_odh-ai-gateway-infra(dd8e7f50-405f-4af6-ade0-2eeb086d43af)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892480-55q9q" podUID="dd8e7f50-405f-4af6-ade0-2eeb086d43af" Nov 01 16:00:09.661738 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:09.661641 4363 generic.go:356] "Generic (PLEG): container finished" podID="ba7a1d02-36e4-4fc8-9cc6-75606b0427bc" containerID="b62b87e7fcb3db436f1f4cbc2f2f4e0111d76446032e68d0a55af9436b3e684f" exitCode=0 Nov 01 16:00:09.662160 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:09.661726 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02433184-qzbf4" event={"ID":"ba7a1d02-36e4-4fc8-9cc6-75606b0427bc","Type":"ContainerDied","Data":"b62b87e7fcb3db436f1f4cbc2f2f4e0111d76446032e68d0a55af9436b3e684f"} Nov 01 16:00:11.417040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:11.414204 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02433184-qzbf4" Nov 01 16:00:11.464051 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:11.462790 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba7a1d02-36e4-4fc8-9cc6-75606b0427bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba7a1d02-36e4-4fc8-9cc6-75606b0427bc-maas-api-openshift-ca\") pod \"ba7a1d02-36e4-4fc8-9cc6-75606b0427bc\" (UID: \"ba7a1d02-36e4-4fc8-9cc6-75606b0427bc\") " Nov 01 16:00:11.464051 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:11.462943 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba7a1d02-36e4-4fc8-9cc6-75606b0427bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba7a1d02-36e4-4fc8-9cc6-75606b0427bc-maas-api-xks-ca\") pod \"ba7a1d02-36e4-4fc8-9cc6-75606b0427bc\" (UID: \"ba7a1d02-36e4-4fc8-9cc6-75606b0427bc\") " Nov 01 16:00:11.464051 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:11.463857 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba7a1d02-36e4-4fc8-9cc6-75606b0427bc-maas-api-openshift-ca" pod "ba7a1d02-36e4-4fc8-9cc6-75606b0427bc" (UID: "ba7a1d02-36e4-4fc8-9cc6-75606b0427bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:00:11.464051 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:11.463964 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba7a1d02-36e4-4fc8-9cc6-75606b0427bc-kube-api-access-vvkkb\" (UniqueName: \"kubernetes.io/projected/ba7a1d02-36e4-4fc8-9cc6-75606b0427bc-kube-api-access-vvkkb\") pod \"ba7a1d02-36e4-4fc8-9cc6-75606b0427bc\" (UID: \"ba7a1d02-36e4-4fc8-9cc6-75606b0427bc\") " Nov 01 16:00:11.464903 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:11.464829 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba7a1d02-36e4-4fc8-9cc6-75606b0427bc-maas-api-openshift-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:11.511276 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:11.509829 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba7a1d02-36e4-4fc8-9cc6-75606b0427bc-maas-api-xks-ca" pod "ba7a1d02-36e4-4fc8-9cc6-75606b0427bc" (UID: "ba7a1d02-36e4-4fc8-9cc6-75606b0427bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:00:11.575688 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:11.575287 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba7a1d02-36e4-4fc8-9cc6-75606b0427bc-maas-api-xks-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:11.601040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:11.596181 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba7a1d02-36e4-4fc8-9cc6-75606b0427bc-kube-api-access-vvkkb" pod "ba7a1d02-36e4-4fc8-9cc6-75606b0427bc" (UID: "ba7a1d02-36e4-4fc8-9cc6-75606b0427bc"). InnerVolumeSpecName "kube-api-access-vvkkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:00:11.676163 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:11.676132 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvkkb\" (UniqueName: \"kubernetes.io/projected/ba7a1d02-36e4-4fc8-9cc6-75606b0427bc-kube-api-access-vvkkb\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:11.730658 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:11.730627 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02433184-qzbf4" Nov 01 16:00:11.730802 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:11.730669 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02433184-qzbf4" event={"ID":"ba7a1d02-36e4-4fc8-9cc6-75606b0427bc","Type":"ContainerDied","Data":"6740b614a795fa7457f594aef4983e55904ac82013a1e039874e0e8998249cfa"} Nov 01 16:00:11.730802 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:11.730732 4363 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6740b614a795fa7457f594aef4983e55904ac82013a1e039874e0e8998249cfa" Nov 01 16:00:14.095187 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:14.095155 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr"] Nov 01 16:00:14.095994 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:14.095968 4363 cpu_manager.go:414] "containerMap: removing container" podUID="ba7a1d02-36e4-4fc8-9cc6-75606b0427bc" containerName="revoke-keys" Nov 01 16:00:14.096175 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:14.096156 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba7a1d02-36e4-4fc8-9cc6-75606b0427bc" containerName="revoke-keys" Nov 01 16:00:14.096469 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:14.096437 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba7a1d02-36e4-4fc8-9cc6-75606b0427bc" containerName="revoke-keys" Nov 01 16:00:14.100282 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:14.100262 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" Nov 01 16:00:14.103485 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:14.103466 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-bd831a-openshift-default-dockercfg-f9pt7\"" Nov 01 16:00:14.120358 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:14.120336 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr"] Nov 01 16:00:14.223234 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:14.223202 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af073176-cb1d-4eaf-9b6c-43787f6a7227-workload-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr\" (UID: \"af073176-cb1d-4eaf-9b6c-43787f6a7227\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" Nov 01 16:00:14.223415 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:14.223243 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af073176-cb1d-4eaf-9b6c-43787f6a7227-istiod-ca-cert\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr\" (UID: \"af073176-cb1d-4eaf-9b6c-43787f6a7227\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" Nov 01 16:00:14.223415 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:14.223276 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5mv9\" (UniqueName: \"kubernetes.io/projected/af073176-cb1d-4eaf-9b6c-43787f6a7227-kube-api-access-v5mv9\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr\" (UID: \"af073176-cb1d-4eaf-9b6c-43787f6a7227\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" Nov 01 16:00:14.223415 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:14.223393 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af073176-cb1d-4eaf-9b6c-43787f6a7227-istio-envoy\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr\" (UID: \"af073176-cb1d-4eaf-9b6c-43787f6a7227\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" Nov 01 16:00:14.223540 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:14.223526 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af073176-cb1d-4eaf-9b6c-43787f6a7227-istio-token\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr\" (UID: \"af073176-cb1d-4eaf-9b6c-43787f6a7227\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" Nov 01 16:00:14.223576 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:14.223554 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af073176-cb1d-4eaf-9b6c-43787f6a7227-istio-podinfo\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr\" (UID: \"af073176-cb1d-4eaf-9b6c-43787f6a7227\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" Nov 01 16:00:14.223632 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:14.223578 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af073176-cb1d-4eaf-9b6c-43787f6a7227-istio-data\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr\" (UID: \"af073176-cb1d-4eaf-9b6c-43787f6a7227\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" Nov 01 16:00:14.223632 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:14.223603 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af073176-cb1d-4eaf-9b6c-43787f6a7227-credential-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr\" (UID: \"af073176-cb1d-4eaf-9b6c-43787f6a7227\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" Nov 01 16:00:14.223724 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:14.223679 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af073176-cb1d-4eaf-9b6c-43787f6a7227-workload-certs\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr\" (UID: \"af073176-cb1d-4eaf-9b6c-43787f6a7227\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" Nov 01 16:00:14.324903 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:14.324863 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af073176-cb1d-4eaf-9b6c-43787f6a7227-workload-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr\" (UID: \"af073176-cb1d-4eaf-9b6c-43787f6a7227\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" Nov 01 16:00:14.325133 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:14.324918 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af073176-cb1d-4eaf-9b6c-43787f6a7227-istiod-ca-cert\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr\" (UID: \"af073176-cb1d-4eaf-9b6c-43787f6a7227\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" Nov 01 16:00:14.325133 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:14.324965 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5mv9\" (UniqueName: \"kubernetes.io/projected/af073176-cb1d-4eaf-9b6c-43787f6a7227-kube-api-access-v5mv9\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr\" (UID: \"af073176-cb1d-4eaf-9b6c-43787f6a7227\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" Nov 01 16:00:14.325133 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:14.324988 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af073176-cb1d-4eaf-9b6c-43787f6a7227-istio-envoy\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr\" (UID: \"af073176-cb1d-4eaf-9b6c-43787f6a7227\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" Nov 01 16:00:14.325133 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:14.325126 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af073176-cb1d-4eaf-9b6c-43787f6a7227-istio-token\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr\" (UID: \"af073176-cb1d-4eaf-9b6c-43787f6a7227\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" Nov 01 16:00:14.325354 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:14.325159 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af073176-cb1d-4eaf-9b6c-43787f6a7227-istio-podinfo\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr\" (UID: \"af073176-cb1d-4eaf-9b6c-43787f6a7227\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" Nov 01 16:00:14.325354 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:14.325184 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af073176-cb1d-4eaf-9b6c-43787f6a7227-istio-data\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr\" (UID: \"af073176-cb1d-4eaf-9b6c-43787f6a7227\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" Nov 01 16:00:14.325354 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:14.325215 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af073176-cb1d-4eaf-9b6c-43787f6a7227-credential-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr\" (UID: \"af073176-cb1d-4eaf-9b6c-43787f6a7227\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" Nov 01 16:00:14.325354 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:14.325246 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af073176-cb1d-4eaf-9b6c-43787f6a7227-workload-certs\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr\" (UID: \"af073176-cb1d-4eaf-9b6c-43787f6a7227\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" Nov 01 16:00:14.325570 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:14.325345 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af073176-cb1d-4eaf-9b6c-43787f6a7227-workload-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr\" (UID: \"af073176-cb1d-4eaf-9b6c-43787f6a7227\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" Nov 01 16:00:14.325635 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:14.325566 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af073176-cb1d-4eaf-9b6c-43787f6a7227-workload-certs\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr\" (UID: \"af073176-cb1d-4eaf-9b6c-43787f6a7227\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" Nov 01 16:00:14.325635 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:14.325601 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af073176-cb1d-4eaf-9b6c-43787f6a7227-credential-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr\" (UID: \"af073176-cb1d-4eaf-9b6c-43787f6a7227\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" Nov 01 16:00:14.325735 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:14.325717 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af073176-cb1d-4eaf-9b6c-43787f6a7227-istiod-ca-cert\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr\" (UID: \"af073176-cb1d-4eaf-9b6c-43787f6a7227\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" Nov 01 16:00:14.325816 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:14.325792 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af073176-cb1d-4eaf-9b6c-43787f6a7227-istio-data\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr\" (UID: \"af073176-cb1d-4eaf-9b6c-43787f6a7227\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" Nov 01 16:00:14.327636 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:14.327608 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af073176-cb1d-4eaf-9b6c-43787f6a7227-istio-podinfo\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr\" (UID: \"af073176-cb1d-4eaf-9b6c-43787f6a7227\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" Nov 01 16:00:14.327777 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:14.327757 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af073176-cb1d-4eaf-9b6c-43787f6a7227-istio-envoy\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr\" (UID: \"af073176-cb1d-4eaf-9b6c-43787f6a7227\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" Nov 01 16:00:14.335738 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:14.335716 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af073176-cb1d-4eaf-9b6c-43787f6a7227-istio-token\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr\" (UID: \"af073176-cb1d-4eaf-9b6c-43787f6a7227\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" Nov 01 16:00:14.335847 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:14.335747 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5mv9\" (UniqueName: \"kubernetes.io/projected/af073176-cb1d-4eaf-9b6c-43787f6a7227-kube-api-access-v5mv9\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr\" (UID: \"af073176-cb1d-4eaf-9b6c-43787f6a7227\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" Nov 01 16:00:14.415449 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:14.415377 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" Nov 01 16:00:14.751008 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:14.750976 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr"] Nov 01 16:00:14.752676 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:00:14.752646 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf073176_cb1d_4eaf_9b6c_43787f6a7227.slice/crio-a0c01501bd0056ad8e8e03a3cc55857877ed2861cd18040cc55a7076adafe40b WatchSource:0}: Error finding container a0c01501bd0056ad8e8e03a3cc55857877ed2861cd18040cc55a7076adafe40b: Status 404 returned error can't find the container with id a0c01501bd0056ad8e8e03a3cc55857877ed2861cd18040cc55a7076adafe40b Nov 01 16:00:14.755453 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:14.755419 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:00:14.755523 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:14.755507 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:00:14.755570 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:14.755554 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:00:14.772357 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:14.772333 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" event={"ID":"af073176-cb1d-4eaf-9b6c-43787f6a7227","Type":"ContainerStarted","Data":"a0c01501bd0056ad8e8e03a3cc55857877ed2861cd18040cc55a7076adafe40b"} Nov 01 16:00:15.805376 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:15.805331 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" event={"ID":"af073176-cb1d-4eaf-9b6c-43787f6a7227","Type":"ContainerStarted","Data":"caf00e813eeb9ce335df97a58f7d406b7312c819f436d8d93cb7da42bcd6817d"} Nov 01 16:00:16.416620 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:16.416591 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" Nov 01 16:00:16.418301 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:16.418277 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 01 16:00:16.418449 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:16.418320 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" podUID="af073176-cb1d-4eaf-9b6c-43787f6a7227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 01 16:00:17.416232 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:17.416196 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 01 16:00:17.416631 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:17.416256 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" podUID="af073176-cb1d-4eaf-9b6c-43787f6a7227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 01 16:00:17.791105 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:17.790592 4363 scope.go:122] "RemoveContainer" containerID="c9a8508308d5de9ece82d019adca1f07a07e72f7f880eec825b74de869d3cc0a" Nov 01 16:00:18.328753 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:00:18.328715 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:00:18.419045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:18.418497 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 01 16:00:18.419045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:18.418564 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" podUID="af073176-cb1d-4eaf-9b6c-43787f6a7227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 01 16:00:18.829358 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:18.829325 4363 generic.go:356] "Generic (PLEG): container finished" podID="15abbe29-c47b-419f-890d-3dd383439df4" containerID="9a89340ea068eb11f52854721a01ac24cff3a080e72a0645dc779eb246a60ef0" exitCode=60 Nov 01 16:00:18.829530 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:18.829396 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892480-xxx82" event={"ID":"15abbe29-c47b-419f-890d-3dd383439df4","Type":"ContainerDied","Data":"9a89340ea068eb11f52854721a01ac24cff3a080e72a0645dc779eb246a60ef0"} Nov 01 16:00:18.829530 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:18.829467 4363 scope.go:122] "RemoveContainer" containerID="c9a8508308d5de9ece82d019adca1f07a07e72f7f880eec825b74de869d3cc0a" Nov 01 16:00:18.829893 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:18.829873 4363 scope.go:122] "RemoveContainer" containerID="9a89340ea068eb11f52854721a01ac24cff3a080e72a0645dc779eb246a60ef0" Nov 01 16:00:18.830210 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:00:18.830184 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892480-xxx82_odh-ai-gateway-infra(15abbe29-c47b-419f-890d-3dd383439df4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892480-xxx82" podUID="15abbe29-c47b-419f-890d-3dd383439df4" Nov 01 16:00:18.845593 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:18.845550 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" podStartSLOduration=4.845538369 podStartE2EDuration="4.845538369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:00:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:00:15.836382648 +0000 UTC m=+749.168120326" watchObservedRunningTime="2026-11-01 16:00:18.845538369 +0000 UTC m=+752.177276022" Nov 01 16:00:19.415772 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:19.415739 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 01 16:00:19.415949 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:19.415807 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" podUID="af073176-cb1d-4eaf-9b6c-43787f6a7227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 01 16:00:19.794095 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:19.793412 4363 scope.go:122] "RemoveContainer" containerID="bcc38dc4f049447d1c00731e40b3a17481f641077bb8cd50a50e00ac686040d2" Nov 01 16:00:19.854175 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:19.854144 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892480-xxx82"] Nov 01 16:00:19.980753 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:19.978412 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5a65503-66jkl"] Nov 01 16:00:20.046636 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:20.046545 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5a65503-66jkl"] Nov 01 16:00:20.047051 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:20.046686 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a65503-66jkl" Nov 01 16:00:20.053051 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:20.052757 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:00:20.067054 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:20.066186 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ed26f60-cdbf-421e-be76-dc5e4682115f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5a65503-66jkl\" (UID: \"3ed26f60-cdbf-421e-be76-dc5e4682115f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a65503-66jkl" Nov 01 16:00:20.067054 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:20.066391 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ed26f60-cdbf-421e-be76-dc5e4682115f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5a65503-66jkl\" (UID: \"3ed26f60-cdbf-421e-be76-dc5e4682115f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a65503-66jkl" Nov 01 16:00:20.067054 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:20.066419 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbtw5\" (UniqueName: \"kubernetes.io/projected/3ed26f60-cdbf-421e-be76-dc5e4682115f-kube-api-access-cbtw5\") pod \"maas-api-revoke-sub-e5a65503-66jkl\" (UID: \"3ed26f60-cdbf-421e-be76-dc5e4682115f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a65503-66jkl" Nov 01 16:00:20.152411 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:20.152388 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892480-xxx82" Nov 01 16:00:20.166948 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:20.166889 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15abbe29-c47b-419f-890d-3dd383439df4-kube-api-access-c8gwd\" (UniqueName: \"kubernetes.io/projected/15abbe29-c47b-419f-890d-3dd383439df4-kube-api-access-c8gwd\") pod \"15abbe29-c47b-419f-890d-3dd383439df4\" (UID: \"15abbe29-c47b-419f-890d-3dd383439df4\") " Nov 01 16:00:20.167324 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:20.167218 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15abbe29-c47b-419f-890d-3dd383439df4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15abbe29-c47b-419f-890d-3dd383439df4-maas-api-service-ca\") pod \"15abbe29-c47b-419f-890d-3dd383439df4\" (UID: \"15abbe29-c47b-419f-890d-3dd383439df4\") " Nov 01 16:00:20.167863 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:20.167785 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15abbe29-c47b-419f-890d-3dd383439df4-maas-api-service-ca" pod "15abbe29-c47b-419f-890d-3dd383439df4" (UID: "15abbe29-c47b-419f-890d-3dd383439df4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:00:20.168125 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:20.168086 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbtw5\" (UniqueName: \"kubernetes.io/projected/3ed26f60-cdbf-421e-be76-dc5e4682115f-kube-api-access-cbtw5\") pod \"maas-api-revoke-sub-e5a65503-66jkl\" (UID: \"3ed26f60-cdbf-421e-be76-dc5e4682115f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a65503-66jkl" Nov 01 16:00:20.168220 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:20.168170 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ed26f60-cdbf-421e-be76-dc5e4682115f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5a65503-66jkl\" (UID: \"3ed26f60-cdbf-421e-be76-dc5e4682115f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a65503-66jkl" Nov 01 16:00:20.168391 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:20.168370 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ed26f60-cdbf-421e-be76-dc5e4682115f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5a65503-66jkl\" (UID: \"3ed26f60-cdbf-421e-be76-dc5e4682115f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a65503-66jkl" Nov 01 16:00:20.168473 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:20.168434 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15abbe29-c47b-419f-890d-3dd383439df4-maas-api-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:20.169402 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:20.169168 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ed26f60-cdbf-421e-be76-dc5e4682115f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5a65503-66jkl\" (UID: \"3ed26f60-cdbf-421e-be76-dc5e4682115f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a65503-66jkl" Nov 01 16:00:20.171186 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:20.169987 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15abbe29-c47b-419f-890d-3dd383439df4-kube-api-access-c8gwd" pod "15abbe29-c47b-419f-890d-3dd383439df4" (UID: "15abbe29-c47b-419f-890d-3dd383439df4"). InnerVolumeSpecName "kube-api-access-c8gwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:00:20.171562 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:20.171539 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ed26f60-cdbf-421e-be76-dc5e4682115f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5a65503-66jkl\" (UID: \"3ed26f60-cdbf-421e-be76-dc5e4682115f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a65503-66jkl" Nov 01 16:00:20.190330 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:20.179726 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbtw5\" (UniqueName: \"kubernetes.io/projected/3ed26f60-cdbf-421e-be76-dc5e4682115f-kube-api-access-cbtw5\") pod \"maas-api-revoke-sub-e5a65503-66jkl\" (UID: \"3ed26f60-cdbf-421e-be76-dc5e4682115f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a65503-66jkl" Nov 01 16:00:20.269382 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:20.269335 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8gwd\" (UniqueName: \"kubernetes.io/projected/15abbe29-c47b-419f-890d-3dd383439df4-kube-api-access-c8gwd\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:20.368894 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:20.368859 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a65503-66jkl" Nov 01 16:00:20.416991 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:20.416475 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 01 16:00:20.416991 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:20.416536 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" podUID="af073176-cb1d-4eaf-9b6c-43787f6a7227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 01 16:00:20.692623 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:20.692533 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5a65503-66jkl"] Nov 01 16:00:20.697189 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:00:20.697151 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ed26f60_cdbf_421e_be76_dc5e4682115f.slice/crio-ecdaabf0bb1640fe12d123d04f84fbb0c8d3a785f7276e916dbe4375971d8c57 WatchSource:0}: Error finding container ecdaabf0bb1640fe12d123d04f84fbb0c8d3a785f7276e916dbe4375971d8c57: Status 404 returned error can't find the container with id ecdaabf0bb1640fe12d123d04f84fbb0c8d3a785f7276e916dbe4375971d8c57 Nov 01 16:00:20.870561 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:20.870509 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892480-xxx82" event={"ID":"15abbe29-c47b-419f-890d-3dd383439df4","Type":"ContainerDied","Data":"866429aaa131904a127255011eb52cb45d8cbfb02a47169a4b03ba67d1b92573"} Nov 01 16:00:20.874245 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:20.870673 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892480-xxx82" Nov 01 16:00:20.874245 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:20.872185 4363 scope.go:122] "RemoveContainer" containerID="9a89340ea068eb11f52854721a01ac24cff3a080e72a0645dc779eb246a60ef0" Nov 01 16:00:20.876242 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:20.875907 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892480-55q9q" event={"ID":"dd8e7f50-405f-4af6-ade0-2eeb086d43af","Type":"ContainerStarted","Data":"2c4d0cc9fc7bf056c07de3c16ae57f8590ad82074e437185be43c9feea53e16e"} Nov 01 16:00:20.879932 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:20.879827 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a65503-66jkl" event={"ID":"3ed26f60-cdbf-421e-be76-dc5e4682115f","Type":"ContainerStarted","Data":"ecdaabf0bb1640fe12d123d04f84fbb0c8d3a785f7276e916dbe4375971d8c57"} Nov 01 16:00:20.907591 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:20.907562 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892480-xxx82"] Nov 01 16:00:20.912067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:20.911997 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892480-xxx82"] Nov 01 16:00:21.511848 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:21.511819 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" Nov 01 16:00:21.512902 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:21.512881 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" Nov 01 16:00:21.517539 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:21.517516 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" Nov 01 16:00:21.889868 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:21.889836 4363 generic.go:356] "Generic (PLEG): container finished" podID="dd8e7f50-405f-4af6-ade0-2eeb086d43af" containerID="2c4d0cc9fc7bf056c07de3c16ae57f8590ad82074e437185be43c9feea53e16e" exitCode=60 Nov 01 16:00:21.889868 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:21.889915 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892480-55q9q" event={"ID":"dd8e7f50-405f-4af6-ade0-2eeb086d43af","Type":"ContainerDied","Data":"2c4d0cc9fc7bf056c07de3c16ae57f8590ad82074e437185be43c9feea53e16e"} Nov 01 16:00:21.889868 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:21.890012 4363 scope.go:122] "RemoveContainer" containerID="bcc38dc4f049447d1c00731e40b3a17481f641077bb8cd50a50e00ac686040d2" Nov 01 16:00:21.891090 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:21.890468 4363 scope.go:122] "RemoveContainer" containerID="2c4d0cc9fc7bf056c07de3c16ae57f8590ad82074e437185be43c9feea53e16e" Nov 01 16:00:21.891090 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:00:21.890758 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892480-55q9q_odh-ai-gateway-infra(dd8e7f50-405f-4af6-ade0-2eeb086d43af)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892480-55q9q" podUID="dd8e7f50-405f-4af6-ade0-2eeb086d43af" Nov 01 16:00:21.892870 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:21.892847 4363 generic.go:356] "Generic (PLEG): container finished" podID="3ed26f60-cdbf-421e-be76-dc5e4682115f" containerID="5c3dbb67b4561eb4c4cab0ae26bea63f3337df73d3f31fbed4da387cf75b7ba1" exitCode=0 Nov 01 16:00:21.892949 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:21.892901 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a65503-66jkl" event={"ID":"3ed26f60-cdbf-421e-be76-dc5e4682115f","Type":"ContainerDied","Data":"5c3dbb67b4561eb4c4cab0ae26bea63f3337df73d3f31fbed4da387cf75b7ba1"} Nov 01 16:00:21.905593 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:21.905516 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892480-55q9q"] Nov 01 16:00:22.393153 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:22.393119 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2a32fd7-qdlkv"] Nov 01 16:00:22.393652 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:22.393638 4363 cpu_manager.go:414] "containerMap: removing container" podUID="15abbe29-c47b-419f-890d-3dd383439df4" containerName="cleanup" Nov 01 16:00:22.393723 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:22.393653 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15abbe29-c47b-419f-890d-3dd383439df4" containerName="cleanup" Nov 01 16:00:22.393723 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:22.393670 4363 cpu_manager.go:414] "containerMap: removing container" podUID="15abbe29-c47b-419f-890d-3dd383439df4" containerName="cleanup" Nov 01 16:00:22.393723 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:22.393675 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15abbe29-c47b-419f-890d-3dd383439df4" containerName="cleanup" Nov 01 16:00:22.393723 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:22.393686 4363 cpu_manager.go:414] "containerMap: removing container" podUID="15abbe29-c47b-419f-890d-3dd383439df4" containerName="cleanup" Nov 01 16:00:22.393723 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:22.393691 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15abbe29-c47b-419f-890d-3dd383439df4" containerName="cleanup" Nov 01 16:00:22.393978 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:22.393798 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="15abbe29-c47b-419f-890d-3dd383439df4" containerName="cleanup" Nov 01 16:00:22.393978 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:22.393807 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="15abbe29-c47b-419f-890d-3dd383439df4" containerName="cleanup" Nov 01 16:00:22.398327 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:22.398301 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2a32fd7-qdlkv" Nov 01 16:00:22.405194 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:22.405162 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2a32fd7-qdlkv"] Nov 01 16:00:22.544997 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:22.544771 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/596d6128-0721-46bc-8267-ae85b4c9f82a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2a32fd7-qdlkv\" (UID: \"596d6128-0721-46bc-8267-ae85b4c9f82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2a32fd7-qdlkv" Nov 01 16:00:22.544997 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:22.544844 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/596d6128-0721-46bc-8267-ae85b4c9f82a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2a32fd7-qdlkv\" (UID: \"596d6128-0721-46bc-8267-ae85b4c9f82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2a32fd7-qdlkv" Nov 01 16:00:22.545273 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:22.545096 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22cmb\" (UniqueName: \"kubernetes.io/projected/596d6128-0721-46bc-8267-ae85b4c9f82a-kube-api-access-22cmb\") pod \"maas-api-revoke-sub-a2a32fd7-qdlkv\" (UID: \"596d6128-0721-46bc-8267-ae85b4c9f82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2a32fd7-qdlkv" Nov 01 16:00:22.646559 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:22.646472 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22cmb\" (UniqueName: \"kubernetes.io/projected/596d6128-0721-46bc-8267-ae85b4c9f82a-kube-api-access-22cmb\") pod \"maas-api-revoke-sub-a2a32fd7-qdlkv\" (UID: \"596d6128-0721-46bc-8267-ae85b4c9f82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2a32fd7-qdlkv" Nov 01 16:00:22.646559 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:22.646543 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/596d6128-0721-46bc-8267-ae85b4c9f82a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2a32fd7-qdlkv\" (UID: \"596d6128-0721-46bc-8267-ae85b4c9f82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2a32fd7-qdlkv" Nov 01 16:00:22.646791 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:22.646583 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/596d6128-0721-46bc-8267-ae85b4c9f82a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2a32fd7-qdlkv\" (UID: \"596d6128-0721-46bc-8267-ae85b4c9f82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2a32fd7-qdlkv" Nov 01 16:00:22.647345 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:22.647318 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/596d6128-0721-46bc-8267-ae85b4c9f82a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2a32fd7-qdlkv\" (UID: \"596d6128-0721-46bc-8267-ae85b4c9f82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2a32fd7-qdlkv" Nov 01 16:00:22.649118 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:22.649089 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/596d6128-0721-46bc-8267-ae85b4c9f82a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2a32fd7-qdlkv\" (UID: \"596d6128-0721-46bc-8267-ae85b4c9f82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2a32fd7-qdlkv" Nov 01 16:00:22.654185 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:22.654163 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22cmb\" (UniqueName: \"kubernetes.io/projected/596d6128-0721-46bc-8267-ae85b4c9f82a-kube-api-access-22cmb\") pod \"maas-api-revoke-sub-a2a32fd7-qdlkv\" (UID: \"596d6128-0721-46bc-8267-ae85b4c9f82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2a32fd7-qdlkv" Nov 01 16:00:22.724757 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:22.724723 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2a32fd7-qdlkv" Nov 01 16:00:22.803450 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:22.803409 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15abbe29-c47b-419f-890d-3dd383439df4" path="/var/lib/kubelet/pods/15abbe29-c47b-419f-890d-3dd383439df4/volumes" Nov 01 16:00:23.008240 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:23.008205 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2a32fd7-qdlkv"] Nov 01 16:00:23.061940 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:23.061911 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a65503-66jkl" Nov 01 16:00:23.098775 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:23.098754 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892480-55q9q" Nov 01 16:00:23.154818 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:23.154735 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ed26f60-cdbf-421e-be76-dc5e4682115f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ed26f60-cdbf-421e-be76-dc5e4682115f-maas-api-openshift-ca\") pod \"3ed26f60-cdbf-421e-be76-dc5e4682115f\" (UID: \"3ed26f60-cdbf-421e-be76-dc5e4682115f\") " Nov 01 16:00:23.154818 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:23.154804 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ed26f60-cdbf-421e-be76-dc5e4682115f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ed26f60-cdbf-421e-be76-dc5e4682115f-maas-api-xks-ca\") pod \"3ed26f60-cdbf-421e-be76-dc5e4682115f\" (UID: \"3ed26f60-cdbf-421e-be76-dc5e4682115f\") " Nov 01 16:00:23.155052 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:23.154937 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ed26f60-cdbf-421e-be76-dc5e4682115f-kube-api-access-cbtw5\" (UniqueName: \"kubernetes.io/projected/3ed26f60-cdbf-421e-be76-dc5e4682115f-kube-api-access-cbtw5\") pod \"3ed26f60-cdbf-421e-be76-dc5e4682115f\" (UID: \"3ed26f60-cdbf-421e-be76-dc5e4682115f\") " Nov 01 16:00:23.155172 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:23.155144 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ed26f60-cdbf-421e-be76-dc5e4682115f-maas-api-openshift-ca" pod "3ed26f60-cdbf-421e-be76-dc5e4682115f" (UID: "3ed26f60-cdbf-421e-be76-dc5e4682115f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:00:23.155459 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:23.155435 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ed26f60-cdbf-421e-be76-dc5e4682115f-maas-api-openshift-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:23.156830 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:23.156806 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ed26f60-cdbf-421e-be76-dc5e4682115f-maas-api-xks-ca" pod "3ed26f60-cdbf-421e-be76-dc5e4682115f" (UID: "3ed26f60-cdbf-421e-be76-dc5e4682115f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:00:23.157077 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:23.157049 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ed26f60-cdbf-421e-be76-dc5e4682115f-kube-api-access-cbtw5" pod "3ed26f60-cdbf-421e-be76-dc5e4682115f" (UID: "3ed26f60-cdbf-421e-be76-dc5e4682115f"). InnerVolumeSpecName "kube-api-access-cbtw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:00:23.256909 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:23.256869 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd8e7f50-405f-4af6-ade0-2eeb086d43af-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd8e7f50-405f-4af6-ade0-2eeb086d43af-maas-api-service-ca\") pod \"dd8e7f50-405f-4af6-ade0-2eeb086d43af\" (UID: \"dd8e7f50-405f-4af6-ade0-2eeb086d43af\") " Nov 01 16:00:23.257101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:23.256955 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd8e7f50-405f-4af6-ade0-2eeb086d43af-kube-api-access-26th2\" (UniqueName: \"kubernetes.io/projected/dd8e7f50-405f-4af6-ade0-2eeb086d43af-kube-api-access-26th2\") pod \"dd8e7f50-405f-4af6-ade0-2eeb086d43af\" (UID: \"dd8e7f50-405f-4af6-ade0-2eeb086d43af\") " Nov 01 16:00:23.257269 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:23.257246 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd8e7f50-405f-4af6-ade0-2eeb086d43af-maas-api-service-ca" pod "dd8e7f50-405f-4af6-ade0-2eeb086d43af" (UID: "dd8e7f50-405f-4af6-ade0-2eeb086d43af"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:00:23.257311 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:23.257260 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ed26f60-cdbf-421e-be76-dc5e4682115f-maas-api-xks-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:23.257311 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:23.257303 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbtw5\" (UniqueName: \"kubernetes.io/projected/3ed26f60-cdbf-421e-be76-dc5e4682115f-kube-api-access-cbtw5\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:23.259285 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:23.259258 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd8e7f50-405f-4af6-ade0-2eeb086d43af-kube-api-access-26th2" pod "dd8e7f50-405f-4af6-ade0-2eeb086d43af" (UID: "dd8e7f50-405f-4af6-ade0-2eeb086d43af"). InnerVolumeSpecName "kube-api-access-26th2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:00:23.363037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:23.359829 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd8e7f50-405f-4af6-ade0-2eeb086d43af-maas-api-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:23.363037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:23.359867 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26th2\" (UniqueName: \"kubernetes.io/projected/dd8e7f50-405f-4af6-ade0-2eeb086d43af-kube-api-access-26th2\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:23.911802 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:23.911734 4363 generic.go:356] "Generic (PLEG): container finished" podID="596d6128-0721-46bc-8267-ae85b4c9f82a" containerID="88837088c4265258d66c1f2f7290b09f331038498e321d7d5338ad944c842feb" exitCode=0 Nov 01 16:00:23.911950 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:23.911817 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2a32fd7-qdlkv" event={"ID":"596d6128-0721-46bc-8267-ae85b4c9f82a","Type":"ContainerDied","Data":"88837088c4265258d66c1f2f7290b09f331038498e321d7d5338ad944c842feb"} Nov 01 16:00:23.911950 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:23.911914 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2a32fd7-qdlkv" event={"ID":"596d6128-0721-46bc-8267-ae85b4c9f82a","Type":"ContainerStarted","Data":"b9ed7a4aac3d760858d42361bad4a5686a12272e1c11ad7d3a61bbe07a258b9e"} Nov 01 16:00:23.913382 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:23.913360 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892480-55q9q" Nov 01 16:00:23.913382 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:23.913371 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892480-55q9q" event={"ID":"dd8e7f50-405f-4af6-ade0-2eeb086d43af","Type":"ContainerDied","Data":"b43d01f811b532341d53bad494a58f86d5a6f046589628696f43a96b4d193060"} Nov 01 16:00:23.913556 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:23.913423 4363 scope.go:122] "RemoveContainer" containerID="2c4d0cc9fc7bf056c07de3c16ae57f8590ad82074e437185be43c9feea53e16e" Nov 01 16:00:23.919850 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:23.919823 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a65503-66jkl" event={"ID":"3ed26f60-cdbf-421e-be76-dc5e4682115f","Type":"ContainerDied","Data":"ecdaabf0bb1640fe12d123d04f84fbb0c8d3a785f7276e916dbe4375971d8c57"} Nov 01 16:00:23.919974 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:23.919897 4363 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecdaabf0bb1640fe12d123d04f84fbb0c8d3a785f7276e916dbe4375971d8c57" Nov 01 16:00:23.919974 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:23.919957 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5a65503-66jkl" Nov 01 16:00:23.955745 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:23.955719 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892480-55q9q"] Nov 01 16:00:23.959646 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:23.959620 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892480-55q9q"] Nov 01 16:00:24.799707 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:24.799666 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd8e7f50-405f-4af6-ade0-2eeb086d43af" path="/var/lib/kubelet/pods/dd8e7f50-405f-4af6-ade0-2eeb086d43af/volumes" Nov 01 16:00:25.644042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:25.642121 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2a32fd7-qdlkv" Nov 01 16:00:25.826549 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:25.826457 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/596d6128-0721-46bc-8267-ae85b4c9f82a-kube-api-access-22cmb\" (UniqueName: \"kubernetes.io/projected/596d6128-0721-46bc-8267-ae85b4c9f82a-kube-api-access-22cmb\") pod \"596d6128-0721-46bc-8267-ae85b4c9f82a\" (UID: \"596d6128-0721-46bc-8267-ae85b4c9f82a\") " Nov 01 16:00:25.851102 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:25.827215 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/596d6128-0721-46bc-8267-ae85b4c9f82a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/596d6128-0721-46bc-8267-ae85b4c9f82a-maas-api-openshift-ca\") pod \"596d6128-0721-46bc-8267-ae85b4c9f82a\" (UID: \"596d6128-0721-46bc-8267-ae85b4c9f82a\") " Nov 01 16:00:25.851102 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:25.827267 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/596d6128-0721-46bc-8267-ae85b4c9f82a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/596d6128-0721-46bc-8267-ae85b4c9f82a-maas-api-xks-ca\") pod \"596d6128-0721-46bc-8267-ae85b4c9f82a\" (UID: \"596d6128-0721-46bc-8267-ae85b4c9f82a\") " Nov 01 16:00:25.851547 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:25.851515 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/596d6128-0721-46bc-8267-ae85b4c9f82a-maas-api-openshift-ca" pod "596d6128-0721-46bc-8267-ae85b4c9f82a" (UID: "596d6128-0721-46bc-8267-ae85b4c9f82a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:00:25.883191 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:25.883154 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/596d6128-0721-46bc-8267-ae85b4c9f82a-maas-api-xks-ca" pod "596d6128-0721-46bc-8267-ae85b4c9f82a" (UID: "596d6128-0721-46bc-8267-ae85b4c9f82a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:00:25.935040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:25.933886 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/596d6128-0721-46bc-8267-ae85b4c9f82a-maas-api-openshift-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:25.935040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:25.933929 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/596d6128-0721-46bc-8267-ae85b4c9f82a-maas-api-xks-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:26.006270 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:26.006210 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/596d6128-0721-46bc-8267-ae85b4c9f82a-kube-api-access-22cmb" pod "596d6128-0721-46bc-8267-ae85b4c9f82a" (UID: "596d6128-0721-46bc-8267-ae85b4c9f82a"). InnerVolumeSpecName "kube-api-access-22cmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:00:26.027077 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:26.024550 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2a32fd7-qdlkv" event={"ID":"596d6128-0721-46bc-8267-ae85b4c9f82a","Type":"ContainerDied","Data":"b9ed7a4aac3d760858d42361bad4a5686a12272e1c11ad7d3a61bbe07a258b9e"} Nov 01 16:00:26.027077 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:26.024614 4363 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9ed7a4aac3d760858d42361bad4a5686a12272e1c11ad7d3a61bbe07a258b9e" Nov 01 16:00:26.027077 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:26.024681 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2a32fd7-qdlkv" Nov 01 16:00:26.034948 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:26.034919 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22cmb\" (UniqueName: \"kubernetes.io/projected/596d6128-0721-46bc-8267-ae85b4c9f82a-kube-api-access-22cmb\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:26.283082 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:26.280545 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:00:26.388039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:26.380951 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 01 16:00:28.998494 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:28.998459 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv"] Nov 01 16:00:28.999740 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:28.999715 4363 cpu_manager.go:414] "containerMap: removing container" podUID="dd8e7f50-405f-4af6-ade0-2eeb086d43af" containerName="cleanup" Nov 01 16:00:29.020234 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.020205 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd8e7f50-405f-4af6-ade0-2eeb086d43af" containerName="cleanup" Nov 01 16:00:29.020467 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.020449 4363 cpu_manager.go:414] "containerMap: removing container" podUID="dd8e7f50-405f-4af6-ade0-2eeb086d43af" containerName="cleanup" Nov 01 16:00:29.020581 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.020566 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd8e7f50-405f-4af6-ade0-2eeb086d43af" containerName="cleanup" Nov 01 16:00:29.020692 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.020680 4363 cpu_manager.go:414] "containerMap: removing container" podUID="596d6128-0721-46bc-8267-ae85b4c9f82a" containerName="revoke-keys" Nov 01 16:00:29.020775 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.020764 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="596d6128-0721-46bc-8267-ae85b4c9f82a" containerName="revoke-keys" Nov 01 16:00:29.020851 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.020841 4363 cpu_manager.go:414] "containerMap: removing container" podUID="dd8e7f50-405f-4af6-ade0-2eeb086d43af" containerName="cleanup" Nov 01 16:00:29.020932 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.020920 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd8e7f50-405f-4af6-ade0-2eeb086d43af" containerName="cleanup" Nov 01 16:00:29.021052 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.021007 4363 cpu_manager.go:414] "containerMap: removing container" podUID="3ed26f60-cdbf-421e-be76-dc5e4682115f" containerName="revoke-keys" Nov 01 16:00:29.021153 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.021139 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ed26f60-cdbf-421e-be76-dc5e4682115f" containerName="revoke-keys" Nov 01 16:00:29.021573 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.021553 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="15abbe29-c47b-419f-890d-3dd383439df4" containerName="cleanup" Nov 01 16:00:29.021730 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.021716 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd8e7f50-405f-4af6-ade0-2eeb086d43af" containerName="cleanup" Nov 01 16:00:29.021830 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.021819 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd8e7f50-405f-4af6-ade0-2eeb086d43af" containerName="cleanup" Nov 01 16:00:29.021913 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.021903 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="596d6128-0721-46bc-8267-ae85b4c9f82a" containerName="revoke-keys" Nov 01 16:00:29.021997 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.021988 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd8e7f50-405f-4af6-ade0-2eeb086d43af" containerName="cleanup" Nov 01 16:00:29.022109 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.022097 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ed26f60-cdbf-421e-be76-dc5e4682115f" containerName="revoke-keys" Nov 01 16:00:29.078266 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.078236 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv"] Nov 01 16:00:29.078504 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.078479 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" Nov 01 16:00:29.243665 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.243244 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6daeee34-6958-40f5-a360-9a58b3bd51d8-istio-data\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv\" (UID: \"6daeee34-6958-40f5-a360-9a58b3bd51d8\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" Nov 01 16:00:29.243665 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.243346 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6daeee34-6958-40f5-a360-9a58b3bd51d8-istio-envoy\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv\" (UID: \"6daeee34-6958-40f5-a360-9a58b3bd51d8\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" Nov 01 16:00:29.243665 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.243397 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6daeee34-6958-40f5-a360-9a58b3bd51d8-istiod-ca-cert\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv\" (UID: \"6daeee34-6958-40f5-a360-9a58b3bd51d8\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" Nov 01 16:00:29.243665 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.243536 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6daeee34-6958-40f5-a360-9a58b3bd51d8-istio-token\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv\" (UID: \"6daeee34-6958-40f5-a360-9a58b3bd51d8\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" Nov 01 16:00:29.243665 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.243617 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zmnv\" (UniqueName: \"kubernetes.io/projected/6daeee34-6958-40f5-a360-9a58b3bd51d8-kube-api-access-6zmnv\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv\" (UID: \"6daeee34-6958-40f5-a360-9a58b3bd51d8\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" Nov 01 16:00:29.243665 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.243655 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6daeee34-6958-40f5-a360-9a58b3bd51d8-credential-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv\" (UID: \"6daeee34-6958-40f5-a360-9a58b3bd51d8\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" Nov 01 16:00:29.244199 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.243756 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6daeee34-6958-40f5-a360-9a58b3bd51d8-workload-certs\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv\" (UID: \"6daeee34-6958-40f5-a360-9a58b3bd51d8\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" Nov 01 16:00:29.244199 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.243800 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6daeee34-6958-40f5-a360-9a58b3bd51d8-istio-podinfo\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv\" (UID: \"6daeee34-6958-40f5-a360-9a58b3bd51d8\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" Nov 01 16:00:29.244199 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.243874 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6daeee34-6958-40f5-a360-9a58b3bd51d8-workload-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv\" (UID: \"6daeee34-6958-40f5-a360-9a58b3bd51d8\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" Nov 01 16:00:29.345446 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.345420 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6daeee34-6958-40f5-a360-9a58b3bd51d8-istio-data\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv\" (UID: \"6daeee34-6958-40f5-a360-9a58b3bd51d8\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" Nov 01 16:00:29.345645 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.345630 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6daeee34-6958-40f5-a360-9a58b3bd51d8-istio-envoy\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv\" (UID: \"6daeee34-6958-40f5-a360-9a58b3bd51d8\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" Nov 01 16:00:29.345711 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.345702 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6daeee34-6958-40f5-a360-9a58b3bd51d8-istiod-ca-cert\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv\" (UID: \"6daeee34-6958-40f5-a360-9a58b3bd51d8\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" Nov 01 16:00:29.345816 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.345807 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6daeee34-6958-40f5-a360-9a58b3bd51d8-istio-token\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv\" (UID: \"6daeee34-6958-40f5-a360-9a58b3bd51d8\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" Nov 01 16:00:29.345893 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.345884 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zmnv\" (UniqueName: \"kubernetes.io/projected/6daeee34-6958-40f5-a360-9a58b3bd51d8-kube-api-access-6zmnv\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv\" (UID: \"6daeee34-6958-40f5-a360-9a58b3bd51d8\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" Nov 01 16:00:29.345961 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.345953 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6daeee34-6958-40f5-a360-9a58b3bd51d8-credential-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv\" (UID: \"6daeee34-6958-40f5-a360-9a58b3bd51d8\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" Nov 01 16:00:29.346042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.346031 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6daeee34-6958-40f5-a360-9a58b3bd51d8-workload-certs\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv\" (UID: \"6daeee34-6958-40f5-a360-9a58b3bd51d8\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" Nov 01 16:00:29.346121 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.346104 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6daeee34-6958-40f5-a360-9a58b3bd51d8-istio-podinfo\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv\" (UID: \"6daeee34-6958-40f5-a360-9a58b3bd51d8\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" Nov 01 16:00:29.346193 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.346164 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6daeee34-6958-40f5-a360-9a58b3bd51d8-workload-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv\" (UID: \"6daeee34-6958-40f5-a360-9a58b3bd51d8\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" Nov 01 16:00:29.346639 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.346613 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6daeee34-6958-40f5-a360-9a58b3bd51d8-istio-data\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv\" (UID: \"6daeee34-6958-40f5-a360-9a58b3bd51d8\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" Nov 01 16:00:29.346639 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.346614 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6daeee34-6958-40f5-a360-9a58b3bd51d8-workload-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv\" (UID: \"6daeee34-6958-40f5-a360-9a58b3bd51d8\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" Nov 01 16:00:29.346872 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.346778 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6daeee34-6958-40f5-a360-9a58b3bd51d8-istiod-ca-cert\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv\" (UID: \"6daeee34-6958-40f5-a360-9a58b3bd51d8\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" Nov 01 16:00:29.347054 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.347004 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6daeee34-6958-40f5-a360-9a58b3bd51d8-credential-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv\" (UID: \"6daeee34-6958-40f5-a360-9a58b3bd51d8\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" Nov 01 16:00:29.347377 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.347338 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6daeee34-6958-40f5-a360-9a58b3bd51d8-workload-certs\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv\" (UID: \"6daeee34-6958-40f5-a360-9a58b3bd51d8\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" Nov 01 16:00:29.349590 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.349475 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6daeee34-6958-40f5-a360-9a58b3bd51d8-istio-envoy\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv\" (UID: \"6daeee34-6958-40f5-a360-9a58b3bd51d8\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" Nov 01 16:00:29.350318 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.350294 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6daeee34-6958-40f5-a360-9a58b3bd51d8-istio-podinfo\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv\" (UID: \"6daeee34-6958-40f5-a360-9a58b3bd51d8\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" Nov 01 16:00:29.354002 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.353982 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6daeee34-6958-40f5-a360-9a58b3bd51d8-istio-token\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv\" (UID: \"6daeee34-6958-40f5-a360-9a58b3bd51d8\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" Nov 01 16:00:29.355603 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.355577 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zmnv\" (UniqueName: \"kubernetes.io/projected/6daeee34-6958-40f5-a360-9a58b3bd51d8-kube-api-access-6zmnv\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv\" (UID: \"6daeee34-6958-40f5-a360-9a58b3bd51d8\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" Nov 01 16:00:29.455539 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.455503 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" Nov 01 16:00:29.779286 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.779258 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv"] Nov 01 16:00:29.781298 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:00:29.781268 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6daeee34_6958_40f5_a360_9a58b3bd51d8.slice/crio-7afe6c52d476e530d71e7448695e4effa4e5c26b049ee4acc3a484fa27544d96 WatchSource:0}: Error finding container 7afe6c52d476e530d71e7448695e4effa4e5c26b049ee4acc3a484fa27544d96: Status 404 returned error can't find the container with id 7afe6c52d476e530d71e7448695e4effa4e5c26b049ee4acc3a484fa27544d96 Nov 01 16:00:29.784490 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.784454 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:00:29.784633 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.784530 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:00:29.784633 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:29.784567 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:00:30.081615 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:30.081520 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" event={"ID":"6daeee34-6958-40f5-a360-9a58b3bd51d8","Type":"ContainerStarted","Data":"d7ae03be4b5c7e4ffa4ab1db04b0f8bd4db603819eeb69eaa27a23ada687c16d"} Nov 01 16:00:30.081615 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:30.081596 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" event={"ID":"6daeee34-6958-40f5-a360-9a58b3bd51d8","Type":"ContainerStarted","Data":"7afe6c52d476e530d71e7448695e4effa4e5c26b049ee4acc3a484fa27544d96"} Nov 01 16:00:30.100123 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:30.100076 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" podStartSLOduration=2.100061291 podStartE2EDuration="2.100061291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:00:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:00:30.098988898 +0000 UTC m=+763.430726575" watchObservedRunningTime="2026-11-01 16:00:30.100061291 +0000 UTC m=+763.431798957" Nov 01 16:00:30.455729 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:30.455689 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" Nov 01 16:00:30.457118 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:30.457091 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 01 16:00:30.457254 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:30.457137 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" podUID="6daeee34-6958-40f5-a360-9a58b3bd51d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 01 16:00:31.456743 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:31.456704 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 01 16:00:31.457249 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:31.456776 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" podUID="6daeee34-6958-40f5-a360-9a58b3bd51d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 01 16:00:32.456142 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:32.456109 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 01 16:00:32.456325 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:32.456168 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" podUID="6daeee34-6958-40f5-a360-9a58b3bd51d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 01 16:00:33.456865 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:33.456829 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 01 16:00:33.457303 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:33.456893 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" podUID="6daeee34-6958-40f5-a360-9a58b3bd51d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 01 16:00:34.456388 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:34.456356 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 01 16:00:34.456569 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:34.456426 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" podUID="6daeee34-6958-40f5-a360-9a58b3bd51d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 01 16:00:35.460459 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:35.459191 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 01 16:00:35.460459 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:35.459258 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" podUID="6daeee34-6958-40f5-a360-9a58b3bd51d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 01 16:00:36.552105 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:36.552074 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" Nov 01 16:00:36.553122 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:36.553096 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" Nov 01 16:00:36.555299 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:36.555273 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" Nov 01 16:00:43.105042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:43.099113 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:00:44.011896 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:44.011863 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl"] Nov 01 16:00:44.027982 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:44.027952 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl"] Nov 01 16:00:44.028330 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:44.028301 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" Nov 01 16:00:44.035700 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:44.035660 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-6b0e0f-openshift-default-dockercfg-bgk88\"" Nov 01 16:00:44.094759 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:44.094664 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-istio-podinfo\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl\" (UID: \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" Nov 01 16:00:44.095073 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:44.095048 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-istio-data\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl\" (UID: \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" Nov 01 16:00:44.095294 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:44.095273 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8qbv\" (UniqueName: \"kubernetes.io/projected/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-kube-api-access-t8qbv\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl\" (UID: \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" Nov 01 16:00:44.095476 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:44.095459 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-istio-envoy\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl\" (UID: \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" Nov 01 16:00:44.095691 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:44.095673 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-istio-token\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl\" (UID: \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" Nov 01 16:00:44.095927 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:44.095910 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-workload-certs\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl\" (UID: \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" Nov 01 16:00:44.096148 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:44.096130 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-credential-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl\" (UID: \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" Nov 01 16:00:44.096314 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:44.096298 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-workload-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl\" (UID: \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" Nov 01 16:00:44.096469 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:44.096453 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-istiod-ca-cert\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl\" (UID: \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" Nov 01 16:00:44.199604 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:44.197853 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-istio-data\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl\" (UID: \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" Nov 01 16:00:44.199604 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:44.197912 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8qbv\" (UniqueName: \"kubernetes.io/projected/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-kube-api-access-t8qbv\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl\" (UID: \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" Nov 01 16:00:44.199604 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:44.197954 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-istio-envoy\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl\" (UID: \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" Nov 01 16:00:44.199604 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:44.198037 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-istio-token\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl\" (UID: \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" Nov 01 16:00:44.199604 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:44.198139 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-workload-certs\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl\" (UID: \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" Nov 01 16:00:44.199604 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:44.198187 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-credential-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl\" (UID: \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" Nov 01 16:00:44.199604 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:44.198219 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-workload-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl\" (UID: \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" Nov 01 16:00:44.199604 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:44.198251 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-istiod-ca-cert\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl\" (UID: \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" Nov 01 16:00:44.199604 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:44.198292 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-istio-data\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl\" (UID: \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" Nov 01 16:00:44.199604 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:44.198375 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-istio-podinfo\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl\" (UID: \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" Nov 01 16:00:44.199604 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:44.199032 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-istiod-ca-cert\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl\" (UID: \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" Nov 01 16:00:44.199604 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:44.199290 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-workload-certs\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl\" (UID: \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" Nov 01 16:00:44.199604 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:44.199563 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-credential-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl\" (UID: \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" Nov 01 16:00:44.202444 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:44.199787 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-workload-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl\" (UID: \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" Nov 01 16:00:44.204928 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:44.204902 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-istio-envoy\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl\" (UID: \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" Nov 01 16:00:44.205523 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:44.205502 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-istio-podinfo\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl\" (UID: \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" Nov 01 16:00:44.209207 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:44.209165 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-istio-token\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl\" (UID: \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" Nov 01 16:00:44.211489 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:44.211449 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8qbv\" (UniqueName: \"kubernetes.io/projected/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-kube-api-access-t8qbv\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl\" (UID: \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" Nov 01 16:00:44.353223 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:44.353193 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" Nov 01 16:00:44.674810 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:44.674783 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl"] Nov 01 16:00:44.676712 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:00:44.676679 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b28b754_a4f9_44a1_8b8c_bcb7854ac1b0.slice/crio-90c9a3dea0985cade70087ca7126c196f71d27988737162f1d17dc10f4652cc3 WatchSource:0}: Error finding container 90c9a3dea0985cade70087ca7126c196f71d27988737162f1d17dc10f4652cc3: Status 404 returned error can't find the container with id 90c9a3dea0985cade70087ca7126c196f71d27988737162f1d17dc10f4652cc3 Nov 01 16:00:44.679887 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:44.679852 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:00:44.680006 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:44.679925 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:00:44.680006 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:44.679954 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:00:45.413583 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:45.413544 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" event={"ID":"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0","Type":"ContainerStarted","Data":"5caefde19962b8bdfbe8aa21d27864e87a211f1a46aec0201e9fed1c1d85b500"} Nov 01 16:00:45.414397 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:45.413602 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" event={"ID":"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0","Type":"ContainerStarted","Data":"90c9a3dea0985cade70087ca7126c196f71d27988737162f1d17dc10f4652cc3"} Nov 01 16:00:45.447064 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:45.446999 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" podStartSLOduration=2.446986003 podStartE2EDuration="2.446986003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:00:45.443628553 +0000 UTC m=+778.775366226" watchObservedRunningTime="2026-11-01 16:00:45.446986003 +0000 UTC m=+778.778723652" Nov 01 16:00:46.354196 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:46.354165 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" Nov 01 16:00:46.355400 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:46.355370 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 01 16:00:46.355519 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:46.355422 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" podUID="0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 01 16:00:47.353787 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:47.353753 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 01 16:00:47.354192 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:47.353816 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" podUID="0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 01 16:00:48.332773 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:00:48.332733 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:00:48.354609 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:48.354575 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 01 16:00:48.355104 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:48.354625 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" podUID="0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 01 16:00:49.366510 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:49.366470 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 01 16:00:49.367127 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:49.367087 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" podUID="0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 01 16:00:50.354132 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:50.354098 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 01 16:00:50.354316 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:50.354153 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" podUID="0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 01 16:00:51.605395 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:51.605359 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" Nov 01 16:00:51.605868 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:51.605784 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" Nov 01 16:00:51.606812 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:51.606791 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" Nov 01 16:00:52.929817 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:52.929783 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm"] Nov 01 16:00:52.935354 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:52.935330 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:00:52.942071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:52.942043 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-bd831a\"/\"default-dockercfg-bxf27\"" Nov 01 16:00:52.946219 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:52.946192 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm"] Nov 01 16:00:52.947784 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:52.947762 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-bd831a\"/\"kube-root-ca.crt\"" Nov 01 16:00:52.947903 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:52.947780 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-bd831a\"/\"openshift-service-ca.crt\"" Nov 01 16:00:53.042114 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:53.042075 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-bd831a\"/\"distinct-w3-bd831a-kserve-self-signed-certs\"" Nov 01 16:00:53.069549 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:53.069517 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7be529d-6534-4d74-a436-6d02335f9e5f-tmp-dir\") pod \"distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm\" (UID: \"f7be529d-6534-4d74-a436-6d02335f9e5f\") " pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:00:53.069694 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:53.069675 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7be529d-6534-4d74-a436-6d02335f9e5f-home\") pod \"distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm\" (UID: \"f7be529d-6534-4d74-a436-6d02335f9e5f\") " pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:00:53.069755 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:53.069737 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7be529d-6534-4d74-a436-6d02335f9e5f-dshm\") pod \"distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm\" (UID: \"f7be529d-6534-4d74-a436-6d02335f9e5f\") " pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:00:53.069804 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:53.069792 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7be529d-6534-4d74-a436-6d02335f9e5f-tls-certs\") pod \"distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm\" (UID: \"f7be529d-6534-4d74-a436-6d02335f9e5f\") " pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:00:53.069860 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:53.069848 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkdd6\" (UniqueName: \"kubernetes.io/projected/f7be529d-6534-4d74-a436-6d02335f9e5f-kube-api-access-jkdd6\") pod \"distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm\" (UID: \"f7be529d-6534-4d74-a436-6d02335f9e5f\") " pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:00:53.069898 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:53.069873 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7be529d-6534-4d74-a436-6d02335f9e5f-model-cache\") pod \"distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm\" (UID: \"f7be529d-6534-4d74-a436-6d02335f9e5f\") " pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:00:53.171888 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:53.171846 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7be529d-6534-4d74-a436-6d02335f9e5f-tls-certs\") pod \"distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm\" (UID: \"f7be529d-6534-4d74-a436-6d02335f9e5f\") " pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:00:53.172254 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:53.172229 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkdd6\" (UniqueName: \"kubernetes.io/projected/f7be529d-6534-4d74-a436-6d02335f9e5f-kube-api-access-jkdd6\") pod \"distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm\" (UID: \"f7be529d-6534-4d74-a436-6d02335f9e5f\") " pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:00:53.172408 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:53.172389 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7be529d-6534-4d74-a436-6d02335f9e5f-model-cache\") pod \"distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm\" (UID: \"f7be529d-6534-4d74-a436-6d02335f9e5f\") " pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:00:53.172622 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:53.172607 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7be529d-6534-4d74-a436-6d02335f9e5f-tmp-dir\") pod \"distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm\" (UID: \"f7be529d-6534-4d74-a436-6d02335f9e5f\") " pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:00:53.172831 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:53.172818 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7be529d-6534-4d74-a436-6d02335f9e5f-home\") pod \"distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm\" (UID: \"f7be529d-6534-4d74-a436-6d02335f9e5f\") " pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:00:53.172998 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:53.172983 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7be529d-6534-4d74-a436-6d02335f9e5f-dshm\") pod \"distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm\" (UID: \"f7be529d-6534-4d74-a436-6d02335f9e5f\") " pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:00:53.178038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:53.174269 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7be529d-6534-4d74-a436-6d02335f9e5f-model-cache\") pod \"distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm\" (UID: \"f7be529d-6534-4d74-a436-6d02335f9e5f\") " pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:00:53.214083 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:53.178466 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7be529d-6534-4d74-a436-6d02335f9e5f-tmp-dir\") pod \"distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm\" (UID: \"f7be529d-6534-4d74-a436-6d02335f9e5f\") " pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:00:53.214083 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:53.179004 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7be529d-6534-4d74-a436-6d02335f9e5f-home\") pod \"distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm\" (UID: \"f7be529d-6534-4d74-a436-6d02335f9e5f\") " pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:00:53.214083 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:53.191624 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7be529d-6534-4d74-a436-6d02335f9e5f-tls-certs\") pod \"distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm\" (UID: \"f7be529d-6534-4d74-a436-6d02335f9e5f\") " pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:00:53.214083 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:53.205570 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7be529d-6534-4d74-a436-6d02335f9e5f-dshm\") pod \"distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm\" (UID: \"f7be529d-6534-4d74-a436-6d02335f9e5f\") " pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:00:53.214083 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:53.207671 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkdd6\" (UniqueName: \"kubernetes.io/projected/f7be529d-6534-4d74-a436-6d02335f9e5f-kube-api-access-jkdd6\") pod \"distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm\" (UID: \"f7be529d-6534-4d74-a436-6d02335f9e5f\") " pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:00:53.251036 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:53.250994 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:00:53.536073 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:53.536045 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm"] Nov 01 16:00:53.536972 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:00:53.536952 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7be529d_6534_4d74_a436_6d02335f9e5f.slice/crio-557105b00d4c771cbf6dcfadc8369b30dff23d201968cbe4a0a69211fe51bcb6 WatchSource:0}: Error finding container 557105b00d4c771cbf6dcfadc8369b30dff23d201968cbe4a0a69211fe51bcb6: Status 404 returned error can't find the container with id 557105b00d4c771cbf6dcfadc8369b30dff23d201968cbe4a0a69211fe51bcb6 Nov 01 16:00:53.547869 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:53.547832 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" event={"ID":"f7be529d-6534-4d74-a436-6d02335f9e5f","Type":"ContainerStarted","Data":"557105b00d4c771cbf6dcfadc8369b30dff23d201968cbe4a0a69211fe51bcb6"} Nov 01 16:00:54.554792 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:54.554757 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" event={"ID":"f7be529d-6534-4d74-a436-6d02335f9e5f","Type":"ContainerStarted","Data":"70aa55c1cb35798a69558f65ca80eff605edd2047accc60f77fccfec2c48da27"} Nov 01 16:00:54.572617 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:54.572569 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podStartSLOduration=2.572556255 podStartE2EDuration="2.572556255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:00:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:00:54.569559532 +0000 UTC m=+787.901297206" watchObservedRunningTime="2026-11-01 16:00:54.572556255 +0000 UTC m=+787.904293904" Nov 01 16:00:55.251713 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:55.251682 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:00:58.832040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:58.828351 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws"] Nov 01 16:00:58.870041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:58.863503 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws"] Nov 01 16:00:58.870041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:58.863637 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" Nov 01 16:00:59.019115 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.017565 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp5lm\" (UniqueName: \"kubernetes.io/projected/07c0b57d-ed0c-4366-9716-256714694541-kube-api-access-rp5lm\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws\" (UID: \"07c0b57d-ed0c-4366-9716-256714694541\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" Nov 01 16:00:59.019115 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.017616 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07c0b57d-ed0c-4366-9716-256714694541-istio-data\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws\" (UID: \"07c0b57d-ed0c-4366-9716-256714694541\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" Nov 01 16:00:59.019115 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.017666 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07c0b57d-ed0c-4366-9716-256714694541-workload-certs\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws\" (UID: \"07c0b57d-ed0c-4366-9716-256714694541\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" Nov 01 16:00:59.019115 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.017703 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07c0b57d-ed0c-4366-9716-256714694541-istiod-ca-cert\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws\" (UID: \"07c0b57d-ed0c-4366-9716-256714694541\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" Nov 01 16:00:59.019115 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.017730 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07c0b57d-ed0c-4366-9716-256714694541-workload-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws\" (UID: \"07c0b57d-ed0c-4366-9716-256714694541\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" Nov 01 16:00:59.019115 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.017756 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07c0b57d-ed0c-4366-9716-256714694541-istio-podinfo\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws\" (UID: \"07c0b57d-ed0c-4366-9716-256714694541\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" Nov 01 16:00:59.019115 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.017868 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07c0b57d-ed0c-4366-9716-256714694541-istio-envoy\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws\" (UID: \"07c0b57d-ed0c-4366-9716-256714694541\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" Nov 01 16:00:59.019115 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.017907 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07c0b57d-ed0c-4366-9716-256714694541-istio-token\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws\" (UID: \"07c0b57d-ed0c-4366-9716-256714694541\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" Nov 01 16:00:59.019115 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.018090 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07c0b57d-ed0c-4366-9716-256714694541-credential-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws\" (UID: \"07c0b57d-ed0c-4366-9716-256714694541\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" Nov 01 16:00:59.150450 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.150412 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07c0b57d-ed0c-4366-9716-256714694541-credential-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws\" (UID: \"07c0b57d-ed0c-4366-9716-256714694541\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" Nov 01 16:00:59.150730 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.150702 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp5lm\" (UniqueName: \"kubernetes.io/projected/07c0b57d-ed0c-4366-9716-256714694541-kube-api-access-rp5lm\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws\" (UID: \"07c0b57d-ed0c-4366-9716-256714694541\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" Nov 01 16:00:59.188067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.180194 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07c0b57d-ed0c-4366-9716-256714694541-istio-data\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws\" (UID: \"07c0b57d-ed0c-4366-9716-256714694541\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" Nov 01 16:00:59.188067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.180268 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07c0b57d-ed0c-4366-9716-256714694541-workload-certs\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws\" (UID: \"07c0b57d-ed0c-4366-9716-256714694541\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" Nov 01 16:00:59.188067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.180306 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07c0b57d-ed0c-4366-9716-256714694541-istiod-ca-cert\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws\" (UID: \"07c0b57d-ed0c-4366-9716-256714694541\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" Nov 01 16:00:59.188067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.180337 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07c0b57d-ed0c-4366-9716-256714694541-workload-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws\" (UID: \"07c0b57d-ed0c-4366-9716-256714694541\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" Nov 01 16:00:59.188067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.180360 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07c0b57d-ed0c-4366-9716-256714694541-istio-podinfo\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws\" (UID: \"07c0b57d-ed0c-4366-9716-256714694541\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" Nov 01 16:00:59.188067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.180498 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07c0b57d-ed0c-4366-9716-256714694541-istio-envoy\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws\" (UID: \"07c0b57d-ed0c-4366-9716-256714694541\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" Nov 01 16:00:59.188067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.180542 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07c0b57d-ed0c-4366-9716-256714694541-istio-token\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws\" (UID: \"07c0b57d-ed0c-4366-9716-256714694541\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" Nov 01 16:00:59.188067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.161910 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07c0b57d-ed0c-4366-9716-256714694541-credential-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws\" (UID: \"07c0b57d-ed0c-4366-9716-256714694541\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" Nov 01 16:00:59.188067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.181421 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07c0b57d-ed0c-4366-9716-256714694541-istio-data\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws\" (UID: \"07c0b57d-ed0c-4366-9716-256714694541\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" Nov 01 16:00:59.188067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.185350 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07c0b57d-ed0c-4366-9716-256714694541-workload-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws\" (UID: \"07c0b57d-ed0c-4366-9716-256714694541\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" Nov 01 16:00:59.188067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.185618 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07c0b57d-ed0c-4366-9716-256714694541-workload-certs\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws\" (UID: \"07c0b57d-ed0c-4366-9716-256714694541\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" Nov 01 16:00:59.188067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.186274 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07c0b57d-ed0c-4366-9716-256714694541-istiod-ca-cert\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws\" (UID: \"07c0b57d-ed0c-4366-9716-256714694541\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" Nov 01 16:00:59.246898 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.246816 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07c0b57d-ed0c-4366-9716-256714694541-istio-podinfo\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws\" (UID: \"07c0b57d-ed0c-4366-9716-256714694541\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" Nov 01 16:00:59.262397 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.262362 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07c0b57d-ed0c-4366-9716-256714694541-istio-envoy\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws\" (UID: \"07c0b57d-ed0c-4366-9716-256714694541\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" Nov 01 16:00:59.300331 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.270583 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6"] Nov 01 16:00:59.315051 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.301705 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07c0b57d-ed0c-4366-9716-256714694541-istio-token\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws\" (UID: \"07c0b57d-ed0c-4366-9716-256714694541\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" Nov 01 16:00:59.343359 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.343328 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" Nov 01 16:00:59.394036 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.387455 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6"] Nov 01 16:00:59.534043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.531283 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp5lm\" (UniqueName: \"kubernetes.io/projected/07c0b57d-ed0c-4366-9716-256714694541-kube-api-access-rp5lm\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws\" (UID: \"07c0b57d-ed0c-4366-9716-256714694541\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" Nov 01 16:00:59.534043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.533002 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/241f70eb-0584-4e76-9616-152a84654bde-istio-podinfo\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6\" (UID: \"241f70eb-0584-4e76-9616-152a84654bde\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" Nov 01 16:00:59.534043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.533145 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8vln\" (UniqueName: \"kubernetes.io/projected/241f70eb-0584-4e76-9616-152a84654bde-kube-api-access-v8vln\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6\" (UID: \"241f70eb-0584-4e76-9616-152a84654bde\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" Nov 01 16:00:59.534043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.533181 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/241f70eb-0584-4e76-9616-152a84654bde-workload-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6\" (UID: \"241f70eb-0584-4e76-9616-152a84654bde\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" Nov 01 16:00:59.534043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.533266 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/241f70eb-0584-4e76-9616-152a84654bde-istio-envoy\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6\" (UID: \"241f70eb-0584-4e76-9616-152a84654bde\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" Nov 01 16:00:59.534043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.533347 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/241f70eb-0584-4e76-9616-152a84654bde-workload-certs\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6\" (UID: \"241f70eb-0584-4e76-9616-152a84654bde\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" Nov 01 16:00:59.534043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.533373 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/241f70eb-0584-4e76-9616-152a84654bde-istiod-ca-cert\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6\" (UID: \"241f70eb-0584-4e76-9616-152a84654bde\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" Nov 01 16:00:59.534043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.533414 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/241f70eb-0584-4e76-9616-152a84654bde-istio-data\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6\" (UID: \"241f70eb-0584-4e76-9616-152a84654bde\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" Nov 01 16:00:59.534043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.533443 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/241f70eb-0584-4e76-9616-152a84654bde-credential-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6\" (UID: \"241f70eb-0584-4e76-9616-152a84654bde\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" Nov 01 16:00:59.534043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.533476 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/241f70eb-0584-4e76-9616-152a84654bde-istio-token\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6\" (UID: \"241f70eb-0584-4e76-9616-152a84654bde\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" Nov 01 16:00:59.556383 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.542962 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" Nov 01 16:00:59.670089 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.638321 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8vln\" (UniqueName: \"kubernetes.io/projected/241f70eb-0584-4e76-9616-152a84654bde-kube-api-access-v8vln\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6\" (UID: \"241f70eb-0584-4e76-9616-152a84654bde\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" Nov 01 16:00:59.670396 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.670363 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/241f70eb-0584-4e76-9616-152a84654bde-workload-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6\" (UID: \"241f70eb-0584-4e76-9616-152a84654bde\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" Nov 01 16:00:59.670646 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.670627 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/241f70eb-0584-4e76-9616-152a84654bde-istio-envoy\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6\" (UID: \"241f70eb-0584-4e76-9616-152a84654bde\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" Nov 01 16:00:59.670844 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.670827 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/241f70eb-0584-4e76-9616-152a84654bde-workload-certs\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6\" (UID: \"241f70eb-0584-4e76-9616-152a84654bde\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" Nov 01 16:00:59.670954 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.670938 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/241f70eb-0584-4e76-9616-152a84654bde-istiod-ca-cert\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6\" (UID: \"241f70eb-0584-4e76-9616-152a84654bde\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" Nov 01 16:00:59.671139 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.671119 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/241f70eb-0584-4e76-9616-152a84654bde-istio-data\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6\" (UID: \"241f70eb-0584-4e76-9616-152a84654bde\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" Nov 01 16:00:59.671265 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.671246 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/241f70eb-0584-4e76-9616-152a84654bde-credential-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6\" (UID: \"241f70eb-0584-4e76-9616-152a84654bde\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" Nov 01 16:00:59.671392 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.671376 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/241f70eb-0584-4e76-9616-152a84654bde-istio-token\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6\" (UID: \"241f70eb-0584-4e76-9616-152a84654bde\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" Nov 01 16:00:59.671535 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.671520 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/241f70eb-0584-4e76-9616-152a84654bde-istio-podinfo\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6\" (UID: \"241f70eb-0584-4e76-9616-152a84654bde\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" Nov 01 16:00:59.680270 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.677606 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/241f70eb-0584-4e76-9616-152a84654bde-istiod-ca-cert\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6\" (UID: \"241f70eb-0584-4e76-9616-152a84654bde\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" Nov 01 16:00:59.693086 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.677873 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/241f70eb-0584-4e76-9616-152a84654bde-workload-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6\" (UID: \"241f70eb-0584-4e76-9616-152a84654bde\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" Nov 01 16:00:59.693253 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.678329 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/241f70eb-0584-4e76-9616-152a84654bde-workload-certs\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6\" (UID: \"241f70eb-0584-4e76-9616-152a84654bde\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" Nov 01 16:00:59.693352 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.678554 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/241f70eb-0584-4e76-9616-152a84654bde-credential-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6\" (UID: \"241f70eb-0584-4e76-9616-152a84654bde\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" Nov 01 16:00:59.693439 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.678778 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/241f70eb-0584-4e76-9616-152a84654bde-istio-data\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6\" (UID: \"241f70eb-0584-4e76-9616-152a84654bde\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" Nov 01 16:00:59.856875 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.847685 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/241f70eb-0584-4e76-9616-152a84654bde-istio-podinfo\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6\" (UID: \"241f70eb-0584-4e76-9616-152a84654bde\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" Nov 01 16:00:59.856875 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.848223 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/241f70eb-0584-4e76-9616-152a84654bde-istio-token\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6\" (UID: \"241f70eb-0584-4e76-9616-152a84654bde\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" Nov 01 16:00:59.856875 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.848826 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8vln\" (UniqueName: \"kubernetes.io/projected/241f70eb-0584-4e76-9616-152a84654bde-kube-api-access-v8vln\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6\" (UID: \"241f70eb-0584-4e76-9616-152a84654bde\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" Nov 01 16:00:59.856875 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.849033 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/241f70eb-0584-4e76-9616-152a84654bde-istio-envoy\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6\" (UID: \"241f70eb-0584-4e76-9616-152a84654bde\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" Nov 01 16:00:59.858067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:00:59.857813 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" Nov 01 16:01:00.291067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:00.291011 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws"] Nov 01 16:01:00.295422 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:00.295392 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6"] Nov 01 16:01:00.298376 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:00.298342 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:00.298493 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:00.298413 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:00.298493 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:00.298441 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:00.298493 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:00.298431 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:00.298645 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:00.298504 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:00.298645 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:00.298549 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:00.913079 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:00.912888 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" event={"ID":"241f70eb-0584-4e76-9616-152a84654bde","Type":"ContainerStarted","Data":"b2911525b548edd8ce427ad4371687a1e03dab1155706e6b512f000fdd921854"} Nov 01 16:01:00.913079 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:00.912951 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" event={"ID":"241f70eb-0584-4e76-9616-152a84654bde","Type":"ContainerStarted","Data":"cbc11ec904a709780a0d421efd11fd91c0dd08c36fc918a3ccdddb6bb905b4d1"} Nov 01 16:01:00.915096 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:00.915069 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" event={"ID":"07c0b57d-ed0c-4366-9716-256714694541","Type":"ContainerStarted","Data":"af74ce07a3b12fa506a685bf3df5118f906d5acf180c7cfbf31991680d62a347"} Nov 01 16:01:00.915224 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:00.915115 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" event={"ID":"07c0b57d-ed0c-4366-9716-256714694541","Type":"ContainerStarted","Data":"ae9c7a9b0d5754646b8b8a22d90b32535dc3b2d0fe287dfe48136a603c066038"} Nov 01 16:01:00.932002 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:00.931957 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" podStartSLOduration=1.931944322 podStartE2EDuration="1.931944322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:00:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:01:00.931355377 +0000 UTC m=+794.263093028" watchObservedRunningTime="2026-11-01 16:01:00.931944322 +0000 UTC m=+794.263681973" Nov 01 16:01:00.956387 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:00.956346 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" podStartSLOduration=2.956333136 podStartE2EDuration="2.956333136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:00:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:01:00.955482703 +0000 UTC m=+794.287220367" watchObservedRunningTime="2026-11-01 16:01:00.956333136 +0000 UTC m=+794.288070777" Nov 01 16:01:01.557378 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:01.557065 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" Nov 01 16:01:01.558757 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:01.558727 4363 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 01 16:01:01.558864 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:01.558780 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" podUID="07c0b57d-ed0c-4366-9716-256714694541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 01 16:01:01.860415 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:01.860381 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" Nov 01 16:01:01.861662 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:01.861632 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 01 16:01:01.861796 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:01.861682 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" podUID="241f70eb-0584-4e76-9616-152a84654bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 01 16:01:02.547001 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:02.546963 4363 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 01 16:01:02.547780 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:02.547748 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" podUID="07c0b57d-ed0c-4366-9716-256714694541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 01 16:01:02.860664 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:02.860627 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 01 16:01:02.860836 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:02.860678 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" podUID="241f70eb-0584-4e76-9616-152a84654bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 01 16:01:03.272099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:03.258086 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:01:03.381042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:03.377072 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:01:03.547273 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:03.547197 4363 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 01 16:01:03.548181 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:03.548125 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" podUID="07c0b57d-ed0c-4366-9716-256714694541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 01 16:01:03.860869 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:03.860836 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 01 16:01:03.861075 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:03.860910 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" podUID="241f70eb-0584-4e76-9616-152a84654bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 01 16:01:04.151504 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:04.151397 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:01:04.546896 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:04.546805 4363 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 01 16:01:04.546896 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:04.546857 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" podUID="07c0b57d-ed0c-4366-9716-256714694541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 01 16:01:04.860708 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:04.860671 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 01 16:01:04.861128 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:04.860726 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" podUID="241f70eb-0584-4e76-9616-152a84654bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 01 16:01:05.547557 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:05.547504 4363 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 01 16:01:05.547739 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:05.547568 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" podUID="07c0b57d-ed0c-4366-9716-256714694541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 01 16:01:05.860707 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:05.860669 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 01 16:01:05.860885 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:05.860727 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" podUID="241f70eb-0584-4e76-9616-152a84654bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 01 16:01:06.546973 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:06.546936 4363 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 01 16:01:06.547199 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:06.546996 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" podUID="07c0b57d-ed0c-4366-9716-256714694541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 01 16:01:06.860522 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:06.860492 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 01 16:01:06.860760 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:06.860735 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" podUID="241f70eb-0584-4e76-9616-152a84654bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 01 16:01:07.866678 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:07.866104 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" Nov 01 16:01:07.866678 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:07.866553 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" Nov 01 16:01:07.867462 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:07.867426 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" Nov 01 16:01:07.933079 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:07.929357 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" Nov 01 16:01:07.933079 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:07.930861 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" Nov 01 16:01:07.935384 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:07.935324 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" Nov 01 16:01:10.039279 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:10.039239 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t"] Nov 01 16:01:10.055042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:10.054431 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:01:10.073747 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:10.073719 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4a083d\"/\"distinct-w0-4a083d-kserve-self-signed-certs\"" Nov 01 16:01:10.073930 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:10.073803 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-4a083d\"/\"openshift-service-ca.crt\"" Nov 01 16:01:10.073930 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:10.073845 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4a083d\"/\"default-dockercfg-lhz8h\"" Nov 01 16:01:10.073930 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:10.073903 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-4a083d\"/\"kube-root-ca.crt\"" Nov 01 16:01:10.075127 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:10.075097 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t"] Nov 01 16:01:10.123621 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:10.123587 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0049d3d4-16f8-4b77-87fa-dacf0921ad20-home\") pod \"distinct-w0-4a083d-kserve-785988c485-2z65t\" (UID: \"0049d3d4-16f8-4b77-87fa-dacf0921ad20\") " pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:01:10.123752 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:10.123661 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0049d3d4-16f8-4b77-87fa-dacf0921ad20-tls-certs\") pod \"distinct-w0-4a083d-kserve-785988c485-2z65t\" (UID: \"0049d3d4-16f8-4b77-87fa-dacf0921ad20\") " pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:01:10.123798 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:10.123766 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0049d3d4-16f8-4b77-87fa-dacf0921ad20-tmp-dir\") pod \"distinct-w0-4a083d-kserve-785988c485-2z65t\" (UID: \"0049d3d4-16f8-4b77-87fa-dacf0921ad20\") " pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:01:10.123833 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:10.123807 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6j9q\" (UniqueName: \"kubernetes.io/projected/0049d3d4-16f8-4b77-87fa-dacf0921ad20-kube-api-access-p6j9q\") pod \"distinct-w0-4a083d-kserve-785988c485-2z65t\" (UID: \"0049d3d4-16f8-4b77-87fa-dacf0921ad20\") " pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:01:10.123944 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:10.123920 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0049d3d4-16f8-4b77-87fa-dacf0921ad20-dshm\") pod \"distinct-w0-4a083d-kserve-785988c485-2z65t\" (UID: \"0049d3d4-16f8-4b77-87fa-dacf0921ad20\") " pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:01:10.124044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:10.124027 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0049d3d4-16f8-4b77-87fa-dacf0921ad20-model-cache\") pod \"distinct-w0-4a083d-kserve-785988c485-2z65t\" (UID: \"0049d3d4-16f8-4b77-87fa-dacf0921ad20\") " pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:01:10.228931 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:10.228887 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0049d3d4-16f8-4b77-87fa-dacf0921ad20-tmp-dir\") pod \"distinct-w0-4a083d-kserve-785988c485-2z65t\" (UID: \"0049d3d4-16f8-4b77-87fa-dacf0921ad20\") " pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:01:10.229738 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:10.229713 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0049d3d4-16f8-4b77-87fa-dacf0921ad20-tmp-dir\") pod \"distinct-w0-4a083d-kserve-785988c485-2z65t\" (UID: \"0049d3d4-16f8-4b77-87fa-dacf0921ad20\") " pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:01:10.229922 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:10.229903 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6j9q\" (UniqueName: \"kubernetes.io/projected/0049d3d4-16f8-4b77-87fa-dacf0921ad20-kube-api-access-p6j9q\") pod \"distinct-w0-4a083d-kserve-785988c485-2z65t\" (UID: \"0049d3d4-16f8-4b77-87fa-dacf0921ad20\") " pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:01:10.230433 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:10.230412 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0049d3d4-16f8-4b77-87fa-dacf0921ad20-dshm\") pod \"distinct-w0-4a083d-kserve-785988c485-2z65t\" (UID: \"0049d3d4-16f8-4b77-87fa-dacf0921ad20\") " pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:01:10.230635 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:10.230607 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0049d3d4-16f8-4b77-87fa-dacf0921ad20-model-cache\") pod \"distinct-w0-4a083d-kserve-785988c485-2z65t\" (UID: \"0049d3d4-16f8-4b77-87fa-dacf0921ad20\") " pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:01:10.230900 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:10.230886 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0049d3d4-16f8-4b77-87fa-dacf0921ad20-home\") pod \"distinct-w0-4a083d-kserve-785988c485-2z65t\" (UID: \"0049d3d4-16f8-4b77-87fa-dacf0921ad20\") " pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:01:10.231091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:10.231076 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0049d3d4-16f8-4b77-87fa-dacf0921ad20-tls-certs\") pod \"distinct-w0-4a083d-kserve-785988c485-2z65t\" (UID: \"0049d3d4-16f8-4b77-87fa-dacf0921ad20\") " pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:01:10.232625 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:10.232593 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0049d3d4-16f8-4b77-87fa-dacf0921ad20-home\") pod \"distinct-w0-4a083d-kserve-785988c485-2z65t\" (UID: \"0049d3d4-16f8-4b77-87fa-dacf0921ad20\") " pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:01:10.233399 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:10.233378 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0049d3d4-16f8-4b77-87fa-dacf0921ad20-model-cache\") pod \"distinct-w0-4a083d-kserve-785988c485-2z65t\" (UID: \"0049d3d4-16f8-4b77-87fa-dacf0921ad20\") " pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:01:10.250042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:10.246219 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0049d3d4-16f8-4b77-87fa-dacf0921ad20-dshm\") pod \"distinct-w0-4a083d-kserve-785988c485-2z65t\" (UID: \"0049d3d4-16f8-4b77-87fa-dacf0921ad20\") " pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:01:10.250042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:10.248754 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6j9q\" (UniqueName: \"kubernetes.io/projected/0049d3d4-16f8-4b77-87fa-dacf0921ad20-kube-api-access-p6j9q\") pod \"distinct-w0-4a083d-kserve-785988c485-2z65t\" (UID: \"0049d3d4-16f8-4b77-87fa-dacf0921ad20\") " pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:01:10.250042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:10.249098 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0049d3d4-16f8-4b77-87fa-dacf0921ad20-tls-certs\") pod \"distinct-w0-4a083d-kserve-785988c485-2z65t\" (UID: \"0049d3d4-16f8-4b77-87fa-dacf0921ad20\") " pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:01:10.386721 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:10.386688 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:01:10.671550 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:10.671520 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t"] Nov 01 16:01:10.672173 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:01:10.672142 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0049d3d4_16f8_4b77_87fa_dacf0921ad20.slice/crio-e3722284e77368ee55ec735b3e3537005497281011e353a1dec6ce5d59f9b3af WatchSource:0}: Error finding container e3722284e77368ee55ec735b3e3537005497281011e353a1dec6ce5d59f9b3af: Status 404 returned error can't find the container with id e3722284e77368ee55ec735b3e3537005497281011e353a1dec6ce5d59f9b3af Nov 01 16:01:11.084177 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:11.084083 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" event={"ID":"0049d3d4-16f8-4b77-87fa-dacf0921ad20","Type":"ContainerStarted","Data":"2df13e314347272901587d10fdc1ea1131d787ea6a562cd60bbfd562130826ea"} Nov 01 16:01:11.084177 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:11.084136 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" event={"ID":"0049d3d4-16f8-4b77-87fa-dacf0921ad20","Type":"ContainerStarted","Data":"e3722284e77368ee55ec735b3e3537005497281011e353a1dec6ce5d59f9b3af"} Nov 01 16:01:11.101144 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:11.101097 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podStartSLOduration=1.101082998 podStartE2EDuration="1.101082998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:01:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:01:11.099924531 +0000 UTC m=+804.431662204" watchObservedRunningTime="2026-11-01 16:01:11.101082998 +0000 UTC m=+804.432820648" Nov 01 16:01:11.391882 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:11.389114 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:01:13.536331 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.536296 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk"] Nov 01 16:01:13.544929 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.544899 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" Nov 01 16:01:13.558591 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.558560 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk"] Nov 01 16:01:13.583473 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.583442 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql"] Nov 01 16:01:13.592311 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.592290 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:01:13.612548 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.612498 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-43fac8\"/\"kube-root-ca.crt\"" Nov 01 16:01:13.612783 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.612547 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-43fac8\"/\"distinct-2-w6-43fac8-kserve-self-signed-certs\"" Nov 01 16:01:13.612783 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.612612 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-43fac8\"/\"openshift-service-ca.crt\"" Nov 01 16:01:13.612973 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.612722 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-43fac8\"/\"default-dockercfg-z2zvx\"" Nov 01 16:01:13.617401 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.617374 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql"] Nov 01 16:01:13.685003 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.684974 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k"] Nov 01 16:01:13.689150 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.689126 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" Nov 01 16:01:13.693993 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.693964 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4-tmp-dir\") pod \"distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql\" (UID: \"e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4\") " pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:01:13.694149 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.694095 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-workload-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk\" (UID: \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" Nov 01 16:01:13.694149 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.694136 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-istio-data\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk\" (UID: \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" Nov 01 16:01:13.694272 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.694160 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmdkv\" (UniqueName: \"kubernetes.io/projected/e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4-kube-api-access-kmdkv\") pod \"distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql\" (UID: \"e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4\") " pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:01:13.694272 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.694223 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-credential-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk\" (UID: \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" Nov 01 16:01:13.694272 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.694262 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4-model-cache\") pod \"distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql\" (UID: \"e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4\") " pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:01:13.694433 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.694303 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-istiod-ca-cert\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk\" (UID: \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" Nov 01 16:01:13.694433 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.694326 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4-home\") pod \"distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql\" (UID: \"e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4\") " pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:01:13.694433 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.694387 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-istio-token\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk\" (UID: \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" Nov 01 16:01:13.694433 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.694409 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4-dshm\") pod \"distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql\" (UID: \"e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4\") " pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:01:13.694703 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.694469 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4-tls-certs\") pod \"distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql\" (UID: \"e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4\") " pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:01:13.694703 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.694519 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-istio-envoy\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk\" (UID: \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" Nov 01 16:01:13.694703 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.694542 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l55gc\" (UniqueName: \"kubernetes.io/projected/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-kube-api-access-l55gc\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk\" (UID: \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" Nov 01 16:01:13.694703 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.694629 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-workload-certs\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk\" (UID: \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" Nov 01 16:01:13.694703 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.694668 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-istio-podinfo\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk\" (UID: \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" Nov 01 16:01:13.747039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.737952 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k"] Nov 01 16:01:13.897059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.875565 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-istio-token\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k\" (UID: \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" Nov 01 16:01:13.897059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.875649 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4-tmp-dir\") pod \"distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql\" (UID: \"e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4\") " pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:01:13.897059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.875705 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-workload-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk\" (UID: \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" Nov 01 16:01:13.897059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.875730 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-istio-data\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk\" (UID: \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" Nov 01 16:01:13.897059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.875755 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmdkv\" (UniqueName: \"kubernetes.io/projected/e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4-kube-api-access-kmdkv\") pod \"distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql\" (UID: \"e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4\") " pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:01:13.897059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.875781 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-workload-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k\" (UID: \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" Nov 01 16:01:13.897059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.875821 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-istio-envoy\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k\" (UID: \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" Nov 01 16:01:13.897059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.875848 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-istio-podinfo\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k\" (UID: \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" Nov 01 16:01:13.897059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.875893 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-credential-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk\" (UID: \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" Nov 01 16:01:13.897059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.875930 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4-model-cache\") pod \"distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql\" (UID: \"e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4\") " pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:01:13.897059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.875968 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-istiod-ca-cert\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk\" (UID: \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" Nov 01 16:01:13.897059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.875995 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4-home\") pod \"distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql\" (UID: \"e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4\") " pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:01:13.897059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.876092 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-istio-token\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk\" (UID: \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" Nov 01 16:01:13.897059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.876117 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4-dshm\") pod \"distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql\" (UID: \"e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4\") " pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:01:13.897059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.876163 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-istiod-ca-cert\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k\" (UID: \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" Nov 01 16:01:13.897059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.876217 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4-tls-certs\") pod \"distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql\" (UID: \"e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4\") " pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:01:13.897059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.876246 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-credential-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k\" (UID: \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" Nov 01 16:01:13.897059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.876326 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-istio-envoy\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk\" (UID: \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" Nov 01 16:01:13.897059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.876352 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l55gc\" (UniqueName: \"kubernetes.io/projected/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-kube-api-access-l55gc\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk\" (UID: \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" Nov 01 16:01:13.897059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.876377 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-workload-certs\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k\" (UID: \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" Nov 01 16:01:13.897059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.876445 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg29p\" (UniqueName: \"kubernetes.io/projected/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-kube-api-access-xg29p\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k\" (UID: \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" Nov 01 16:01:13.897059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.876506 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-workload-certs\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk\" (UID: \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" Nov 01 16:01:13.897059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.876540 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-istio-data\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k\" (UID: \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" Nov 01 16:01:13.897059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.876583 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-istio-podinfo\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk\" (UID: \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" Nov 01 16:01:13.897059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.893598 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4-home\") pod \"distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql\" (UID: \"e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4\") " pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:01:13.897059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.894237 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4-tmp-dir\") pod \"distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql\" (UID: \"e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4\") " pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:01:13.897059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.894551 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-workload-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk\" (UID: \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" Nov 01 16:01:13.897059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.894784 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-istio-data\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk\" (UID: \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" Nov 01 16:01:13.897059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.895378 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-credential-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk\" (UID: \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" Nov 01 16:01:13.897059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.895649 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4-model-cache\") pod \"distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql\" (UID: \"e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4\") " pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:01:13.897059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.896271 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-istiod-ca-cert\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk\" (UID: \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" Nov 01 16:01:13.909823 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.906566 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-workload-certs\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk\" (UID: \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" Nov 01 16:01:13.909823 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.906931 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmdkv\" (UniqueName: \"kubernetes.io/projected/e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4-kube-api-access-kmdkv\") pod \"distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql\" (UID: \"e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4\") " pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:01:13.912874 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.912540 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-istio-envoy\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk\" (UID: \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" Nov 01 16:01:13.913877 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.913825 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-istio-podinfo\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk\" (UID: \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" Nov 01 16:01:13.918536 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.916993 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-istio-token\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk\" (UID: \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" Nov 01 16:01:13.918536 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.917190 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4-dshm\") pod \"distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql\" (UID: \"e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4\") " pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:01:13.919072 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.918878 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4-tls-certs\") pod \"distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql\" (UID: \"e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4\") " pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:01:13.921850 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.921825 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l55gc\" (UniqueName: \"kubernetes.io/projected/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-kube-api-access-l55gc\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk\" (UID: \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" Nov 01 16:01:13.925341 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.925319 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:01:13.977439 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.977404 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg29p\" (UniqueName: \"kubernetes.io/projected/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-kube-api-access-xg29p\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k\" (UID: \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" Nov 01 16:01:13.977603 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.977454 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-istio-data\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k\" (UID: \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" Nov 01 16:01:13.977603 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.977498 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-istio-token\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k\" (UID: \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" Nov 01 16:01:13.977603 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.977554 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-workload-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k\" (UID: \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" Nov 01 16:01:13.977603 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.977575 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-istio-envoy\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k\" (UID: \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" Nov 01 16:01:13.977603 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.977596 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-istio-podinfo\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k\" (UID: \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" Nov 01 16:01:13.977866 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.977667 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-istiod-ca-cert\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k\" (UID: \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" Nov 01 16:01:13.977866 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.977697 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-credential-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k\" (UID: \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" Nov 01 16:01:13.977866 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.977724 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-workload-certs\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k\" (UID: \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" Nov 01 16:01:13.978163 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.978138 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-workload-certs\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k\" (UID: \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" Nov 01 16:01:13.978557 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.978533 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-istio-data\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k\" (UID: \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" Nov 01 16:01:13.979622 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.978980 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-istiod-ca-cert\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k\" (UID: \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" Nov 01 16:01:13.979622 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.979269 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-workload-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k\" (UID: \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" Nov 01 16:01:13.979622 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.979524 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-credential-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k\" (UID: \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" Nov 01 16:01:13.981052 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.980976 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-istio-podinfo\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k\" (UID: \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" Nov 01 16:01:13.981155 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.981131 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-istio-envoy\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k\" (UID: \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" Nov 01 16:01:13.988943 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.985950 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-istio-token\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k\" (UID: \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" Nov 01 16:01:13.995995 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:13.989878 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg29p\" (UniqueName: \"kubernetes.io/projected/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-kube-api-access-xg29p\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k\" (UID: \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" Nov 01 16:01:14.035915 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.035884 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" Nov 01 16:01:14.060717 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.060693 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j"] Nov 01 16:01:14.065948 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.065920 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" Nov 01 16:01:14.080709 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.080676 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j"] Nov 01 16:01:14.183083 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.182973 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9427e75b-2800-4bee-8711-b1e1306d8f09-istio-podinfo\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j\" (UID: \"9427e75b-2800-4bee-8711-b1e1306d8f09\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" Nov 01 16:01:14.183243 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.183188 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnxjl\" (UniqueName: \"kubernetes.io/projected/9427e75b-2800-4bee-8711-b1e1306d8f09-kube-api-access-pnxjl\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j\" (UID: \"9427e75b-2800-4bee-8711-b1e1306d8f09\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" Nov 01 16:01:14.183315 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.183256 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9427e75b-2800-4bee-8711-b1e1306d8f09-istiod-ca-cert\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j\" (UID: \"9427e75b-2800-4bee-8711-b1e1306d8f09\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" Nov 01 16:01:14.183419 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.183396 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9427e75b-2800-4bee-8711-b1e1306d8f09-workload-certs\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j\" (UID: \"9427e75b-2800-4bee-8711-b1e1306d8f09\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" Nov 01 16:01:14.183475 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.183437 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9427e75b-2800-4bee-8711-b1e1306d8f09-workload-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j\" (UID: \"9427e75b-2800-4bee-8711-b1e1306d8f09\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" Nov 01 16:01:14.183546 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.183525 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9427e75b-2800-4bee-8711-b1e1306d8f09-istio-envoy\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j\" (UID: \"9427e75b-2800-4bee-8711-b1e1306d8f09\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" Nov 01 16:01:14.183634 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.183614 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9427e75b-2800-4bee-8711-b1e1306d8f09-istio-token\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j\" (UID: \"9427e75b-2800-4bee-8711-b1e1306d8f09\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" Nov 01 16:01:14.183684 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.183657 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9427e75b-2800-4bee-8711-b1e1306d8f09-istio-data\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j\" (UID: \"9427e75b-2800-4bee-8711-b1e1306d8f09\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" Nov 01 16:01:14.183827 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.183809 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9427e75b-2800-4bee-8711-b1e1306d8f09-credential-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j\" (UID: \"9427e75b-2800-4bee-8711-b1e1306d8f09\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" Nov 01 16:01:14.184899 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.184874 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" Nov 01 16:01:14.230968 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.230941 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql"] Nov 01 16:01:14.231109 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:01:14.230977 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode24b88bf_ca80_4cff_ad74_5e3a69a5b7d4.slice/crio-81f597a986648e1ca4e300dc865b67e136b368e5b132f186e30a441ac41bb957 WatchSource:0}: Error finding container 81f597a986648e1ca4e300dc865b67e136b368e5b132f186e30a441ac41bb957: Status 404 returned error can't find the container with id 81f597a986648e1ca4e300dc865b67e136b368e5b132f186e30a441ac41bb957 Nov 01 16:01:14.284922 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.284792 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9427e75b-2800-4bee-8711-b1e1306d8f09-istiod-ca-cert\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j\" (UID: \"9427e75b-2800-4bee-8711-b1e1306d8f09\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" Nov 01 16:01:14.284922 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.284865 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9427e75b-2800-4bee-8711-b1e1306d8f09-workload-certs\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j\" (UID: \"9427e75b-2800-4bee-8711-b1e1306d8f09\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" Nov 01 16:01:14.284922 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.284891 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9427e75b-2800-4bee-8711-b1e1306d8f09-workload-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j\" (UID: \"9427e75b-2800-4bee-8711-b1e1306d8f09\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" Nov 01 16:01:14.285399 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.285258 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9427e75b-2800-4bee-8711-b1e1306d8f09-istio-envoy\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j\" (UID: \"9427e75b-2800-4bee-8711-b1e1306d8f09\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" Nov 01 16:01:14.285399 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.285333 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9427e75b-2800-4bee-8711-b1e1306d8f09-istio-token\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j\" (UID: \"9427e75b-2800-4bee-8711-b1e1306d8f09\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" Nov 01 16:01:14.285399 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.285363 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9427e75b-2800-4bee-8711-b1e1306d8f09-istio-data\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j\" (UID: \"9427e75b-2800-4bee-8711-b1e1306d8f09\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" Nov 01 16:01:14.286742 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.285628 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9427e75b-2800-4bee-8711-b1e1306d8f09-credential-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j\" (UID: \"9427e75b-2800-4bee-8711-b1e1306d8f09\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" Nov 01 16:01:14.286742 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.285660 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9427e75b-2800-4bee-8711-b1e1306d8f09-istio-podinfo\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j\" (UID: \"9427e75b-2800-4bee-8711-b1e1306d8f09\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" Nov 01 16:01:14.286742 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.285726 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnxjl\" (UniqueName: \"kubernetes.io/projected/9427e75b-2800-4bee-8711-b1e1306d8f09-kube-api-access-pnxjl\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j\" (UID: \"9427e75b-2800-4bee-8711-b1e1306d8f09\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" Nov 01 16:01:14.288410 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.287267 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9427e75b-2800-4bee-8711-b1e1306d8f09-credential-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j\" (UID: \"9427e75b-2800-4bee-8711-b1e1306d8f09\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" Nov 01 16:01:14.288410 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.287547 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9427e75b-2800-4bee-8711-b1e1306d8f09-workload-certs\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j\" (UID: \"9427e75b-2800-4bee-8711-b1e1306d8f09\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" Nov 01 16:01:14.288410 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.287834 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9427e75b-2800-4bee-8711-b1e1306d8f09-workload-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j\" (UID: \"9427e75b-2800-4bee-8711-b1e1306d8f09\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" Nov 01 16:01:14.288410 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.288364 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9427e75b-2800-4bee-8711-b1e1306d8f09-istiod-ca-cert\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j\" (UID: \"9427e75b-2800-4bee-8711-b1e1306d8f09\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" Nov 01 16:01:14.289435 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.289087 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9427e75b-2800-4bee-8711-b1e1306d8f09-istio-data\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j\" (UID: \"9427e75b-2800-4bee-8711-b1e1306d8f09\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" Nov 01 16:01:14.289435 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.289409 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9427e75b-2800-4bee-8711-b1e1306d8f09-istio-podinfo\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j\" (UID: \"9427e75b-2800-4bee-8711-b1e1306d8f09\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" Nov 01 16:01:14.291088 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.290992 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9427e75b-2800-4bee-8711-b1e1306d8f09-istio-envoy\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j\" (UID: \"9427e75b-2800-4bee-8711-b1e1306d8f09\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" Nov 01 16:01:14.298992 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.298969 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnxjl\" (UniqueName: \"kubernetes.io/projected/9427e75b-2800-4bee-8711-b1e1306d8f09-kube-api-access-pnxjl\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j\" (UID: \"9427e75b-2800-4bee-8711-b1e1306d8f09\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" Nov 01 16:01:14.299581 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.299562 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9427e75b-2800-4bee-8711-b1e1306d8f09-istio-token\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j\" (UID: \"9427e75b-2800-4bee-8711-b1e1306d8f09\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" Nov 01 16:01:14.364518 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.364488 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k"] Nov 01 16:01:14.366111 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:01:14.366073 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4070ca0f_1230_4eed_992c_b3f6bce6a4bb.slice/crio-ec9a12e1d0eecb78a65553ff14afb9fc6ea6616c9e38eef12467da8500175c6c WatchSource:0}: Error finding container ec9a12e1d0eecb78a65553ff14afb9fc6ea6616c9e38eef12467da8500175c6c: Status 404 returned error can't find the container with id ec9a12e1d0eecb78a65553ff14afb9fc6ea6616c9e38eef12467da8500175c6c Nov 01 16:01:14.369670 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.369626 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:14.369796 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.369700 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:14.369796 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.369733 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:14.414064 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.414040 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" Nov 01 16:01:14.590422 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:01:14.590377 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aaf0c7e_19c6_4da9_8f12_0cd7ed7dc459.slice/crio-faf56527c1acb87097f900a62733ace0e1bd5a10766c38c2b16d936d04fb7b5b WatchSource:0}: Error finding container faf56527c1acb87097f900a62733ace0e1bd5a10766c38c2b16d936d04fb7b5b: Status 404 returned error can't find the container with id faf56527c1acb87097f900a62733ace0e1bd5a10766c38c2b16d936d04fb7b5b Nov 01 16:01:14.628879 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.595352 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk"] Nov 01 16:01:14.628879 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.599300 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:14.628879 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.599383 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:14.628879 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.599425 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:14.727160 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.727130 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j"] Nov 01 16:01:14.729908 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:01:14.729868 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9427e75b_2800_4bee_8711_b1e1306d8f09.slice/crio-d4f1854c1dafda38865c6222182fb180267ff10077b433d0cc52b51669d6b24e WatchSource:0}: Error finding container d4f1854c1dafda38865c6222182fb180267ff10077b433d0cc52b51669d6b24e: Status 404 returned error can't find the container with id d4f1854c1dafda38865c6222182fb180267ff10077b433d0cc52b51669d6b24e Nov 01 16:01:14.733439 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.733407 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:14.733510 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.733475 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:14.733510 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:14.733505 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:15.143130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:15.143087 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" event={"ID":"e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4","Type":"ContainerStarted","Data":"f2adbe3cf2e7a60c142f65b1279658679fa4e5efabad423f928331184d7ee7d0"} Nov 01 16:01:15.143400 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:15.143367 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" event={"ID":"e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4","Type":"ContainerStarted","Data":"81f597a986648e1ca4e300dc865b67e136b368e5b132f186e30a441ac41bb957"} Nov 01 16:01:15.145154 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:15.145121 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" event={"ID":"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459","Type":"ContainerStarted","Data":"5d4172fa160490c368e0f1fde63fb6de21ef76011542bd53966bb6c2abe80c2d"} Nov 01 16:01:15.145282 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:15.145217 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" event={"ID":"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459","Type":"ContainerStarted","Data":"faf56527c1acb87097f900a62733ace0e1bd5a10766c38c2b16d936d04fb7b5b"} Nov 01 16:01:15.147180 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:15.147133 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" event={"ID":"4070ca0f-1230-4eed-992c-b3f6bce6a4bb","Type":"ContainerStarted","Data":"0f8a65fdecd9f9f6a94b9d39b2f3d7f99e0466c5a33d37a06bb6903f4f8bf4b3"} Nov 01 16:01:15.147180 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:15.147174 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" event={"ID":"4070ca0f-1230-4eed-992c-b3f6bce6a4bb","Type":"ContainerStarted","Data":"ec9a12e1d0eecb78a65553ff14afb9fc6ea6616c9e38eef12467da8500175c6c"} Nov 01 16:01:15.148975 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:15.148956 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" event={"ID":"9427e75b-2800-4bee-8711-b1e1306d8f09","Type":"ContainerStarted","Data":"89fc859a7cce033af5e69edd157263ee4c5548312cf3b3998a6b2c01d2e6bb2d"} Nov 01 16:01:15.149080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:15.148989 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" event={"ID":"9427e75b-2800-4bee-8711-b1e1306d8f09","Type":"ContainerStarted","Data":"d4f1854c1dafda38865c6222182fb180267ff10077b433d0cc52b51669d6b24e"} Nov 01 16:01:15.163110 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:15.163052 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podStartSLOduration=2.1630337219999998 podStartE2EDuration="2.163033722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:01:15.158038821 +0000 UTC m=+808.489776469" watchObservedRunningTime="2026-11-01 16:01:15.163033722 +0000 UTC m=+808.494771370" Nov 01 16:01:15.182878 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:15.182722 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" podStartSLOduration=2.182672665 podStartE2EDuration="2.182672665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:01:15.176051097 +0000 UTC m=+808.507788748" watchObservedRunningTime="2026-11-01 16:01:15.182672665 +0000 UTC m=+808.514410317" Nov 01 16:01:15.185960 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:15.185935 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" Nov 01 16:01:15.187641 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:15.187616 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 01 16:01:15.187773 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:15.187665 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" podUID="4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 01 16:01:15.193227 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:15.193044 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" podStartSLOduration=2.193030461 podStartE2EDuration="2.193030461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:01:15.192617551 +0000 UTC m=+808.524355202" watchObservedRunningTime="2026-11-01 16:01:15.193030461 +0000 UTC m=+808.524768103" Nov 01 16:01:15.215715 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:15.215664 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" podStartSLOduration=1.215650721 podStartE2EDuration="1.215650721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:01:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:01:15.21246643 +0000 UTC m=+808.544204081" watchObservedRunningTime="2026-11-01 16:01:15.215650721 +0000 UTC m=+808.547388370" Nov 01 16:01:15.418169 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:15.418081 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" Nov 01 16:01:15.420485 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:15.420453 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 01 16:01:15.420739 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:15.420713 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" podUID="9427e75b-2800-4bee-8711-b1e1306d8f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 01 16:01:15.928143 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:15.928105 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:01:16.048619 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:16.048582 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" Nov 01 16:01:16.057365 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:16.056987 4363 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 01 16:01:16.057365 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:16.057064 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" podUID="4070ca0f-1230-4eed-992c-b3f6bce6a4bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 01 16:01:16.187568 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:16.187474 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 01 16:01:16.187803 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:16.187772 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" podUID="4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 01 16:01:16.424930 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:16.424896 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 01 16:01:16.425245 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:16.425218 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" podUID="9427e75b-2800-4bee-8711-b1e1306d8f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 01 16:01:17.038044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:17.037113 4363 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 01 16:01:17.038044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:17.037179 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" podUID="4070ca0f-1230-4eed-992c-b3f6bce6a4bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 01 16:01:17.195042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:17.192770 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 01 16:01:17.195042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:17.192833 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" podUID="4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 01 16:01:17.418044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:17.417524 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 01 16:01:17.418044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:17.417614 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" podUID="9427e75b-2800-4bee-8711-b1e1306d8f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 01 16:01:18.039604 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:18.039565 4363 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 01 16:01:18.040262 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:18.040213 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" podUID="4070ca0f-1230-4eed-992c-b3f6bce6a4bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 01 16:01:18.191074 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:18.191034 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 01 16:01:18.206272 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:18.206231 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" podUID="4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 01 16:01:18.424886 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:18.424417 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 01 16:01:18.424886 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:18.424482 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" podUID="9427e75b-2800-4bee-8711-b1e1306d8f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 01 16:01:18.460309 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:01:18.460258 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:01:19.045039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:19.044162 4363 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 01 16:01:19.045039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:19.044221 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" podUID="4070ca0f-1230-4eed-992c-b3f6bce6a4bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 01 16:01:19.195112 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:19.190438 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 01 16:01:19.195112 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:19.190501 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" podUID="4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 01 16:01:19.453192 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:19.453156 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 01 16:01:19.453518 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:19.453469 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" podUID="9427e75b-2800-4bee-8711-b1e1306d8f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 01 16:01:20.036482 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:20.036439 4363 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 01 16:01:20.036745 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:20.036716 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" podUID="4070ca0f-1230-4eed-992c-b3f6bce6a4bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 01 16:01:20.188893 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:20.188426 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 01 16:01:20.188893 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:20.188492 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" podUID="4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 01 16:01:20.406044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:20.391274 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:01:20.456048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:20.455617 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 01 16:01:20.456048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:20.455676 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" podUID="9427e75b-2800-4bee-8711-b1e1306d8f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 01 16:01:20.611426 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:20.611392 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:01:20.966071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:20.966037 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:01:21.044418 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:21.044126 4363 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 01 16:01:21.044418 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:21.044187 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" podUID="4070ca0f-1230-4eed-992c-b3f6bce6a4bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 01 16:01:21.337197 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:21.337090 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 01 16:01:21.337772 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:21.337737 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" podUID="4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 01 16:01:21.419998 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:21.419537 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 01 16:01:21.419998 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:21.419601 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" podUID="9427e75b-2800-4bee-8711-b1e1306d8f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 01 16:01:22.080042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:22.039461 4363 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 01 16:01:22.080370 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:22.080325 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" podUID="4070ca0f-1230-4eed-992c-b3f6bce6a4bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 01 16:01:22.206609 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:22.206569 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 01 16:01:22.206880 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:22.206845 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" podUID="4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 01 16:01:22.418591 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:22.418553 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 01 16:01:22.419267 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:22.419233 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" podUID="9427e75b-2800-4bee-8711-b1e1306d8f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 01 16:01:23.050042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:23.039645 4363 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 01 16:01:23.050042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:23.039724 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" podUID="4070ca0f-1230-4eed-992c-b3f6bce6a4bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 01 16:01:23.188435 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:23.188396 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 01 16:01:23.188754 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:23.188723 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" podUID="4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 01 16:01:23.697225 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:23.697165 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:23.952499 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:23.951502 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:01:24.005040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:24.004107 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:01:24.281289 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:24.281164 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:01:24.330912 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:24.322993 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:24.344917 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:24.338114 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:24.490476 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:24.489815 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:01:24.490476 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:24.489882 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" podUID="9427e75b-2800-4bee-8711-b1e1306d8f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:24.707056 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:24.703266 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:01:24.707056 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:24.704696 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:01:24.971096 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:24.962124 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded" Nov 01 16:01:24.971096 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:24.963803 4363 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:25.009042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:25.008442 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:25.040041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:25.009437 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:01:25.070049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:25.069115 4363 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:01:25.070049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:25.069182 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" podUID="4070ca0f-1230-4eed-992c-b3f6bce6a4bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:25.302549 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:25.302453 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:25.332610 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:25.332576 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:01:25.332875 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:25.332846 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" podUID="4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:25.342178 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:25.342131 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:25.369163 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:25.369133 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 16:01:25.393419 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:25.371091 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:25.394744 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:25.394717 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 16:01:25.552040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:25.548182 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:01:25.552040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:25.548247 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" podUID="9427e75b-2800-4bee-8711-b1e1306d8f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:25.724041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:25.704719 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:26.075960 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:26.075860 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:01:26.235598 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:26.235553 4363 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:01:26.235875 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:26.235839 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" podUID="4070ca0f-1230-4eed-992c-b3f6bce6a4bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:26.361756 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:26.345927 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 16:01:26.361756 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:26.347127 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:26.361756 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:26.347522 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:01:26.393088 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:26.393058 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 16:01:26.422175 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:26.405419 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:01:26.630689 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:26.630561 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" podUID="4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:26.631412 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:26.502452 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:01:26.784482 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:26.773950 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" Nov 01 16:01:26.823079 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:26.788348 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:01:26.823079 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:26.822543 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" podUID="9427e75b-2800-4bee-8711-b1e1306d8f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:27.046054 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:27.045954 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" Nov 01 16:01:27.062791 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:27.062767 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" Nov 01 16:01:27.079123 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:27.079084 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:27.194041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:27.177677 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:01:27.194041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:27.178566 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:01:27.290599 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:27.290542 4363 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:01:27.310721 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:27.308588 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" podUID="4070ca0f-1230-4eed-992c-b3f6bce6a4bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:27.628669 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:27.628632 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" Nov 01 16:01:27.630034 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:27.629860 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" Nov 01 16:01:27.631386 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:27.631333 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" Nov 01 16:01:27.864158 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:27.864124 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:01:27.864509 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:27.864196 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" podUID="9427e75b-2800-4bee-8711-b1e1306d8f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:28.001433 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:28.001354 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" Nov 01 16:01:28.003141 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:28.003115 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" Nov 01 16:01:28.029074 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:28.029043 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" Nov 01 16:01:28.566593 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:28.566563 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw"] Nov 01 16:01:28.571697 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:28.571675 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" Nov 01 16:01:28.580162 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:28.580134 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw"] Nov 01 16:01:28.629093 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:28.629061 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3d831af-96a0-4040-8568-e092380a88c1-istio-token\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw\" (UID: \"b3d831af-96a0-4040-8568-e092380a88c1\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" Nov 01 16:01:28.629245 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:28.629216 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3d831af-96a0-4040-8568-e092380a88c1-istio-podinfo\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw\" (UID: \"b3d831af-96a0-4040-8568-e092380a88c1\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" Nov 01 16:01:28.629294 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:28.629265 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3d831af-96a0-4040-8568-e092380a88c1-istiod-ca-cert\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw\" (UID: \"b3d831af-96a0-4040-8568-e092380a88c1\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" Nov 01 16:01:28.629358 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:28.629331 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3d831af-96a0-4040-8568-e092380a88c1-workload-certs\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw\" (UID: \"b3d831af-96a0-4040-8568-e092380a88c1\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" Nov 01 16:01:28.629401 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:28.629393 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3d831af-96a0-4040-8568-e092380a88c1-credential-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw\" (UID: \"b3d831af-96a0-4040-8568-e092380a88c1\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" Nov 01 16:01:28.629445 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:28.629409 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3d831af-96a0-4040-8568-e092380a88c1-istio-data\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw\" (UID: \"b3d831af-96a0-4040-8568-e092380a88c1\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" Nov 01 16:01:28.629445 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:28.629425 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbnkf\" (UniqueName: \"kubernetes.io/projected/b3d831af-96a0-4040-8568-e092380a88c1-kube-api-access-sbnkf\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw\" (UID: \"b3d831af-96a0-4040-8568-e092380a88c1\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" Nov 01 16:01:28.629592 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:28.629573 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3d831af-96a0-4040-8568-e092380a88c1-workload-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw\" (UID: \"b3d831af-96a0-4040-8568-e092380a88c1\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" Nov 01 16:01:28.629654 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:28.629638 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3d831af-96a0-4040-8568-e092380a88c1-istio-envoy\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw\" (UID: \"b3d831af-96a0-4040-8568-e092380a88c1\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" Nov 01 16:01:28.731350 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:28.731315 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3d831af-96a0-4040-8568-e092380a88c1-istio-token\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw\" (UID: \"b3d831af-96a0-4040-8568-e092380a88c1\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" Nov 01 16:01:28.731525 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:28.731386 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3d831af-96a0-4040-8568-e092380a88c1-istio-podinfo\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw\" (UID: \"b3d831af-96a0-4040-8568-e092380a88c1\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" Nov 01 16:01:28.731525 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:28.731404 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3d831af-96a0-4040-8568-e092380a88c1-istiod-ca-cert\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw\" (UID: \"b3d831af-96a0-4040-8568-e092380a88c1\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" Nov 01 16:01:28.731525 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:28.731425 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3d831af-96a0-4040-8568-e092380a88c1-workload-certs\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw\" (UID: \"b3d831af-96a0-4040-8568-e092380a88c1\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" Nov 01 16:01:28.731525 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:28.731474 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3d831af-96a0-4040-8568-e092380a88c1-credential-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw\" (UID: \"b3d831af-96a0-4040-8568-e092380a88c1\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" Nov 01 16:01:28.731525 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:28.731488 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3d831af-96a0-4040-8568-e092380a88c1-istio-data\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw\" (UID: \"b3d831af-96a0-4040-8568-e092380a88c1\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" Nov 01 16:01:28.731525 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:28.731509 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbnkf\" (UniqueName: \"kubernetes.io/projected/b3d831af-96a0-4040-8568-e092380a88c1-kube-api-access-sbnkf\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw\" (UID: \"b3d831af-96a0-4040-8568-e092380a88c1\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" Nov 01 16:01:28.732211 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:28.731577 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3d831af-96a0-4040-8568-e092380a88c1-workload-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw\" (UID: \"b3d831af-96a0-4040-8568-e092380a88c1\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" Nov 01 16:01:28.732211 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:28.731616 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3d831af-96a0-4040-8568-e092380a88c1-istio-envoy\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw\" (UID: \"b3d831af-96a0-4040-8568-e092380a88c1\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" Nov 01 16:01:28.732211 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:28.731965 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3d831af-96a0-4040-8568-e092380a88c1-credential-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw\" (UID: \"b3d831af-96a0-4040-8568-e092380a88c1\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" Nov 01 16:01:28.732211 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:28.732064 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3d831af-96a0-4040-8568-e092380a88c1-istio-data\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw\" (UID: \"b3d831af-96a0-4040-8568-e092380a88c1\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" Nov 01 16:01:28.732211 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:28.732172 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3d831af-96a0-4040-8568-e092380a88c1-workload-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw\" (UID: \"b3d831af-96a0-4040-8568-e092380a88c1\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" Nov 01 16:01:28.732418 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:28.732278 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3d831af-96a0-4040-8568-e092380a88c1-workload-certs\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw\" (UID: \"b3d831af-96a0-4040-8568-e092380a88c1\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" Nov 01 16:01:28.732418 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:28.732322 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3d831af-96a0-4040-8568-e092380a88c1-istiod-ca-cert\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw\" (UID: \"b3d831af-96a0-4040-8568-e092380a88c1\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" Nov 01 16:01:28.733773 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:28.733752 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3d831af-96a0-4040-8568-e092380a88c1-istio-envoy\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw\" (UID: \"b3d831af-96a0-4040-8568-e092380a88c1\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" Nov 01 16:01:28.733902 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:28.733884 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3d831af-96a0-4040-8568-e092380a88c1-istio-podinfo\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw\" (UID: \"b3d831af-96a0-4040-8568-e092380a88c1\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" Nov 01 16:01:28.738608 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:28.738584 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3d831af-96a0-4040-8568-e092380a88c1-istio-token\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw\" (UID: \"b3d831af-96a0-4040-8568-e092380a88c1\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" Nov 01 16:01:28.739342 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:28.739323 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbnkf\" (UniqueName: \"kubernetes.io/projected/b3d831af-96a0-4040-8568-e092380a88c1-kube-api-access-sbnkf\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw\" (UID: \"b3d831af-96a0-4040-8568-e092380a88c1\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" Nov 01 16:01:28.921071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:28.899103 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" Nov 01 16:01:29.295288 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:29.295171 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw"] Nov 01 16:01:29.313287 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:01:29.313208 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3d831af_96a0_4040_8568_e092380a88c1.slice/crio-103d4f786d309019628abecfbf9735da1ba2f29738fe240270b977bb0d838153 WatchSource:0}: Error finding container 103d4f786d309019628abecfbf9735da1ba2f29738fe240270b977bb0d838153: Status 404 returned error can't find the container with id 103d4f786d309019628abecfbf9735da1ba2f29738fe240270b977bb0d838153 Nov 01 16:01:29.315451 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:29.315415 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:29.315560 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:29.315491 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:29.315560 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:29.315523 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:29.335807 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:29.335777 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" event={"ID":"b3d831af-96a0-4040-8568-e092380a88c1","Type":"ContainerStarted","Data":"103d4f786d309019628abecfbf9735da1ba2f29738fe240270b977bb0d838153"} Nov 01 16:01:30.344325 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:30.344288 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" event={"ID":"b3d831af-96a0-4040-8568-e092380a88c1","Type":"ContainerStarted","Data":"253b665ee5711dcb85fc8258bfbeba181f9ca1279a9748d8ae24d378a3ca5290"} Nov 01 16:01:30.369394 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:30.369347 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" podStartSLOduration=2.369334206 podStartE2EDuration="2.369334206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:01:30.368457302 +0000 UTC m=+823.700194953" watchObservedRunningTime="2026-11-01 16:01:30.369334206 +0000 UTC m=+823.701071856" Nov 01 16:01:30.916041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:30.903731 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 01 16:01:30.916041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:30.903804 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" podUID="b3d831af-96a0-4040-8568-e092380a88c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 01 16:01:30.916041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:30.904131 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" Nov 01 16:01:31.910052 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:31.906357 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 01 16:01:31.910052 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:31.906417 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" podUID="b3d831af-96a0-4040-8568-e092380a88c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 01 16:01:32.909421 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:32.908960 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 01 16:01:32.909421 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:32.909054 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" podUID="b3d831af-96a0-4040-8568-e092380a88c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 01 16:01:33.911038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:33.910509 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 01 16:01:33.911038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:33.910574 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" podUID="b3d831af-96a0-4040-8568-e092380a88c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 01 16:01:33.942850 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:33.942753 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:01:33.955354 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:33.955331 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:01:34.909660 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:34.908391 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 01 16:01:34.909660 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:34.908498 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" podUID="b3d831af-96a0-4040-8568-e092380a88c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 01 16:01:35.900248 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:35.900218 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 01 16:01:35.900729 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:35.900265 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" podUID="b3d831af-96a0-4040-8568-e092380a88c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 01 16:01:36.904419 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:36.904389 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" Nov 01 16:01:36.904830 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:36.904790 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" Nov 01 16:01:36.905570 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:36.905548 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" Nov 01 16:01:44.038682 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.038643 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8"] Nov 01 16:01:44.042983 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.042933 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" Nov 01 16:01:44.061259 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.061232 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79"] Nov 01 16:01:44.068730 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.068701 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" Nov 01 16:01:44.078459 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.078435 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8"] Nov 01 16:01:44.086214 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.086190 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79"] Nov 01 16:01:44.133268 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.133233 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz"] Nov 01 16:01:44.138443 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.138419 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" Nov 01 16:01:44.150240 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.150208 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz"] Nov 01 16:01:44.205051 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.204976 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edcab30a-6e4e-4019-9fd7-96fe7b085711-workload-certs\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79\" (UID: \"edcab30a-6e4e-4019-9fd7-96fe7b085711\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" Nov 01 16:01:44.266479 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.262075 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edcab30a-6e4e-4019-9fd7-96fe7b085711-istio-podinfo\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79\" (UID: \"edcab30a-6e4e-4019-9fd7-96fe7b085711\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" Nov 01 16:01:44.267769 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.267728 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edcab30a-6e4e-4019-9fd7-96fe7b085711-istio-token\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79\" (UID: \"edcab30a-6e4e-4019-9fd7-96fe7b085711\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" Nov 01 16:01:44.268076 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.268057 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwrd6\" (UniqueName: \"kubernetes.io/projected/edcab30a-6e4e-4019-9fd7-96fe7b085711-kube-api-access-cwrd6\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79\" (UID: \"edcab30a-6e4e-4019-9fd7-96fe7b085711\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" Nov 01 16:01:44.268302 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.268286 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcab30a-6e4e-4019-9fd7-96fe7b085711-credential-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79\" (UID: \"edcab30a-6e4e-4019-9fd7-96fe7b085711\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" Nov 01 16:01:44.268488 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.268471 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b704396-3411-40af-95fe-4b67a41ae69f-credential-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8\" (UID: \"3b704396-3411-40af-95fe-4b67a41ae69f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" Nov 01 16:01:44.268708 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.268685 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b704396-3411-40af-95fe-4b67a41ae69f-workload-certs\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8\" (UID: \"3b704396-3411-40af-95fe-4b67a41ae69f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" Nov 01 16:01:44.269053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.269033 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b704396-3411-40af-95fe-4b67a41ae69f-istio-envoy\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8\" (UID: \"3b704396-3411-40af-95fe-4b67a41ae69f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" Nov 01 16:01:44.269261 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.269237 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b704396-3411-40af-95fe-4b67a41ae69f-istiod-ca-cert\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8\" (UID: \"3b704396-3411-40af-95fe-4b67a41ae69f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" Nov 01 16:01:44.269404 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.269382 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edcab30a-6e4e-4019-9fd7-96fe7b085711-istio-envoy\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79\" (UID: \"edcab30a-6e4e-4019-9fd7-96fe7b085711\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" Nov 01 16:01:44.269552 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.269539 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b704396-3411-40af-95fe-4b67a41ae69f-istio-podinfo\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8\" (UID: \"3b704396-3411-40af-95fe-4b67a41ae69f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" Nov 01 16:01:44.271299 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.271266 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcab30a-6e4e-4019-9fd7-96fe7b085711-workload-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79\" (UID: \"edcab30a-6e4e-4019-9fd7-96fe7b085711\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" Nov 01 16:01:44.271589 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.271570 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edcab30a-6e4e-4019-9fd7-96fe7b085711-istiod-ca-cert\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79\" (UID: \"edcab30a-6e4e-4019-9fd7-96fe7b085711\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" Nov 01 16:01:44.271858 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.271825 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b704396-3411-40af-95fe-4b67a41ae69f-istio-token\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8\" (UID: \"3b704396-3411-40af-95fe-4b67a41ae69f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" Nov 01 16:01:44.272037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.272006 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm6ln\" (UniqueName: \"kubernetes.io/projected/3b704396-3411-40af-95fe-4b67a41ae69f-kube-api-access-mm6ln\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8\" (UID: \"3b704396-3411-40af-95fe-4b67a41ae69f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" Nov 01 16:01:44.275794 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.275775 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b704396-3411-40af-95fe-4b67a41ae69f-workload-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8\" (UID: \"3b704396-3411-40af-95fe-4b67a41ae69f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" Nov 01 16:01:44.275990 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.275964 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b704396-3411-40af-95fe-4b67a41ae69f-istio-data\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8\" (UID: \"3b704396-3411-40af-95fe-4b67a41ae69f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" Nov 01 16:01:44.276307 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.276257 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edcab30a-6e4e-4019-9fd7-96fe7b085711-istio-data\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79\" (UID: \"edcab30a-6e4e-4019-9fd7-96fe7b085711\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" Nov 01 16:01:44.324648 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.305180 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv"] Nov 01 16:01:44.332065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.327234 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv"] Nov 01 16:01:44.332065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.327360 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" Nov 01 16:01:44.339692 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.339530 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4"] Nov 01 16:01:44.358103 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.358061 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4"] Nov 01 16:01:44.371391 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.371104 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" Nov 01 16:01:44.380464 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.380435 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05ec5955-5241-423b-8161-1ac7608f4fed-istio-data\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv\" (UID: \"05ec5955-5241-423b-8161-1ac7608f4fed\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" Nov 01 16:01:44.380674 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.380653 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05ec5955-5241-423b-8161-1ac7608f4fed-istiod-ca-cert\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv\" (UID: \"05ec5955-5241-423b-8161-1ac7608f4fed\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" Nov 01 16:01:44.380794 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.380778 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ec5955-5241-423b-8161-1ac7608f4fed-workload-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv\" (UID: \"05ec5955-5241-423b-8161-1ac7608f4fed\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" Nov 01 16:01:44.380917 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.380903 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b704396-3411-40af-95fe-4b67a41ae69f-workload-certs\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8\" (UID: \"3b704396-3411-40af-95fe-4b67a41ae69f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" Nov 01 16:01:44.381038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.381009 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408aa757-573f-4cd4-b81d-c26c9af252f2-istio-data\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz\" (UID: \"408aa757-573f-4cd4-b81d-c26c9af252f2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" Nov 01 16:01:44.381173 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.381157 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edcab30a-6e4e-4019-9fd7-96fe7b085711-istio-envoy\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79\" (UID: \"edcab30a-6e4e-4019-9fd7-96fe7b085711\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" Nov 01 16:01:44.381288 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.381273 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b704396-3411-40af-95fe-4b67a41ae69f-istio-podinfo\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8\" (UID: \"3b704396-3411-40af-95fe-4b67a41ae69f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" Nov 01 16:01:44.381405 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.381390 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm6ln\" (UniqueName: \"kubernetes.io/projected/3b704396-3411-40af-95fe-4b67a41ae69f-kube-api-access-mm6ln\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8\" (UID: \"3b704396-3411-40af-95fe-4b67a41ae69f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" Nov 01 16:01:44.381537 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.381515 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcab30a-6e4e-4019-9fd7-96fe7b085711-workload-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79\" (UID: \"edcab30a-6e4e-4019-9fd7-96fe7b085711\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" Nov 01 16:01:44.381669 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.381653 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05ec5955-5241-423b-8161-1ac7608f4fed-istio-envoy\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv\" (UID: \"05ec5955-5241-423b-8161-1ac7608f4fed\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" Nov 01 16:01:44.382361 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.382338 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l525n\" (UniqueName: \"kubernetes.io/projected/05ec5955-5241-423b-8161-1ac7608f4fed-kube-api-access-l525n\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv\" (UID: \"05ec5955-5241-423b-8161-1ac7608f4fed\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" Nov 01 16:01:44.383079 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.383003 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8rzn\" (UniqueName: \"kubernetes.io/projected/408aa757-573f-4cd4-b81d-c26c9af252f2-kube-api-access-q8rzn\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz\" (UID: \"408aa757-573f-4cd4-b81d-c26c9af252f2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" Nov 01 16:01:44.383803 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.383335 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcab30a-6e4e-4019-9fd7-96fe7b085711-workload-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79\" (UID: \"edcab30a-6e4e-4019-9fd7-96fe7b085711\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" Nov 01 16:01:44.383939 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.382813 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b704396-3411-40af-95fe-4b67a41ae69f-workload-certs\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8\" (UID: \"3b704396-3411-40af-95fe-4b67a41ae69f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" Nov 01 16:01:44.384130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.384105 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b704396-3411-40af-95fe-4b67a41ae69f-istio-data\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8\" (UID: \"3b704396-3411-40af-95fe-4b67a41ae69f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" Nov 01 16:01:44.384263 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.384248 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408aa757-573f-4cd4-b81d-c26c9af252f2-istio-envoy\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz\" (UID: \"408aa757-573f-4cd4-b81d-c26c9af252f2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" Nov 01 16:01:44.384363 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.384351 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408aa757-573f-4cd4-b81d-c26c9af252f2-istiod-ca-cert\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz\" (UID: \"408aa757-573f-4cd4-b81d-c26c9af252f2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" Nov 01 16:01:44.384479 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.384467 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edcab30a-6e4e-4019-9fd7-96fe7b085711-workload-certs\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79\" (UID: \"edcab30a-6e4e-4019-9fd7-96fe7b085711\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" Nov 01 16:01:44.384604 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.384587 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edcab30a-6e4e-4019-9fd7-96fe7b085711-istio-podinfo\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79\" (UID: \"edcab30a-6e4e-4019-9fd7-96fe7b085711\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" Nov 01 16:01:44.384699 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.384644 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b704396-3411-40af-95fe-4b67a41ae69f-istio-data\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8\" (UID: \"3b704396-3411-40af-95fe-4b67a41ae69f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" Nov 01 16:01:44.384790 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.384767 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edcab30a-6e4e-4019-9fd7-96fe7b085711-istio-token\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79\" (UID: \"edcab30a-6e4e-4019-9fd7-96fe7b085711\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" Nov 01 16:01:44.388071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.385187 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwrd6\" (UniqueName: \"kubernetes.io/projected/edcab30a-6e4e-4019-9fd7-96fe7b085711-kube-api-access-cwrd6\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79\" (UID: \"edcab30a-6e4e-4019-9fd7-96fe7b085711\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" Nov 01 16:01:44.388071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.385233 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ec5955-5241-423b-8161-1ac7608f4fed-credential-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv\" (UID: \"05ec5955-5241-423b-8161-1ac7608f4fed\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" Nov 01 16:01:44.388071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.385307 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcab30a-6e4e-4019-9fd7-96fe7b085711-credential-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79\" (UID: \"edcab30a-6e4e-4019-9fd7-96fe7b085711\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" Nov 01 16:01:44.388071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.385337 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05ec5955-5241-423b-8161-1ac7608f4fed-istio-podinfo\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv\" (UID: \"05ec5955-5241-423b-8161-1ac7608f4fed\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" Nov 01 16:01:44.388071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.385391 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05ec5955-5241-423b-8161-1ac7608f4fed-workload-certs\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv\" (UID: \"05ec5955-5241-423b-8161-1ac7608f4fed\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" Nov 01 16:01:44.388071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.385426 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b704396-3411-40af-95fe-4b67a41ae69f-istio-envoy\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8\" (UID: \"3b704396-3411-40af-95fe-4b67a41ae69f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" Nov 01 16:01:44.388071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.385451 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408aa757-573f-4cd4-b81d-c26c9af252f2-credential-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz\" (UID: \"408aa757-573f-4cd4-b81d-c26c9af252f2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" Nov 01 16:01:44.388071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.385484 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b704396-3411-40af-95fe-4b67a41ae69f-istiod-ca-cert\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8\" (UID: \"3b704396-3411-40af-95fe-4b67a41ae69f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" Nov 01 16:01:44.388071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.385515 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edcab30a-6e4e-4019-9fd7-96fe7b085711-istiod-ca-cert\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79\" (UID: \"edcab30a-6e4e-4019-9fd7-96fe7b085711\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" Nov 01 16:01:44.388071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.385568 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b704396-3411-40af-95fe-4b67a41ae69f-istio-token\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8\" (UID: \"3b704396-3411-40af-95fe-4b67a41ae69f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" Nov 01 16:01:44.388071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.385592 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408aa757-573f-4cd4-b81d-c26c9af252f2-istio-token\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz\" (UID: \"408aa757-573f-4cd4-b81d-c26c9af252f2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" Nov 01 16:01:44.388071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.385630 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b704396-3411-40af-95fe-4b67a41ae69f-workload-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8\" (UID: \"3b704396-3411-40af-95fe-4b67a41ae69f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" Nov 01 16:01:44.388071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.385658 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408aa757-573f-4cd4-b81d-c26c9af252f2-istio-podinfo\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz\" (UID: \"408aa757-573f-4cd4-b81d-c26c9af252f2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" Nov 01 16:01:44.388071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.385704 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408aa757-573f-4cd4-b81d-c26c9af252f2-workload-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz\" (UID: \"408aa757-573f-4cd4-b81d-c26c9af252f2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" Nov 01 16:01:44.388071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.385740 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edcab30a-6e4e-4019-9fd7-96fe7b085711-istio-data\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79\" (UID: \"edcab30a-6e4e-4019-9fd7-96fe7b085711\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" Nov 01 16:01:44.388071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.385796 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05ec5955-5241-423b-8161-1ac7608f4fed-istio-token\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv\" (UID: \"05ec5955-5241-423b-8161-1ac7608f4fed\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" Nov 01 16:01:44.388071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.385836 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408aa757-573f-4cd4-b81d-c26c9af252f2-workload-certs\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz\" (UID: \"408aa757-573f-4cd4-b81d-c26c9af252f2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" Nov 01 16:01:44.388071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.385889 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b704396-3411-40af-95fe-4b67a41ae69f-credential-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8\" (UID: \"3b704396-3411-40af-95fe-4b67a41ae69f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" Nov 01 16:01:44.388071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.386442 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b704396-3411-40af-95fe-4b67a41ae69f-credential-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8\" (UID: \"3b704396-3411-40af-95fe-4b67a41ae69f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" Nov 01 16:01:44.388071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.387126 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcab30a-6e4e-4019-9fd7-96fe7b085711-credential-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79\" (UID: \"edcab30a-6e4e-4019-9fd7-96fe7b085711\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" Nov 01 16:01:44.389347 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.389328 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edcab30a-6e4e-4019-9fd7-96fe7b085711-workload-certs\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79\" (UID: \"edcab30a-6e4e-4019-9fd7-96fe7b085711\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" Nov 01 16:01:44.389707 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.389687 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b704396-3411-40af-95fe-4b67a41ae69f-workload-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8\" (UID: \"3b704396-3411-40af-95fe-4b67a41ae69f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" Nov 01 16:01:44.389935 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.389910 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b704396-3411-40af-95fe-4b67a41ae69f-istio-envoy\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8\" (UID: \"3b704396-3411-40af-95fe-4b67a41ae69f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" Nov 01 16:01:44.390257 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.390233 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edcab30a-6e4e-4019-9fd7-96fe7b085711-istio-data\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79\" (UID: \"edcab30a-6e4e-4019-9fd7-96fe7b085711\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" Nov 01 16:01:44.390745 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.390726 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b704396-3411-40af-95fe-4b67a41ae69f-istiod-ca-cert\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8\" (UID: \"3b704396-3411-40af-95fe-4b67a41ae69f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" Nov 01 16:01:44.391326 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.391301 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edcab30a-6e4e-4019-9fd7-96fe7b085711-istiod-ca-cert\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79\" (UID: \"edcab30a-6e4e-4019-9fd7-96fe7b085711\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" Nov 01 16:01:44.393150 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.393123 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edcab30a-6e4e-4019-9fd7-96fe7b085711-istio-envoy\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79\" (UID: \"edcab30a-6e4e-4019-9fd7-96fe7b085711\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" Nov 01 16:01:44.395048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.395007 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edcab30a-6e4e-4019-9fd7-96fe7b085711-istio-podinfo\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79\" (UID: \"edcab30a-6e4e-4019-9fd7-96fe7b085711\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" Nov 01 16:01:44.396655 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.395836 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b704396-3411-40af-95fe-4b67a41ae69f-istio-podinfo\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8\" (UID: \"3b704396-3411-40af-95fe-4b67a41ae69f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" Nov 01 16:01:44.398880 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.398854 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm6ln\" (UniqueName: \"kubernetes.io/projected/3b704396-3411-40af-95fe-4b67a41ae69f-kube-api-access-mm6ln\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8\" (UID: \"3b704396-3411-40af-95fe-4b67a41ae69f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" Nov 01 16:01:44.400711 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.400689 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edcab30a-6e4e-4019-9fd7-96fe7b085711-istio-token\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79\" (UID: \"edcab30a-6e4e-4019-9fd7-96fe7b085711\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" Nov 01 16:01:44.402084 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.402061 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwrd6\" (UniqueName: \"kubernetes.io/projected/edcab30a-6e4e-4019-9fd7-96fe7b085711-kube-api-access-cwrd6\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79\" (UID: \"edcab30a-6e4e-4019-9fd7-96fe7b085711\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" Nov 01 16:01:44.403697 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.403674 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b704396-3411-40af-95fe-4b67a41ae69f-istio-token\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8\" (UID: \"3b704396-3411-40af-95fe-4b67a41ae69f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" Nov 01 16:01:44.487271 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.487231 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ec5955-5241-423b-8161-1ac7608f4fed-credential-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv\" (UID: \"05ec5955-5241-423b-8161-1ac7608f4fed\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" Nov 01 16:01:44.487453 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.487303 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05ec5955-5241-423b-8161-1ac7608f4fed-istio-podinfo\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv\" (UID: \"05ec5955-5241-423b-8161-1ac7608f4fed\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" Nov 01 16:01:44.487453 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.487345 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef2387a8-1f76-4653-9047-0f5824e6cfbb-istiod-ca-cert\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4\" (UID: \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" Nov 01 16:01:44.487453 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.487384 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05ec5955-5241-423b-8161-1ac7608f4fed-workload-certs\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv\" (UID: \"05ec5955-5241-423b-8161-1ac7608f4fed\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" Nov 01 16:01:44.487632 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.487470 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408aa757-573f-4cd4-b81d-c26c9af252f2-credential-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz\" (UID: \"408aa757-573f-4cd4-b81d-c26c9af252f2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" Nov 01 16:01:44.487632 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.487528 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef2387a8-1f76-4653-9047-0f5824e6cfbb-workload-certs\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4\" (UID: \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" Nov 01 16:01:44.487632 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.487559 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2387a8-1f76-4653-9047-0f5824e6cfbb-credential-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4\" (UID: \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" Nov 01 16:01:44.487632 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.487625 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408aa757-573f-4cd4-b81d-c26c9af252f2-istio-token\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz\" (UID: \"408aa757-573f-4cd4-b81d-c26c9af252f2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" Nov 01 16:01:44.487835 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.487644 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ec5955-5241-423b-8161-1ac7608f4fed-credential-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv\" (UID: \"05ec5955-5241-423b-8161-1ac7608f4fed\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" Nov 01 16:01:44.487835 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.487680 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408aa757-573f-4cd4-b81d-c26c9af252f2-istio-podinfo\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz\" (UID: \"408aa757-573f-4cd4-b81d-c26c9af252f2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" Nov 01 16:01:44.487942 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.487836 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408aa757-573f-4cd4-b81d-c26c9af252f2-workload-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz\" (UID: \"408aa757-573f-4cd4-b81d-c26c9af252f2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" Nov 01 16:01:44.487942 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.487877 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2387a8-1f76-4653-9047-0f5824e6cfbb-workload-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4\" (UID: \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" Nov 01 16:01:44.488105 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.487939 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05ec5955-5241-423b-8161-1ac7608f4fed-istio-token\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv\" (UID: \"05ec5955-5241-423b-8161-1ac7608f4fed\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" Nov 01 16:01:44.488105 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.487966 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05ec5955-5241-423b-8161-1ac7608f4fed-workload-certs\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv\" (UID: \"05ec5955-5241-423b-8161-1ac7608f4fed\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" Nov 01 16:01:44.488105 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.487988 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef2387a8-1f76-4653-9047-0f5824e6cfbb-istio-envoy\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4\" (UID: \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" Nov 01 16:01:44.488415 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.488389 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408aa757-573f-4cd4-b81d-c26c9af252f2-credential-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz\" (UID: \"408aa757-573f-4cd4-b81d-c26c9af252f2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" Nov 01 16:01:44.488586 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.488516 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408aa757-573f-4cd4-b81d-c26c9af252f2-workload-certs\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz\" (UID: \"408aa757-573f-4cd4-b81d-c26c9af252f2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" Nov 01 16:01:44.488669 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.488648 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05ec5955-5241-423b-8161-1ac7608f4fed-istio-data\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv\" (UID: \"05ec5955-5241-423b-8161-1ac7608f4fed\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" Nov 01 16:01:44.488728 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.488676 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05ec5955-5241-423b-8161-1ac7608f4fed-istiod-ca-cert\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv\" (UID: \"05ec5955-5241-423b-8161-1ac7608f4fed\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" Nov 01 16:01:44.488728 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.488709 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ec5955-5241-423b-8161-1ac7608f4fed-workload-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv\" (UID: \"05ec5955-5241-423b-8161-1ac7608f4fed\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" Nov 01 16:01:44.489151 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.489127 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408aa757-573f-4cd4-b81d-c26c9af252f2-workload-certs\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz\" (UID: \"408aa757-573f-4cd4-b81d-c26c9af252f2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" Nov 01 16:01:44.489287 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.489229 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ec5955-5241-423b-8161-1ac7608f4fed-workload-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv\" (UID: \"05ec5955-5241-423b-8161-1ac7608f4fed\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" Nov 01 16:01:44.489409 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.489383 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05ec5955-5241-423b-8161-1ac7608f4fed-istiod-ca-cert\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv\" (UID: \"05ec5955-5241-423b-8161-1ac7608f4fed\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" Nov 01 16:01:44.489474 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.489435 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408aa757-573f-4cd4-b81d-c26c9af252f2-workload-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz\" (UID: \"408aa757-573f-4cd4-b81d-c26c9af252f2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" Nov 01 16:01:44.489754 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.489704 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05ec5955-5241-423b-8161-1ac7608f4fed-istio-data\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv\" (UID: \"05ec5955-5241-423b-8161-1ac7608f4fed\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" Nov 01 16:01:44.490065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.488741 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef2387a8-1f76-4653-9047-0f5824e6cfbb-istio-podinfo\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4\" (UID: \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" Nov 01 16:01:44.490151 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.490115 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408aa757-573f-4cd4-b81d-c26c9af252f2-istio-data\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz\" (UID: \"408aa757-573f-4cd4-b81d-c26c9af252f2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" Nov 01 16:01:44.490215 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.490172 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05ec5955-5241-423b-8161-1ac7608f4fed-istio-envoy\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv\" (UID: \"05ec5955-5241-423b-8161-1ac7608f4fed\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" Nov 01 16:01:44.490215 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.490198 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l525n\" (UniqueName: \"kubernetes.io/projected/05ec5955-5241-423b-8161-1ac7608f4fed-kube-api-access-l525n\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv\" (UID: \"05ec5955-5241-423b-8161-1ac7608f4fed\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" Nov 01 16:01:44.490321 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.490225 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8rzn\" (UniqueName: \"kubernetes.io/projected/408aa757-573f-4cd4-b81d-c26c9af252f2-kube-api-access-q8rzn\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz\" (UID: \"408aa757-573f-4cd4-b81d-c26c9af252f2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" Nov 01 16:01:44.490321 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.490244 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408aa757-573f-4cd4-b81d-c26c9af252f2-istio-envoy\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz\" (UID: \"408aa757-573f-4cd4-b81d-c26c9af252f2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" Nov 01 16:01:44.490453 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.490432 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408aa757-573f-4cd4-b81d-c26c9af252f2-istio-data\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz\" (UID: \"408aa757-573f-4cd4-b81d-c26c9af252f2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" Nov 01 16:01:44.490692 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.490667 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408aa757-573f-4cd4-b81d-c26c9af252f2-istiod-ca-cert\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz\" (UID: \"408aa757-573f-4cd4-b81d-c26c9af252f2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" Nov 01 16:01:44.490763 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.490729 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef2387a8-1f76-4653-9047-0f5824e6cfbb-istio-token\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4\" (UID: \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" Nov 01 16:01:44.490818 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.490790 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef2387a8-1f76-4653-9047-0f5824e6cfbb-istio-data\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4\" (UID: \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" Nov 01 16:01:44.490871 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.490816 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2l2k\" (UniqueName: \"kubernetes.io/projected/ef2387a8-1f76-4653-9047-0f5824e6cfbb-kube-api-access-z2l2k\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4\" (UID: \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" Nov 01 16:01:44.491366 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.491315 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05ec5955-5241-423b-8161-1ac7608f4fed-istio-podinfo\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv\" (UID: \"05ec5955-5241-423b-8161-1ac7608f4fed\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" Nov 01 16:01:44.492258 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.492232 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408aa757-573f-4cd4-b81d-c26c9af252f2-istio-podinfo\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz\" (UID: \"408aa757-573f-4cd4-b81d-c26c9af252f2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" Nov 01 16:01:44.492628 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.492607 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408aa757-573f-4cd4-b81d-c26c9af252f2-istiod-ca-cert\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz\" (UID: \"408aa757-573f-4cd4-b81d-c26c9af252f2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" Nov 01 16:01:44.492836 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.492817 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05ec5955-5241-423b-8161-1ac7608f4fed-istio-envoy\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv\" (UID: \"05ec5955-5241-423b-8161-1ac7608f4fed\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" Nov 01 16:01:44.493966 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.493944 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408aa757-573f-4cd4-b81d-c26c9af252f2-istio-envoy\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz\" (UID: \"408aa757-573f-4cd4-b81d-c26c9af252f2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" Nov 01 16:01:44.496072 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.496046 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408aa757-573f-4cd4-b81d-c26c9af252f2-istio-token\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz\" (UID: \"408aa757-573f-4cd4-b81d-c26c9af252f2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" Nov 01 16:01:44.496367 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.496320 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05ec5955-5241-423b-8161-1ac7608f4fed-istio-token\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv\" (UID: \"05ec5955-5241-423b-8161-1ac7608f4fed\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" Nov 01 16:01:44.497575 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.497550 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8rzn\" (UniqueName: \"kubernetes.io/projected/408aa757-573f-4cd4-b81d-c26c9af252f2-kube-api-access-q8rzn\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz\" (UID: \"408aa757-573f-4cd4-b81d-c26c9af252f2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" Nov 01 16:01:44.498652 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.498612 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l525n\" (UniqueName: \"kubernetes.io/projected/05ec5955-5241-423b-8161-1ac7608f4fed-kube-api-access-l525n\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv\" (UID: \"05ec5955-5241-423b-8161-1ac7608f4fed\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" Nov 01 16:01:44.592079 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.591986 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2387a8-1f76-4653-9047-0f5824e6cfbb-workload-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4\" (UID: \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" Nov 01 16:01:44.592248 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.592077 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef2387a8-1f76-4653-9047-0f5824e6cfbb-istio-envoy\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4\" (UID: \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" Nov 01 16:01:44.592248 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.592144 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef2387a8-1f76-4653-9047-0f5824e6cfbb-istio-podinfo\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4\" (UID: \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" Nov 01 16:01:44.592372 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.592270 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef2387a8-1f76-4653-9047-0f5824e6cfbb-istio-token\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4\" (UID: \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" Nov 01 16:01:44.592372 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.592319 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef2387a8-1f76-4653-9047-0f5824e6cfbb-istio-data\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4\" (UID: \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" Nov 01 16:01:44.592372 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.592345 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2l2k\" (UniqueName: \"kubernetes.io/projected/ef2387a8-1f76-4653-9047-0f5824e6cfbb-kube-api-access-z2l2k\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4\" (UID: \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" Nov 01 16:01:44.592556 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.592416 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef2387a8-1f76-4653-9047-0f5824e6cfbb-istiod-ca-cert\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4\" (UID: \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" Nov 01 16:01:44.592556 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.592447 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2387a8-1f76-4653-9047-0f5824e6cfbb-workload-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4\" (UID: \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" Nov 01 16:01:44.592556 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.592484 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef2387a8-1f76-4653-9047-0f5824e6cfbb-workload-certs\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4\" (UID: \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" Nov 01 16:01:44.592556 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.592512 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2387a8-1f76-4653-9047-0f5824e6cfbb-credential-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4\" (UID: \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" Nov 01 16:01:44.592781 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.592734 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef2387a8-1f76-4653-9047-0f5824e6cfbb-istio-data\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4\" (UID: \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" Nov 01 16:01:44.592914 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.592890 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2387a8-1f76-4653-9047-0f5824e6cfbb-credential-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4\" (UID: \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" Nov 01 16:01:44.593009 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.592989 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef2387a8-1f76-4653-9047-0f5824e6cfbb-workload-certs\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4\" (UID: \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" Nov 01 16:01:44.593364 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.593333 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef2387a8-1f76-4653-9047-0f5824e6cfbb-istiod-ca-cert\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4\" (UID: \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" Nov 01 16:01:44.594972 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.594952 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef2387a8-1f76-4653-9047-0f5824e6cfbb-istio-envoy\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4\" (UID: \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" Nov 01 16:01:44.595342 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.595316 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef2387a8-1f76-4653-9047-0f5824e6cfbb-istio-podinfo\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4\" (UID: \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" Nov 01 16:01:44.600316 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.600282 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef2387a8-1f76-4653-9047-0f5824e6cfbb-istio-token\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4\" (UID: \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" Nov 01 16:01:44.600424 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.600348 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2l2k\" (UniqueName: \"kubernetes.io/projected/ef2387a8-1f76-4653-9047-0f5824e6cfbb-kube-api-access-z2l2k\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4\" (UID: \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" Nov 01 16:01:44.662396 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.662370 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" Nov 01 16:01:44.685232 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.685200 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" Nov 01 16:01:44.696169 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.696143 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" Nov 01 16:01:44.736260 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.736224 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" Nov 01 16:01:44.772137 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.771997 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" Nov 01 16:01:44.975978 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:01:44.975947 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b704396_3411_40af_95fe_4b67a41ae69f.slice/crio-a6c54014bf0b4bc8ddcc1664fc98471ff051093f426b675fe65d7d81dd51655b WatchSource:0}: Error finding container a6c54014bf0b4bc8ddcc1664fc98471ff051093f426b675fe65d7d81dd51655b: Status 404 returned error can't find the container with id a6c54014bf0b4bc8ddcc1664fc98471ff051093f426b675fe65d7d81dd51655b Nov 01 16:01:44.975978 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.975960 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8"] Nov 01 16:01:44.979405 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.979365 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:44.979513 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.979447 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:44.979513 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:44.979489 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:45.232045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:45.232000 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv"] Nov 01 16:01:45.234463 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:01:45.234432 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05ec5955_5241_423b_8161_1ac7608f4fed.slice/crio-6993cd40ba81d6c97fe770becac6e02a93d9df092fdb83388d08ec41d6c3da20 WatchSource:0}: Error finding container 6993cd40ba81d6c97fe770becac6e02a93d9df092fdb83388d08ec41d6c3da20: Status 404 returned error can't find the container with id 6993cd40ba81d6c97fe770becac6e02a93d9df092fdb83388d08ec41d6c3da20 Nov 01 16:01:45.236843 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:45.236809 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:45.236965 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:45.236884 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:45.236965 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:45.236929 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:45.245124 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:45.245105 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79"] Nov 01 16:01:45.251621 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:45.251589 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:45.251693 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:45.251670 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:45.251731 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:45.251712 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:45.312610 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:01:45.312544 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef2387a8_1f76_4653_9047_0f5824e6cfbb.slice/crio-3383aee1e0a81e8efd76a3f026fbbfadd19794f80feb4d65f8b418429a0bd7d8 WatchSource:0}: Error finding container 3383aee1e0a81e8efd76a3f026fbbfadd19794f80feb4d65f8b418429a0bd7d8: Status 404 returned error can't find the container with id 3383aee1e0a81e8efd76a3f026fbbfadd19794f80feb4d65f8b418429a0bd7d8 Nov 01 16:01:45.316851 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:45.316805 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:45.316959 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:45.316901 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:45.316959 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:45.316949 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:45.319177 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:45.318063 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4"] Nov 01 16:01:45.378313 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:45.378102 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz"] Nov 01 16:01:45.391158 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:45.390471 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:45.391158 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:45.390558 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:45.391158 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:45.390600 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:45.500677 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:45.500647 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" event={"ID":"edcab30a-6e4e-4019-9fd7-96fe7b085711","Type":"ContainerStarted","Data":"15bfb16861b677eac440a82beb7cd7f2a9964e25b9351c73f69ca970eeb0ebed"} Nov 01 16:01:45.500849 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:45.500704 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" event={"ID":"edcab30a-6e4e-4019-9fd7-96fe7b085711","Type":"ContainerStarted","Data":"bee7486032ea1fa6bfaa1e4dc638577ac6e968081f97c17456978514d1989d0d"} Nov 01 16:01:45.503815 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:45.503394 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" event={"ID":"05ec5955-5241-423b-8161-1ac7608f4fed","Type":"ContainerStarted","Data":"c3f9604d2929249e6d9581fe956ae902f56c348867bcbc7a8dc1422d707df3b4"} Nov 01 16:01:45.503815 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:45.503452 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" event={"ID":"05ec5955-5241-423b-8161-1ac7608f4fed","Type":"ContainerStarted","Data":"6993cd40ba81d6c97fe770becac6e02a93d9df092fdb83388d08ec41d6c3da20"} Nov 01 16:01:45.508614 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:45.508504 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" event={"ID":"ef2387a8-1f76-4653-9047-0f5824e6cfbb","Type":"ContainerStarted","Data":"3383aee1e0a81e8efd76a3f026fbbfadd19794f80feb4d65f8b418429a0bd7d8"} Nov 01 16:01:45.518188 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:45.518080 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" event={"ID":"3b704396-3411-40af-95fe-4b67a41ae69f","Type":"ContainerStarted","Data":"6d837305a83abbd54c37311b0b33ff057014a6ad06193106ea3fe7f41ed5e19f"} Nov 01 16:01:45.518188 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:45.518138 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" event={"ID":"3b704396-3411-40af-95fe-4b67a41ae69f","Type":"ContainerStarted","Data":"a6c54014bf0b4bc8ddcc1664fc98471ff051093f426b675fe65d7d81dd51655b"} Nov 01 16:01:45.523979 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:45.522710 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" podStartSLOduration=1.5226929340000002 podStartE2EDuration="1.522692934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:01:45.521315618 +0000 UTC m=+838.853053270" watchObservedRunningTime="2026-11-01 16:01:45.522692934 +0000 UTC m=+838.854430582" Nov 01 16:01:45.525849 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:45.525717 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" event={"ID":"408aa757-573f-4cd4-b81d-c26c9af252f2","Type":"ContainerStarted","Data":"0e472d5787e85111b689389b62c19eba08f683f4a4dc969356616ead202f0437"} Nov 01 16:01:45.549819 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:45.548658 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" podStartSLOduration=1.548639823 podStartE2EDuration="1.548639823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:01:45.543986324 +0000 UTC m=+838.875723974" watchObservedRunningTime="2026-11-01 16:01:45.548639823 +0000 UTC m=+838.880377475" Nov 01 16:01:45.563045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:45.562399 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" podStartSLOduration=1.562380047 podStartE2EDuration="1.562380047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:01:45.561611617 +0000 UTC m=+838.893349269" watchObservedRunningTime="2026-11-01 16:01:45.562380047 +0000 UTC m=+838.894117697" Nov 01 16:01:45.665512 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:45.665093 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" Nov 01 16:01:45.685584 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:45.685437 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" Nov 01 16:01:45.696720 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:45.696692 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" Nov 01 16:01:46.597075 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:46.597035 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" event={"ID":"408aa757-573f-4cd4-b81d-c26c9af252f2","Type":"ContainerStarted","Data":"becc68c69d5083a9e078509dcb7c90251f33191fb317ea61e3a59037fa7cee65"} Nov 01 16:01:46.617041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:46.612052 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" event={"ID":"ef2387a8-1f76-4653-9047-0f5824e6cfbb","Type":"ContainerStarted","Data":"5b3486d4b53f8d69654824e4e3d3aa10bf2fbba2dab07c4b14a96e26c66ef5a0"} Nov 01 16:01:46.700043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:46.696533 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 16:01:46.700043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:46.696611 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" podUID="3b704396-3411-40af-95fe-4b67a41ae69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 16:01:46.706085 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:46.706050 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 01 16:01:46.706413 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:46.706378 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" podUID="05ec5955-5241-423b-8161-1ac7608f4fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 01 16:01:46.706715 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:46.706680 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 01 16:01:46.706920 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:46.706890 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" podUID="edcab30a-6e4e-4019-9fd7-96fe7b085711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 01 16:01:46.976101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:46.974302 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" podStartSLOduration=2.9742791889999998 podStartE2EDuration="2.974279189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:01:46.930546064 +0000 UTC m=+840.262283716" watchObservedRunningTime="2026-11-01 16:01:46.974279189 +0000 UTC m=+840.306016840" Nov 01 16:01:47.208192 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:47.080922 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" podStartSLOduration=3.080899546 podStartE2EDuration="3.080899546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:01:46.666997702 +0000 UTC m=+839.998735359" watchObservedRunningTime="2026-11-01 16:01:47.080899546 +0000 UTC m=+840.412637197" Nov 01 16:01:47.291305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:47.124069 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 16:01:47.345381 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:47.345327 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" podUID="ef2387a8-1f76-4653-9047-0f5824e6cfbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 16:01:47.396042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:47.384632 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" Nov 01 16:01:47.459968 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:47.456641 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 16:01:47.459968 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:47.456701 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" podUID="408aa757-573f-4cd4-b81d-c26c9af252f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 16:01:47.571205 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:47.570758 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" Nov 01 16:01:47.672196 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:47.672153 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 16:01:47.672878 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:47.672838 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" podUID="3b704396-3411-40af-95fe-4b67a41ae69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 16:01:47.685958 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:47.685923 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 01 16:01:47.686217 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:47.686188 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" podUID="05ec5955-5241-423b-8161-1ac7608f4fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 01 16:01:47.697068 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:47.697010 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 01 16:01:47.697273 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:47.697247 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" podUID="edcab30a-6e4e-4019-9fd7-96fe7b085711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 01 16:01:47.739070 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:47.737112 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 16:01:47.739070 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:47.737167 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" podUID="ef2387a8-1f76-4653-9047-0f5824e6cfbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 16:01:47.774544 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:47.774509 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 16:01:47.794451 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:47.794402 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" podUID="408aa757-573f-4cd4-b81d-c26c9af252f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 16:01:48.591410 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:48.591369 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:01:48.618070 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:48.615831 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:01:48.618070 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:48.616238 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:01:48.665187 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:48.665127 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:48.666645 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:48.666616 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 16:01:48.666863 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:48.666838 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" podUID="3b704396-3411-40af-95fe-4b67a41ae69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 16:01:48.741454 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:48.741415 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:48.751776 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:48.751745 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 01 16:01:48.752010 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:48.751983 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" podUID="edcab30a-6e4e-4019-9fd7-96fe7b085711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 01 16:01:48.780041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:48.774646 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 16:01:48.780041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:48.774704 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" podUID="408aa757-573f-4cd4-b81d-c26c9af252f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 16:01:48.802758 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:48.787591 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 16:01:48.802758 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:48.787646 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" podUID="ef2387a8-1f76-4653-9047-0f5824e6cfbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 16:01:48.802758 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:48.787728 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 01 16:01:48.802758 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:48.787752 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" podUID="05ec5955-5241-423b-8161-1ac7608f4fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 01 16:01:49.281485 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:48.966767 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:01:49.619046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:49.597300 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:01:49.632068 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:49.631646 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 16:01:49.675042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:49.641137 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:49.728046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:49.641324 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:49.894099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:49.695123 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:49.894099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:49.695166 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 16:01:49.894099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:49.726752 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:01:49.894099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:49.890171 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" podUID="3b704396-3411-40af-95fe-4b67a41ae69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 16:01:49.894099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:49.822530 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:49.894099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:49.727439 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 01 16:01:49.894099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:49.890263 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" podUID="05ec5955-5241-423b-8161-1ac7608f4fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 01 16:01:49.894099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:49.890278 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:01:49.894099 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:01:49.819330 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:01:49.894099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:49.890407 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 16:01:49.894099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:49.890455 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:01:50.016043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:50.015514 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 01 16:01:50.016043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:50.015572 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" podUID="edcab30a-6e4e-4019-9fd7-96fe7b085711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 01 16:01:50.228218 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:50.228123 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 16:01:50.228484 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:50.228449 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" podUID="ef2387a8-1f76-4653-9047-0f5824e6cfbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 16:01:50.230045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:50.228952 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 16:01:50.234107 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:50.234074 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" podUID="408aa757-573f-4cd4-b81d-c26c9af252f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 16:01:50.325042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:50.303356 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:01:50.386408 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:50.386361 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:01:50.461734 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:50.461684 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": context deadline exceeded" Nov 01 16:01:50.619436 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:50.619392 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:01:50.732466 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:50.732418 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:50.762361 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:50.762306 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 01 16:01:50.762602 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:50.762575 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" podUID="05ec5955-5241-423b-8161-1ac7608f4fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 01 16:01:50.762809 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:50.762791 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 16:01:50.762926 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:50.762908 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" podUID="3b704396-3411-40af-95fe-4b67a41ae69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 16:01:50.900510 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:50.892092 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:01:50.900510 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:50.892423 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:01:50.943191 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:50.943144 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:51.001044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:50.983127 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 16:01:51.001044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:50.983184 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" podUID="ef2387a8-1f76-4653-9047-0f5824e6cfbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 16:01:51.001044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:50.983311 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 16:01:51.001044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:50.983334 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" podUID="408aa757-573f-4cd4-b81d-c26c9af252f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 16:01:51.082863 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:51.043869 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 01 16:01:51.082863 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:51.043946 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" podUID="edcab30a-6e4e-4019-9fd7-96fe7b085711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 01 16:01:51.338424 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:51.338274 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:51.393598 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:51.393544 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": context deadline exceeded" Nov 01 16:01:51.671425 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:51.671387 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 16:01:51.671699 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:51.671664 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" podUID="3b704396-3411-40af-95fe-4b67a41ae69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 16:01:51.675738 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:51.675702 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:51.752349 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:51.752287 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:01:51.789156 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:51.789105 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 01 16:01:51.789475 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:51.789395 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" podUID="05ec5955-5241-423b-8161-1ac7608f4fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 01 16:01:51.791294 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:51.791268 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 16:01:51.791458 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:51.791434 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" podUID="ef2387a8-1f76-4653-9047-0f5824e6cfbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 16:01:51.791849 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:51.791827 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 01 16:01:51.792006 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:51.791986 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" podUID="edcab30a-6e4e-4019-9fd7-96fe7b085711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 01 16:01:51.792499 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:51.792467 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 16:01:51.792676 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:51.792657 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" podUID="408aa757-573f-4cd4-b81d-c26c9af252f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 16:01:51.897246 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:51.897190 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:01:51.901150 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:51.901102 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:51.972083 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:01:51.962413 4363 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.153s" Nov 01 16:01:52.001892 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:52.001852 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:52.357097 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:52.351423 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:52.700515 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:52.700408 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:52.717040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:52.701537 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 16:01:52.717040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:52.702874 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" podUID="3b704396-3411-40af-95fe-4b67a41ae69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 16:01:52.717040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:52.702226 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 01 16:01:52.717040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:52.702928 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" podUID="05ec5955-5241-423b-8161-1ac7608f4fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 01 16:01:52.717040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:52.702291 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 01 16:01:52.717040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:52.702967 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" podUID="edcab30a-6e4e-4019-9fd7-96fe7b085711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 01 16:01:52.799072 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:52.798199 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 16:01:52.799072 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:52.798260 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" podUID="408aa757-573f-4cd4-b81d-c26c9af252f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 16:01:52.818573 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:52.817807 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:52.830922 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:52.830890 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 16:01:52.831218 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:52.831182 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" podUID="ef2387a8-1f76-4653-9047-0f5824e6cfbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 16:01:52.916071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:52.915112 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:52.916071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:52.915488 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:53.125123 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:53.125074 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:53.393132 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:53.365102 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:01:53.670212 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:53.670116 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 16:01:53.670466 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:53.670421 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" podUID="3b704396-3411-40af-95fe-4b67a41ae69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 16:01:53.741305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:53.741261 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:01:53.745297 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:53.745255 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 16:01:53.771592 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:53.771258 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 01 16:01:53.771592 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:53.771309 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" podUID="edcab30a-6e4e-4019-9fd7-96fe7b085711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 01 16:01:53.771592 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:53.771383 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 01 16:01:53.771592 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:53.771406 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" podUID="05ec5955-5241-423b-8161-1ac7608f4fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 01 16:01:53.772811 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:53.772262 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 16:01:53.772811 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:53.772302 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" podUID="408aa757-573f-4cd4-b81d-c26c9af252f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 16:01:53.788280 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:53.770169 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" podUID="ef2387a8-1f76-4653-9047-0f5824e6cfbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 16:01:53.828214 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:53.828166 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:53.925766 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:53.925659 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:53.927191 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:53.925992 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:54.135298 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:54.134151 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:54.253691 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:54.253307 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:54.392052 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:54.390153 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:54.696042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:54.695904 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 16:01:54.696042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:54.695981 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" podUID="3b704396-3411-40af-95fe-4b67a41ae69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 16:01:54.721045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:54.711602 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 01 16:01:54.721045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:54.711662 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" podUID="05ec5955-5241-423b-8161-1ac7608f4fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 01 16:01:54.721045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:54.711741 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 01 16:01:54.721045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:54.711764 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" podUID="edcab30a-6e4e-4019-9fd7-96fe7b085711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 01 16:01:54.737006 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:54.736889 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 16:01:54.737006 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:54.736946 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" podUID="ef2387a8-1f76-4653-9047-0f5824e6cfbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 16:01:54.797042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:54.779113 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:01:54.797042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:54.782701 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 16:01:54.797042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:54.782749 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" podUID="408aa757-573f-4cd4-b81d-c26c9af252f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 16:01:54.853096 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:54.840648 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:54.959101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:54.935626 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:54.959101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:54.935962 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": context deadline exceeded" Nov 01 16:01:54.959101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:54.936047 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:54.959101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:54.949313 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:55.138078 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:55.138007 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:55.392046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:55.391977 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:55.690893 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:55.690526 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 16:01:55.690893 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:55.690593 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" podUID="3b704396-3411-40af-95fe-4b67a41ae69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 16:01:55.695053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:55.691960 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 01 16:01:55.695053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:55.692003 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" podUID="05ec5955-5241-423b-8161-1ac7608f4fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 01 16:01:55.716036 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:55.715906 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 01 16:01:55.716036 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:55.715966 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" podUID="edcab30a-6e4e-4019-9fd7-96fe7b085711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 01 16:01:55.759769 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:55.759722 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 16:01:55.760080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:55.760047 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" podUID="ef2387a8-1f76-4653-9047-0f5824e6cfbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 16:01:55.804429 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:55.802339 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:01:55.804429 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:55.804324 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 16:01:55.804429 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:55.804370 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" podUID="408aa757-573f-4cd4-b81d-c26c9af252f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 16:01:55.849043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:55.845788 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:55.948166 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:55.947673 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:01:55.948166 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:55.947981 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:01:56.144054 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:56.143180 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:56.407297 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:56.404135 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:56.603662 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:56.600881 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": context deadline exceeded" Nov 01 16:01:56.718268 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:56.718174 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 16:01:56.718540 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:56.718500 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" podUID="3b704396-3411-40af-95fe-4b67a41ae69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 16:01:56.867532 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:56.867485 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:01:56.871056 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:56.871007 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 16:01:56.898157 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:56.898113 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" podUID="ef2387a8-1f76-4653-9047-0f5824e6cfbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 16:01:56.898777 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:56.879447 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:56.979105 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:56.959485 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:56.989288 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:56.975161 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:01:56.999067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:56.975925 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 16:01:56.999349 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:56.999319 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" podUID="408aa757-573f-4cd4-b81d-c26c9af252f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 16:01:57.057079 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:57.056991 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 01 16:01:57.057373 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:57.057342 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" podUID="edcab30a-6e4e-4019-9fd7-96fe7b085711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 01 16:01:57.139855 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:57.113945 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 01 16:01:57.139855 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:57.114046 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" podUID="05ec5955-5241-423b-8161-1ac7608f4fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 01 16:01:57.167043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:57.143355 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 01 16:01:57.237130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:57.204076 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:01:57.407238 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:57.407185 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:57.746043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:57.704144 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 16:01:57.746043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:57.704220 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" podUID="3b704396-3411-40af-95fe-4b67a41ae69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 16:01:57.783171 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:57.758565 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 01 16:01:57.783171 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:57.758648 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" podUID="05ec5955-5241-423b-8161-1ac7608f4fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 01 16:01:57.818312 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:57.813647 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 16:01:57.818312 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:57.813714 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" podUID="ef2387a8-1f76-4653-9047-0f5824e6cfbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 16:01:57.855819 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:57.855784 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 16:01:57.856143 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:57.856096 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" podUID="408aa757-573f-4cd4-b81d-c26c9af252f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 16:01:57.937609 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:57.890704 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 01 16:01:57.937931 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:57.937893 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" podUID="edcab30a-6e4e-4019-9fd7-96fe7b085711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 01 16:01:57.938117 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:57.892040 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:01:57.938536 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:57.908197 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:01:57.938701 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:57.938678 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" podUID="4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:57.939928 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:57.939901 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:58.009124 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:58.009002 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:58.181319 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:58.181245 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 01 16:01:58.631042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:58.606969 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:01:58.791663 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:58.786156 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 01 16:01:58.791663 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:58.786228 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" podUID="edcab30a-6e4e-4019-9fd7-96fe7b085711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 01 16:01:58.820633 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:58.820595 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 16:01:58.820903 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:58.820868 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" podUID="3b704396-3411-40af-95fe-4b67a41ae69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 16:01:58.825650 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:58.825609 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 16:01:58.833475 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:58.833422 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" podUID="408aa757-573f-4cd4-b81d-c26c9af252f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 16:01:58.833709 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:58.828654 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 16:01:58.833869 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:58.833844 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" podUID="ef2387a8-1f76-4653-9047-0f5824e6cfbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 16:01:58.834459 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:58.834438 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 01 16:01:58.834628 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:58.834607 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" podUID="05ec5955-5241-423b-8161-1ac7608f4fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 01 16:01:58.931182 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:58.926820 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:58.983274 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:58.943688 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:58.983274 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:58.943723 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:01:59.356103 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:59.347938 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:59.356103 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:59.349582 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:01:59.448076 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:59.420803 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:01:59.448076 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:59.420861 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" podUID="9427e75b-2800-4bee-8711-b1e1306d8f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:59.691549 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:59.672911 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" Nov 01 16:01:59.730882 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:59.726989 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" Nov 01 16:01:59.730882 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:59.727108 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" Nov 01 16:01:59.732199 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:59.681074 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:59.736153 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:59.722791 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 01 16:01:59.745928 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:59.745887 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" podUID="edcab30a-6e4e-4019-9fd7-96fe7b085711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 01 16:01:59.746220 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:59.739820 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" Nov 01 16:01:59.759088 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:59.723594 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 01 16:01:59.779179 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:59.779138 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" podUID="05ec5955-5241-423b-8161-1ac7608f4fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 01 16:01:59.779823 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:59.775354 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" Nov 01 16:01:59.789177 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:59.725548 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 16:01:59.835518 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:59.835482 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" podUID="3b704396-3411-40af-95fe-4b67a41ae69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 16:01:59.840068 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:59.774137 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 16:01:59.840325 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:59.840297 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" podUID="408aa757-573f-4cd4-b81d-c26c9af252f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 16:01:59.840445 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:59.794141 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 16:01:59.840566 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:59.840546 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" podUID="ef2387a8-1f76-4653-9047-0f5824e6cfbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 16:01:59.994299 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:59.994185 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:01:59.995727 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:01:59.995701 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:02:00.023157 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:00.023106 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:00.080042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:00.023397 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:00.364052 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:00.351298 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:00.379047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:00.371660 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:00.446586 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:00.428214 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:00.598051 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:00.588541 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:02:00.739377 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:00.737793 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 16:02:00.739377 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:00.737860 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" podUID="3b704396-3411-40af-95fe-4b67a41ae69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 16:02:00.752407 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:00.752364 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 01 16:02:00.752652 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:00.752618 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" podUID="edcab30a-6e4e-4019-9fd7-96fe7b085711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 01 16:02:00.753491 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:00.753366 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 01 16:02:00.756182 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:00.756152 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" podUID="05ec5955-5241-423b-8161-1ac7608f4fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 01 16:02:00.786054 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:00.758227 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 16:02:00.870040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:00.868096 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" podUID="ef2387a8-1f76-4653-9047-0f5824e6cfbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 16:02:00.870040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:00.789133 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:02:00.870040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:00.787442 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 16:02:00.875041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:00.790177 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 16:02:00.875041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:00.871263 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" podUID="408aa757-573f-4cd4-b81d-c26c9af252f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 16:02:01.129389 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:01.112981 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:01.129968 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:01.129927 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:02:01.433093 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:01.411185 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:01.447822 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:01.447775 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:02:01.610932 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:01.608140 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:01.667147 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:01.667112 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 16:02:01.667375 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:01.667347 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" podUID="3b704396-3411-40af-95fe-4b67a41ae69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 16:02:01.743099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:01.740257 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 16:02:01.743099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:01.740318 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" podUID="ef2387a8-1f76-4653-9047-0f5824e6cfbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 16:02:01.743099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:01.740660 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 01 16:02:01.743099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:01.740693 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" podUID="edcab30a-6e4e-4019-9fd7-96fe7b085711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 01 16:02:01.743099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:01.741084 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 01 16:02:01.743099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:01.741131 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" podUID="05ec5955-5241-423b-8161-1ac7608f4fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 01 16:02:01.800048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:01.798192 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 16:02:01.800048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:01.798255 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" podUID="408aa757-573f-4cd4-b81d-c26c9af252f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 16:02:01.819092 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:01.809106 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:02:02.120457 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:02.120406 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:02:02.132598 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:02.132556 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:02:02.138433 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:02.138390 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 01 16:02:02.484094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:02.482311 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:02.555305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:02.555278 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:02:02.675425 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:02.675387 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 16:02:02.675670 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:02.675639 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" podUID="3b704396-3411-40af-95fe-4b67a41ae69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 16:02:02.742095 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:02.717209 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 01 16:02:02.742373 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:02.742335 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" podUID="edcab30a-6e4e-4019-9fd7-96fe7b085711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 01 16:02:02.769040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:02.743758 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 16:02:02.846143 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:02.846095 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" podUID="ef2387a8-1f76-4653-9047-0f5824e6cfbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 16:02:02.846408 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:02.772304 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 16:02:02.846547 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:02.846526 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" podUID="408aa757-573f-4cd4-b81d-c26c9af252f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 16:02:02.846652 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:02.784404 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 01 16:02:02.871373 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:02.871338 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" podUID="05ec5955-5241-423b-8161-1ac7608f4fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 01 16:02:03.132284 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:03.130988 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:03.144064 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:03.136760 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:03.712043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:03.663811 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 16:02:03.712043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:03.663882 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" podUID="3b704396-3411-40af-95fe-4b67a41ae69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 16:02:03.712043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:03.707797 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 01 16:02:03.712043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:03.707880 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" podUID="05ec5955-5241-423b-8161-1ac7608f4fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 01 16:02:03.712043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:03.708772 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 01 16:02:03.712043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:03.708816 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" podUID="edcab30a-6e4e-4019-9fd7-96fe7b085711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 01 16:02:03.744310 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:03.744273 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 16:02:03.744601 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:03.744566 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" podUID="ef2387a8-1f76-4653-9047-0f5824e6cfbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 16:02:03.784657 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:03.784223 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 16:02:03.784657 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:03.784294 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" podUID="408aa757-573f-4cd4-b81d-c26c9af252f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 16:02:03.902046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:03.899818 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:02:04.132192 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:04.132135 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:04.160905 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:04.160862 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:04.268556 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:04.266695 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:04.291218 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:04.291171 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:04.303615 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:04.302570 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 16:02:04.772857 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:04.772433 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 16:02:04.772857 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:04.772498 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" podUID="408aa757-573f-4cd4-b81d-c26c9af252f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 16:02:04.938056 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:04.906869 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:02:04.938056 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:04.934333 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:02:04.961127 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:04.960740 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:05.184111 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:05.181861 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:02:05.394251 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:05.394210 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:02:05.602859 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:05.602824 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" Nov 01 16:02:05.613369 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:05.613338 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" Nov 01 16:02:05.651464 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:05.651433 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" Nov 01 16:02:05.663164 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:05.663125 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" Nov 01 16:02:05.716483 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:05.716447 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:02:05.721039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:05.716727 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" podUID="05ec5955-5241-423b-8161-1ac7608f4fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:05.747050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:05.743962 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:02:05.747050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:05.744045 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" podUID="ef2387a8-1f76-4653-9047-0f5824e6cfbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:06.368714 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:06.368680 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" Nov 01 16:02:06.372176 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:06.372145 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" Nov 01 16:02:06.726100 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:06.723184 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:02:06.726100 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:06.723248 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" podUID="05ec5955-5241-423b-8161-1ac7608f4fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:06.795041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:06.791853 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:02:06.817436 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:06.801974 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" podUID="ef2387a8-1f76-4653-9047-0f5824e6cfbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:07.213042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:07.212581 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" Nov 01 16:02:07.216392 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:07.215935 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" Nov 01 16:02:07.599232 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:07.598620 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" Nov 01 16:02:07.627040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:07.625355 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" Nov 01 16:02:13.459116 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.459079 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn"] Nov 01 16:02:13.465095 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.464977 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" Nov 01 16:02:13.477809 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.477782 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn"] Nov 01 16:02:13.482971 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.482945 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv"] Nov 01 16:02:13.497345 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.497322 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" Nov 01 16:02:13.508507 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.508484 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv"] Nov 01 16:02:13.553180 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.553145 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c10680c0-ac33-4914-931e-15c3728484bc-workload-certs\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn\" (UID: \"c10680c0-ac33-4914-931e-15c3728484bc\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" Nov 01 16:02:13.553381 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.553253 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c10680c0-ac33-4914-931e-15c3728484bc-istio-podinfo\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn\" (UID: \"c10680c0-ac33-4914-931e-15c3728484bc\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" Nov 01 16:02:13.553381 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.553305 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c10680c0-ac33-4914-931e-15c3728484bc-istio-token\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn\" (UID: \"c10680c0-ac33-4914-931e-15c3728484bc\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" Nov 01 16:02:13.553381 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.553332 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10680c0-ac33-4914-931e-15c3728484bc-credential-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn\" (UID: \"c10680c0-ac33-4914-931e-15c3728484bc\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" Nov 01 16:02:13.553595 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.553421 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c10680c0-ac33-4914-931e-15c3728484bc-istio-envoy\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn\" (UID: \"c10680c0-ac33-4914-931e-15c3728484bc\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" Nov 01 16:02:13.553595 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.553501 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10680c0-ac33-4914-931e-15c3728484bc-workload-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn\" (UID: \"c10680c0-ac33-4914-931e-15c3728484bc\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" Nov 01 16:02:13.553595 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.553541 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c10680c0-ac33-4914-931e-15c3728484bc-istio-data\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn\" (UID: \"c10680c0-ac33-4914-931e-15c3728484bc\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" Nov 01 16:02:13.553755 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.553629 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57klw\" (UniqueName: \"kubernetes.io/projected/c10680c0-ac33-4914-931e-15c3728484bc-kube-api-access-57klw\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn\" (UID: \"c10680c0-ac33-4914-931e-15c3728484bc\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" Nov 01 16:02:13.553755 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.553682 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c10680c0-ac33-4914-931e-15c3728484bc-istiod-ca-cert\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn\" (UID: \"c10680c0-ac33-4914-931e-15c3728484bc\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" Nov 01 16:02:13.655074 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.655037 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-istiod-ca-cert\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv\" (UID: \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" Nov 01 16:02:13.655257 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.655080 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hszw2\" (UniqueName: \"kubernetes.io/projected/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-kube-api-access-hszw2\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv\" (UID: \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" Nov 01 16:02:13.655257 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.655205 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57klw\" (UniqueName: \"kubernetes.io/projected/c10680c0-ac33-4914-931e-15c3728484bc-kube-api-access-57klw\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn\" (UID: \"c10680c0-ac33-4914-931e-15c3728484bc\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" Nov 01 16:02:13.655257 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.655253 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-istio-envoy\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv\" (UID: \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" Nov 01 16:02:13.655425 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.655291 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-istio-data\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv\" (UID: \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" Nov 01 16:02:13.655425 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.655311 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c10680c0-ac33-4914-931e-15c3728484bc-istiod-ca-cert\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn\" (UID: \"c10680c0-ac33-4914-931e-15c3728484bc\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" Nov 01 16:02:13.655528 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.655426 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c10680c0-ac33-4914-931e-15c3728484bc-workload-certs\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn\" (UID: \"c10680c0-ac33-4914-931e-15c3728484bc\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" Nov 01 16:02:13.655528 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.655506 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c10680c0-ac33-4914-931e-15c3728484bc-istio-podinfo\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn\" (UID: \"c10680c0-ac33-4914-931e-15c3728484bc\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" Nov 01 16:02:13.655647 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.655545 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-workload-certs\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv\" (UID: \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" Nov 01 16:02:13.655647 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.655564 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-credential-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv\" (UID: \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" Nov 01 16:02:13.655647 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.655588 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c10680c0-ac33-4914-931e-15c3728484bc-istio-token\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn\" (UID: \"c10680c0-ac33-4914-931e-15c3728484bc\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" Nov 01 16:02:13.655807 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.655732 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10680c0-ac33-4914-931e-15c3728484bc-credential-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn\" (UID: \"c10680c0-ac33-4914-931e-15c3728484bc\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" Nov 01 16:02:13.655807 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.655777 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-istio-podinfo\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv\" (UID: \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" Nov 01 16:02:13.655949 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.655826 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c10680c0-ac33-4914-931e-15c3728484bc-istio-envoy\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn\" (UID: \"c10680c0-ac33-4914-931e-15c3728484bc\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" Nov 01 16:02:13.655949 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.655898 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c10680c0-ac33-4914-931e-15c3728484bc-workload-certs\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn\" (UID: \"c10680c0-ac33-4914-931e-15c3728484bc\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" Nov 01 16:02:13.656190 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.655953 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10680c0-ac33-4914-931e-15c3728484bc-workload-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn\" (UID: \"c10680c0-ac33-4914-931e-15c3728484bc\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" Nov 01 16:02:13.656190 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.655983 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-istio-token\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv\" (UID: \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" Nov 01 16:02:13.656190 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.656082 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c10680c0-ac33-4914-931e-15c3728484bc-istio-data\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn\" (UID: \"c10680c0-ac33-4914-931e-15c3728484bc\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" Nov 01 16:02:13.656190 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.656132 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-workload-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv\" (UID: \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" Nov 01 16:02:13.656624 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.656602 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c10680c0-ac33-4914-931e-15c3728484bc-istio-data\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn\" (UID: \"c10680c0-ac33-4914-931e-15c3728484bc\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" Nov 01 16:02:13.656789 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.656568 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c10680c0-ac33-4914-931e-15c3728484bc-istiod-ca-cert\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn\" (UID: \"c10680c0-ac33-4914-931e-15c3728484bc\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" Nov 01 16:02:13.656789 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.656729 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10680c0-ac33-4914-931e-15c3728484bc-workload-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn\" (UID: \"c10680c0-ac33-4914-931e-15c3728484bc\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" Nov 01 16:02:13.656959 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.656857 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10680c0-ac33-4914-931e-15c3728484bc-credential-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn\" (UID: \"c10680c0-ac33-4914-931e-15c3728484bc\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" Nov 01 16:02:13.658403 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.658376 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c10680c0-ac33-4914-931e-15c3728484bc-istio-envoy\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn\" (UID: \"c10680c0-ac33-4914-931e-15c3728484bc\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" Nov 01 16:02:13.658588 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.658573 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c10680c0-ac33-4914-931e-15c3728484bc-istio-podinfo\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn\" (UID: \"c10680c0-ac33-4914-931e-15c3728484bc\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" Nov 01 16:02:13.670821 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.670789 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt"] Nov 01 16:02:13.674812 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.674786 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57klw\" (UniqueName: \"kubernetes.io/projected/c10680c0-ac33-4914-931e-15c3728484bc-kube-api-access-57klw\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn\" (UID: \"c10680c0-ac33-4914-931e-15c3728484bc\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" Nov 01 16:02:13.691071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.679686 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" Nov 01 16:02:13.702301 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.702268 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt"] Nov 01 16:02:13.713202 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.713141 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c10680c0-ac33-4914-931e-15c3728484bc-istio-token\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn\" (UID: \"c10680c0-ac33-4914-931e-15c3728484bc\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" Nov 01 16:02:13.757933 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.757897 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d1f840-a59c-4f46-84b0-d04c3cd1d500-workload-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt\" (UID: \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" Nov 01 16:02:13.758108 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.757959 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-workload-certs\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv\" (UID: \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" Nov 01 16:02:13.758108 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.758027 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-credential-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv\" (UID: \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" Nov 01 16:02:13.758108 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.758068 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-istio-podinfo\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv\" (UID: \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" Nov 01 16:02:13.758278 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.758117 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41d1f840-a59c-4f46-84b0-d04c3cd1d500-istio-token\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt\" (UID: \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" Nov 01 16:02:13.758278 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.758161 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41d1f840-a59c-4f46-84b0-d04c3cd1d500-istio-envoy\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt\" (UID: \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" Nov 01 16:02:13.758278 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.758182 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-istio-token\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv\" (UID: \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" Nov 01 16:02:13.758278 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.758210 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41d1f840-a59c-4f46-84b0-d04c3cd1d500-istio-data\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt\" (UID: \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" Nov 01 16:02:13.758278 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.758248 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-workload-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv\" (UID: \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" Nov 01 16:02:13.758540 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.758297 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-istiod-ca-cert\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv\" (UID: \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" Nov 01 16:02:13.758540 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.758317 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hszw2\" (UniqueName: \"kubernetes.io/projected/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-kube-api-access-hszw2\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv\" (UID: \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" Nov 01 16:02:13.758540 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.758372 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-istio-envoy\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv\" (UID: \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" Nov 01 16:02:13.758540 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.758385 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-workload-certs\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv\" (UID: \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" Nov 01 16:02:13.758540 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.758403 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d1f840-a59c-4f46-84b0-d04c3cd1d500-credential-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt\" (UID: \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" Nov 01 16:02:13.758540 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.758416 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-credential-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv\" (UID: \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" Nov 01 16:02:13.758540 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.758487 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-istio-data\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv\" (UID: \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" Nov 01 16:02:13.758905 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.758695 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-workload-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv\" (UID: \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" Nov 01 16:02:13.759217 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.759193 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41d1f840-a59c-4f46-84b0-d04c3cd1d500-workload-certs\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt\" (UID: \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" Nov 01 16:02:13.759399 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.759382 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41d1f840-a59c-4f46-84b0-d04c3cd1d500-istiod-ca-cert\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt\" (UID: \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" Nov 01 16:02:13.759599 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.759568 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41d1f840-a59c-4f46-84b0-d04c3cd1d500-istio-podinfo\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt\" (UID: \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" Nov 01 16:02:13.759715 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.759310 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-istiod-ca-cert\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv\" (UID: \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" Nov 01 16:02:13.759715 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.759614 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8zv6\" (UniqueName: \"kubernetes.io/projected/41d1f840-a59c-4f46-84b0-d04c3cd1d500-kube-api-access-c8zv6\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt\" (UID: \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" Nov 01 16:02:13.759715 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.759291 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-istio-data\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv\" (UID: \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" Nov 01 16:02:13.760756 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.760738 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-istio-podinfo\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv\" (UID: \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" Nov 01 16:02:13.760835 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.760761 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-istio-envoy\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv\" (UID: \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" Nov 01 16:02:13.767769 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.767742 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hszw2\" (UniqueName: \"kubernetes.io/projected/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-kube-api-access-hszw2\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv\" (UID: \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" Nov 01 16:02:13.767856 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.767795 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-istio-token\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv\" (UID: \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" Nov 01 16:02:13.797285 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.797258 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" Nov 01 16:02:13.827765 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.827730 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" Nov 01 16:02:13.863041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.860866 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41d1f840-a59c-4f46-84b0-d04c3cd1d500-istiod-ca-cert\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt\" (UID: \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" Nov 01 16:02:13.863041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.860939 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41d1f840-a59c-4f46-84b0-d04c3cd1d500-istio-podinfo\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt\" (UID: \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" Nov 01 16:02:13.863041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.860982 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8zv6\" (UniqueName: \"kubernetes.io/projected/41d1f840-a59c-4f46-84b0-d04c3cd1d500-kube-api-access-c8zv6\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt\" (UID: \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" Nov 01 16:02:13.863041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.861040 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d1f840-a59c-4f46-84b0-d04c3cd1d500-workload-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt\" (UID: \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" Nov 01 16:02:13.863041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.861187 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41d1f840-a59c-4f46-84b0-d04c3cd1d500-istio-token\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt\" (UID: \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" Nov 01 16:02:13.863041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.861254 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41d1f840-a59c-4f46-84b0-d04c3cd1d500-istio-envoy\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt\" (UID: \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" Nov 01 16:02:13.863041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.861317 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41d1f840-a59c-4f46-84b0-d04c3cd1d500-istio-data\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt\" (UID: \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" Nov 01 16:02:13.863041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.861515 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d1f840-a59c-4f46-84b0-d04c3cd1d500-credential-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt\" (UID: \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" Nov 01 16:02:13.863041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.861581 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41d1f840-a59c-4f46-84b0-d04c3cd1d500-workload-certs\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt\" (UID: \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" Nov 01 16:02:13.863041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.862216 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41d1f840-a59c-4f46-84b0-d04c3cd1d500-workload-certs\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt\" (UID: \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" Nov 01 16:02:13.863041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.862858 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41d1f840-a59c-4f46-84b0-d04c3cd1d500-istiod-ca-cert\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt\" (UID: \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" Nov 01 16:02:13.867517 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.864223 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d1f840-a59c-4f46-84b0-d04c3cd1d500-workload-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt\" (UID: \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" Nov 01 16:02:13.867517 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.864498 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41d1f840-a59c-4f46-84b0-d04c3cd1d500-istio-data\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt\" (UID: \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" Nov 01 16:02:13.867517 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.865114 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d1f840-a59c-4f46-84b0-d04c3cd1d500-credential-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt\" (UID: \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" Nov 01 16:02:13.978239 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.944764 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41d1f840-a59c-4f46-84b0-d04c3cd1d500-istio-token\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt\" (UID: \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" Nov 01 16:02:13.978239 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:13.947228 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8zv6\" (UniqueName: \"kubernetes.io/projected/41d1f840-a59c-4f46-84b0-d04c3cd1d500-kube-api-access-c8zv6\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt\" (UID: \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" Nov 01 16:02:14.046813 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.046778 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8"] Nov 01 16:02:14.054805 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.054779 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8"] Nov 01 16:02:14.055116 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.055098 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" Nov 01 16:02:14.058621 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.058538 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41d1f840-a59c-4f46-84b0-d04c3cd1d500-istio-envoy\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt\" (UID: \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" Nov 01 16:02:14.058621 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.058550 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41d1f840-a59c-4f46-84b0-d04c3cd1d500-istio-podinfo\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt\" (UID: \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" Nov 01 16:02:14.142240 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:02:14.142195 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc10680c0_ac33_4914_931e_15c3728484bc.slice/crio-9db9da70022d60f8ae7c4ff0da869d7671a396e30779f97fce6d04d6a6ddab01 WatchSource:0}: Error finding container 9db9da70022d60f8ae7c4ff0da869d7671a396e30779f97fce6d04d6a6ddab01: Status 404 returned error can't find the container with id 9db9da70022d60f8ae7c4ff0da869d7671a396e30779f97fce6d04d6a6ddab01 Nov 01 16:02:14.144974 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.144931 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:14.145091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.145039 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:14.145091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.145083 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:14.147974 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.147952 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn"] Nov 01 16:02:14.178052 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.177988 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74344814-e7cb-4b6f-b295-bb441f192023-istiod-ca-cert\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8\" (UID: \"74344814-e7cb-4b6f-b295-bb441f192023\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" Nov 01 16:02:14.178210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.178090 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74344814-e7cb-4b6f-b295-bb441f192023-istio-data\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8\" (UID: \"74344814-e7cb-4b6f-b295-bb441f192023\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" Nov 01 16:02:14.178210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.178193 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74344814-e7cb-4b6f-b295-bb441f192023-workload-certs\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8\" (UID: \"74344814-e7cb-4b6f-b295-bb441f192023\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" Nov 01 16:02:14.178350 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.178253 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74344814-e7cb-4b6f-b295-bb441f192023-istio-token\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8\" (UID: \"74344814-e7cb-4b6f-b295-bb441f192023\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" Nov 01 16:02:14.178350 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.178279 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv45k\" (UniqueName: \"kubernetes.io/projected/74344814-e7cb-4b6f-b295-bb441f192023-kube-api-access-xv45k\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8\" (UID: \"74344814-e7cb-4b6f-b295-bb441f192023\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" Nov 01 16:02:14.178350 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.178332 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74344814-e7cb-4b6f-b295-bb441f192023-workload-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8\" (UID: \"74344814-e7cb-4b6f-b295-bb441f192023\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" Nov 01 16:02:14.178473 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.178415 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74344814-e7cb-4b6f-b295-bb441f192023-istio-envoy\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8\" (UID: \"74344814-e7cb-4b6f-b295-bb441f192023\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" Nov 01 16:02:14.178512 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.178477 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74344814-e7cb-4b6f-b295-bb441f192023-istio-podinfo\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8\" (UID: \"74344814-e7cb-4b6f-b295-bb441f192023\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" Nov 01 16:02:14.178578 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.178559 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74344814-e7cb-4b6f-b295-bb441f192023-credential-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8\" (UID: \"74344814-e7cb-4b6f-b295-bb441f192023\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" Nov 01 16:02:14.246362 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.244139 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt"] Nov 01 16:02:14.249896 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.249874 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" Nov 01 16:02:14.261828 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.261796 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt"] Nov 01 16:02:14.280815 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.280781 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74344814-e7cb-4b6f-b295-bb441f192023-workload-certs\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8\" (UID: \"74344814-e7cb-4b6f-b295-bb441f192023\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" Nov 01 16:02:14.293600 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.291264 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74344814-e7cb-4b6f-b295-bb441f192023-istio-token\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8\" (UID: \"74344814-e7cb-4b6f-b295-bb441f192023\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" Nov 01 16:02:14.293600 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.291306 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv45k\" (UniqueName: \"kubernetes.io/projected/74344814-e7cb-4b6f-b295-bb441f192023-kube-api-access-xv45k\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8\" (UID: \"74344814-e7cb-4b6f-b295-bb441f192023\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" Nov 01 16:02:14.293600 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.291375 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74344814-e7cb-4b6f-b295-bb441f192023-workload-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8\" (UID: \"74344814-e7cb-4b6f-b295-bb441f192023\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" Nov 01 16:02:14.293600 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.291482 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74344814-e7cb-4b6f-b295-bb441f192023-istio-envoy\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8\" (UID: \"74344814-e7cb-4b6f-b295-bb441f192023\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" Nov 01 16:02:14.293600 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.291560 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74344814-e7cb-4b6f-b295-bb441f192023-istio-podinfo\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8\" (UID: \"74344814-e7cb-4b6f-b295-bb441f192023\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" Nov 01 16:02:14.293600 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.291662 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74344814-e7cb-4b6f-b295-bb441f192023-credential-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8\" (UID: \"74344814-e7cb-4b6f-b295-bb441f192023\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" Nov 01 16:02:14.293600 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.291720 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74344814-e7cb-4b6f-b295-bb441f192023-istiod-ca-cert\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8\" (UID: \"74344814-e7cb-4b6f-b295-bb441f192023\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" Nov 01 16:02:14.293600 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.291765 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74344814-e7cb-4b6f-b295-bb441f192023-istio-data\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8\" (UID: \"74344814-e7cb-4b6f-b295-bb441f192023\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" Nov 01 16:02:14.293600 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.292174 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74344814-e7cb-4b6f-b295-bb441f192023-istio-data\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8\" (UID: \"74344814-e7cb-4b6f-b295-bb441f192023\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" Nov 01 16:02:14.293600 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.281919 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74344814-e7cb-4b6f-b295-bb441f192023-workload-certs\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8\" (UID: \"74344814-e7cb-4b6f-b295-bb441f192023\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" Nov 01 16:02:14.293600 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.292905 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74344814-e7cb-4b6f-b295-bb441f192023-workload-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8\" (UID: \"74344814-e7cb-4b6f-b295-bb441f192023\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" Nov 01 16:02:14.293600 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.293439 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74344814-e7cb-4b6f-b295-bb441f192023-credential-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8\" (UID: \"74344814-e7cb-4b6f-b295-bb441f192023\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" Nov 01 16:02:14.294361 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.294178 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74344814-e7cb-4b6f-b295-bb441f192023-istiod-ca-cert\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8\" (UID: \"74344814-e7cb-4b6f-b295-bb441f192023\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" Nov 01 16:02:14.295765 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.295742 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74344814-e7cb-4b6f-b295-bb441f192023-istio-envoy\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8\" (UID: \"74344814-e7cb-4b6f-b295-bb441f192023\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" Nov 01 16:02:14.300186 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.297970 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74344814-e7cb-4b6f-b295-bb441f192023-istio-podinfo\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8\" (UID: \"74344814-e7cb-4b6f-b295-bb441f192023\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" Nov 01 16:02:14.307200 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.307113 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv"] Nov 01 16:02:14.308740 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:02:14.308703 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74a60e78_8fd2_43a1_9a8d_a1d13ee76a5c.slice/crio-7fd1e38f496d126d8c338a22a677f3571b153c21461881c4c59e00e1889d5b85 WatchSource:0}: Error finding container 7fd1e38f496d126d8c338a22a677f3571b153c21461881c4c59e00e1889d5b85: Status 404 returned error can't find the container with id 7fd1e38f496d126d8c338a22a677f3571b153c21461881c4c59e00e1889d5b85 Nov 01 16:02:14.313097 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.312850 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74344814-e7cb-4b6f-b295-bb441f192023-istio-token\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8\" (UID: \"74344814-e7cb-4b6f-b295-bb441f192023\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" Nov 01 16:02:14.313097 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.312943 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv45k\" (UniqueName: \"kubernetes.io/projected/74344814-e7cb-4b6f-b295-bb441f192023-kube-api-access-xv45k\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8\" (UID: \"74344814-e7cb-4b6f-b295-bb441f192023\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" Nov 01 16:02:14.314216 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.313878 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:14.314216 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.313966 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:14.314216 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.314064 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:14.319821 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.319804 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" Nov 01 16:02:14.378062 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.375574 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" Nov 01 16:02:14.396195 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.395363 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07f5c387-a86b-4118-bf29-dad1f686db58-istio-podinfo\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt\" (UID: \"07f5c387-a86b-4118-bf29-dad1f686db58\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" Nov 01 16:02:14.396195 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.395487 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07f5c387-a86b-4118-bf29-dad1f686db58-workload-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt\" (UID: \"07f5c387-a86b-4118-bf29-dad1f686db58\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" Nov 01 16:02:14.396195 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.395576 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07f5c387-a86b-4118-bf29-dad1f686db58-istiod-ca-cert\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt\" (UID: \"07f5c387-a86b-4118-bf29-dad1f686db58\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" Nov 01 16:02:14.396195 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.395601 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07f5c387-a86b-4118-bf29-dad1f686db58-istio-data\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt\" (UID: \"07f5c387-a86b-4118-bf29-dad1f686db58\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" Nov 01 16:02:14.396195 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.395637 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6djgw\" (UniqueName: \"kubernetes.io/projected/07f5c387-a86b-4118-bf29-dad1f686db58-kube-api-access-6djgw\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt\" (UID: \"07f5c387-a86b-4118-bf29-dad1f686db58\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" Nov 01 16:02:14.396195 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.395692 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07f5c387-a86b-4118-bf29-dad1f686db58-workload-certs\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt\" (UID: \"07f5c387-a86b-4118-bf29-dad1f686db58\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" Nov 01 16:02:14.396195 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.395820 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07f5c387-a86b-4118-bf29-dad1f686db58-istio-envoy\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt\" (UID: \"07f5c387-a86b-4118-bf29-dad1f686db58\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" Nov 01 16:02:14.396195 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.395883 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07f5c387-a86b-4118-bf29-dad1f686db58-credential-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt\" (UID: \"07f5c387-a86b-4118-bf29-dad1f686db58\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" Nov 01 16:02:14.396195 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.395932 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07f5c387-a86b-4118-bf29-dad1f686db58-istio-token\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt\" (UID: \"07f5c387-a86b-4118-bf29-dad1f686db58\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" Nov 01 16:02:14.501370 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.498602 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07f5c387-a86b-4118-bf29-dad1f686db58-istio-envoy\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt\" (UID: \"07f5c387-a86b-4118-bf29-dad1f686db58\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" Nov 01 16:02:14.501370 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.498690 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07f5c387-a86b-4118-bf29-dad1f686db58-credential-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt\" (UID: \"07f5c387-a86b-4118-bf29-dad1f686db58\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" Nov 01 16:02:14.501370 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.498754 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07f5c387-a86b-4118-bf29-dad1f686db58-istio-token\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt\" (UID: \"07f5c387-a86b-4118-bf29-dad1f686db58\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" Nov 01 16:02:14.501370 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.498842 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07f5c387-a86b-4118-bf29-dad1f686db58-istio-podinfo\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt\" (UID: \"07f5c387-a86b-4118-bf29-dad1f686db58\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" Nov 01 16:02:14.501370 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.498949 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07f5c387-a86b-4118-bf29-dad1f686db58-workload-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt\" (UID: \"07f5c387-a86b-4118-bf29-dad1f686db58\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" Nov 01 16:02:14.501370 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.499060 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07f5c387-a86b-4118-bf29-dad1f686db58-istiod-ca-cert\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt\" (UID: \"07f5c387-a86b-4118-bf29-dad1f686db58\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" Nov 01 16:02:14.501370 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.499089 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07f5c387-a86b-4118-bf29-dad1f686db58-istio-data\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt\" (UID: \"07f5c387-a86b-4118-bf29-dad1f686db58\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" Nov 01 16:02:14.501370 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.499123 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6djgw\" (UniqueName: \"kubernetes.io/projected/07f5c387-a86b-4118-bf29-dad1f686db58-kube-api-access-6djgw\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt\" (UID: \"07f5c387-a86b-4118-bf29-dad1f686db58\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" Nov 01 16:02:14.501370 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.499178 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07f5c387-a86b-4118-bf29-dad1f686db58-workload-certs\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt\" (UID: \"07f5c387-a86b-4118-bf29-dad1f686db58\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" Nov 01 16:02:14.501370 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.499649 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07f5c387-a86b-4118-bf29-dad1f686db58-workload-certs\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt\" (UID: \"07f5c387-a86b-4118-bf29-dad1f686db58\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" Nov 01 16:02:14.501370 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.500379 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07f5c387-a86b-4118-bf29-dad1f686db58-workload-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt\" (UID: \"07f5c387-a86b-4118-bf29-dad1f686db58\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" Nov 01 16:02:14.501370 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.500659 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07f5c387-a86b-4118-bf29-dad1f686db58-credential-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt\" (UID: \"07f5c387-a86b-4118-bf29-dad1f686db58\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" Nov 01 16:02:14.511197 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.508338 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07f5c387-a86b-4118-bf29-dad1f686db58-istio-data\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt\" (UID: \"07f5c387-a86b-4118-bf29-dad1f686db58\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" Nov 01 16:02:14.511197 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.509135 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07f5c387-a86b-4118-bf29-dad1f686db58-istiod-ca-cert\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt\" (UID: \"07f5c387-a86b-4118-bf29-dad1f686db58\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" Nov 01 16:02:14.526126 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.526046 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07f5c387-a86b-4118-bf29-dad1f686db58-istio-token\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt\" (UID: \"07f5c387-a86b-4118-bf29-dad1f686db58\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" Nov 01 16:02:14.527064 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.526931 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07f5c387-a86b-4118-bf29-dad1f686db58-istio-podinfo\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt\" (UID: \"07f5c387-a86b-4118-bf29-dad1f686db58\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" Nov 01 16:02:14.530420 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.529960 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6djgw\" (UniqueName: \"kubernetes.io/projected/07f5c387-a86b-4118-bf29-dad1f686db58-kube-api-access-6djgw\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt\" (UID: \"07f5c387-a86b-4118-bf29-dad1f686db58\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" Nov 01 16:02:14.530420 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.530376 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07f5c387-a86b-4118-bf29-dad1f686db58-istio-envoy\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt\" (UID: \"07f5c387-a86b-4118-bf29-dad1f686db58\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" Nov 01 16:02:14.585848 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.585804 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" Nov 01 16:02:14.659692 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.659642 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt"] Nov 01 16:02:14.680233 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.679429 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:14.680233 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.679528 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:14.680233 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.679582 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:14.788000 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:02:14.787353 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74344814_e7cb_4b6f_b295_bb441f192023.slice/crio-cc4bf8c178d37d3485925fa878326d665140db7fa55e153dbae7f93761b06231 WatchSource:0}: Error finding container cc4bf8c178d37d3485925fa878326d665140db7fa55e153dbae7f93761b06231: Status 404 returned error can't find the container with id cc4bf8c178d37d3485925fa878326d665140db7fa55e153dbae7f93761b06231 Nov 01 16:02:14.995325 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.995255 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:14.995477 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.995346 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:14.995477 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:14.995385 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:15.048854 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:15.048717 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8"] Nov 01 16:02:15.055476 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:15.055397 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt"] Nov 01 16:02:15.079581 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:02:15.066926 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f5c387_a86b_4118_bf29_dad1f686db58.slice/crio-255bfbe9db72ad5d4c3e2bb935980f3e002244ae055d15a9feeb875f875172a1 WatchSource:0}: Error finding container 255bfbe9db72ad5d4c3e2bb935980f3e002244ae055d15a9feeb875f875172a1: Status 404 returned error can't find the container with id 255bfbe9db72ad5d4c3e2bb935980f3e002244ae055d15a9feeb875f875172a1 Nov 01 16:02:15.079581 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:15.073309 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:15.079581 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:15.073396 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:15.079581 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:15.073436 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:15.105668 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:15.105632 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" event={"ID":"41d1f840-a59c-4f46-84b0-d04c3cd1d500","Type":"ContainerStarted","Data":"f11b60de5c029ba956d3e5cef1d1d8fcc91a7654adb0f2f27eaf452d502bf650"} Nov 01 16:02:15.105931 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:15.105908 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" event={"ID":"41d1f840-a59c-4f46-84b0-d04c3cd1d500","Type":"ContainerStarted","Data":"cd1d0422ce395e279f51c9dd3135b764e149cfb737775c01377443f90a73ffd5"} Nov 01 16:02:15.109611 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:15.109312 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" event={"ID":"74344814-e7cb-4b6f-b295-bb441f192023","Type":"ContainerStarted","Data":"cc4bf8c178d37d3485925fa878326d665140db7fa55e153dbae7f93761b06231"} Nov 01 16:02:15.112897 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:15.111509 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" event={"ID":"c10680c0-ac33-4914-931e-15c3728484bc","Type":"ContainerStarted","Data":"e323a502222833dc923d26c2a6ec20177c3ac61e1ab12e27f21f15b860a277c0"} Nov 01 16:02:15.112897 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:15.111562 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" event={"ID":"c10680c0-ac33-4914-931e-15c3728484bc","Type":"ContainerStarted","Data":"9db9da70022d60f8ae7c4ff0da869d7671a396e30779f97fce6d04d6a6ddab01"} Nov 01 16:02:15.118775 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:15.118725 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" event={"ID":"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c","Type":"ContainerStarted","Data":"9a7c872d94e083fdc805f44d27c1cc61c2bd1b5828fc585d17f010d0168a821f"} Nov 01 16:02:15.119134 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:15.119113 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" event={"ID":"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c","Type":"ContainerStarted","Data":"7fd1e38f496d126d8c338a22a677f3571b153c21461881c4c59e00e1889d5b85"} Nov 01 16:02:15.156789 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:15.156700 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" podStartSLOduration=2.156679108 podStartE2EDuration="2.156679108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:02:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:02:15.156140048 +0000 UTC m=+868.487877699" watchObservedRunningTime="2026-11-01 16:02:15.156679108 +0000 UTC m=+868.488416823" Nov 01 16:02:15.224341 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:15.224009 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" podStartSLOduration=2.2239876020000002 podStartE2EDuration="2.223987602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:02:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:02:15.221340471 +0000 UTC m=+868.553078122" watchObservedRunningTime="2026-11-01 16:02:15.223987602 +0000 UTC m=+868.555725254" Nov 01 16:02:15.244183 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:15.244051 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" podStartSLOduration=2.2440111050000002 podStartE2EDuration="2.244011105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:02:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:02:15.24323335 +0000 UTC m=+868.574971001" watchObservedRunningTime="2026-11-01 16:02:15.244011105 +0000 UTC m=+868.575748755" Nov 01 16:02:15.333844 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:15.333812 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" Nov 01 16:02:15.808336 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:15.808302 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" Nov 01 16:02:15.817356 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:15.817325 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 16:02:15.817643 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:15.817604 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" podUID="c10680c0-ac33-4914-931e-15c3728484bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 16:02:15.832221 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:15.832191 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" Nov 01 16:02:15.832745 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:15.832715 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 01 16:02:15.832926 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:15.832902 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" podUID="74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 01 16:02:16.262708 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:16.262665 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" event={"ID":"07f5c387-a86b-4118-bf29-dad1f686db58","Type":"ContainerStarted","Data":"481fcb02af97305e92c5bc5a150e826137f71145b69c869d4be82e2f085555d6"} Nov 01 16:02:16.267518 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:16.267486 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" event={"ID":"07f5c387-a86b-4118-bf29-dad1f686db58","Type":"ContainerStarted","Data":"255bfbe9db72ad5d4c3e2bb935980f3e002244ae055d15a9feeb875f875172a1"} Nov 01 16:02:16.287745 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:16.287708 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" event={"ID":"74344814-e7cb-4b6f-b295-bb441f192023","Type":"ContainerStarted","Data":"00a54faa1bcc3176a4d7bb7c40007969ddd95f826cee12a7c991e871bfdf9a2e"} Nov 01 16:02:16.321046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:16.308778 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" podStartSLOduration=2.30875715 podStartE2EDuration="2.30875715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:02:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:02:16.294888412 +0000 UTC m=+869.626626073" watchObservedRunningTime="2026-11-01 16:02:16.30875715 +0000 UTC m=+869.640494801" Nov 01 16:02:16.345337 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:16.338176 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 01 16:02:16.345337 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:16.338240 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" podUID="41d1f840-a59c-4f46-84b0-d04c3cd1d500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 01 16:02:16.383986 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:16.383120 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" Nov 01 16:02:16.383986 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:16.383253 4363 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 01 16:02:16.383986 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:16.383299 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" podUID="74344814-e7cb-4b6f-b295-bb441f192023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 01 16:02:16.419039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:16.409699 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" podStartSLOduration=3.409678971 podStartE2EDuration="3.409678971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:02:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:02:16.352505429 +0000 UTC m=+869.684243080" watchObservedRunningTime="2026-11-01 16:02:16.409678971 +0000 UTC m=+869.741416622" Nov 01 16:02:16.590316 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:16.590223 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" Nov 01 16:02:16.629242 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:16.629189 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 16:02:16.629533 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:16.629487 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" podUID="07f5c387-a86b-4118-bf29-dad1f686db58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 16:02:16.645831 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:16.645790 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:16.756217 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:16.756168 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:16.805698 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:16.805237 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 16:02:16.805698 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:16.805296 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" podUID="c10680c0-ac33-4914-931e-15c3728484bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 16:02:16.839042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:16.829005 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 01 16:02:16.839042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:16.829089 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" podUID="74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 01 16:02:17.145326 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:17.145230 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 01 16:02:17.337904 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:17.337858 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 01 16:02:17.338126 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:17.337922 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" podUID="41d1f840-a59c-4f46-84b0-d04c3cd1d500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 01 16:02:17.383049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:17.382472 4363 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 01 16:02:17.383049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:17.382535 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" podUID="74344814-e7cb-4b6f-b295-bb441f192023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 01 16:02:17.394063 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:17.393589 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:17.599365 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:17.592414 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:17.599365 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:17.593598 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 16:02:17.599365 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:17.593658 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" podUID="07f5c387-a86b-4118-bf29-dad1f686db58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 16:02:17.698040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:17.697234 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:17.698040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:17.698044 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 16:02:17.781043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:17.763105 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:02:17.781043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:17.763859 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:02:17.963100 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:17.962423 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 01 16:02:17.963100 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:17.962484 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" podUID="74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 01 16:02:18.076256 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:18.063531 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 16:02:18.076256 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:18.063597 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" podUID="c10680c0-ac33-4914-931e-15c3728484bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 16:02:18.203043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:18.188347 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 01 16:02:18.237099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:18.230281 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 16:02:18.356382 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:18.337997 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 01 16:02:18.356382 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:18.338091 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" podUID="41d1f840-a59c-4f46-84b0-d04c3cd1d500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 01 16:02:18.380042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:18.378397 4363 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 01 16:02:18.380042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:18.378487 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" podUID="74344814-e7cb-4b6f-b295-bb441f192023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 01 16:02:18.458706 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:18.458494 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:18.478047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:18.472130 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:02:18.649044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:18.646512 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 16:02:18.649044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:18.646573 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" podUID="07f5c387-a86b-4118-bf29-dad1f686db58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 16:02:18.760041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:18.759223 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:18.790039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:18.786672 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:18.860041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:18.859691 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 01 16:02:18.860041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:18.859766 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" podUID="74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 01 16:02:19.239488 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:19.202239 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:19.371590 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:19.343698 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 16:02:19.371590 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:19.343775 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" podUID="c10680c0-ac33-4914-931e-15c3728484bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 16:02:19.395739 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:19.395685 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 01 16:02:19.420585 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:19.401146 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" podUID="41d1f840-a59c-4f46-84b0-d04c3cd1d500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 01 16:02:19.484458 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:19.484412 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:02:19.561460 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:19.488743 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:19.845681 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:02:19.497336 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:02:19.859192 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:19.519764 4363 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 01 16:02:19.859478 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:19.859446 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" podUID="74344814-e7cb-4b6f-b295-bb441f192023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 01 16:02:19.859663 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:19.800104 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:19.860120 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:19.673607 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:02:19.860276 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:19.641357 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 16:02:19.860380 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:19.860362 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" podUID="07f5c387-a86b-4118-bf29-dad1f686db58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 16:02:19.860542 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:19.841241 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 16:02:19.996868 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:02:19.996834 4363 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.021s" Nov 01 16:02:20.052327 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:20.052303 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 16:02:20.068071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:19.860665 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" podUID="c10680c0-ac33-4914-931e-15c3728484bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 16:02:20.068329 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:20.049803 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 01 16:02:20.068523 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:20.068480 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" podUID="74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 01 16:02:20.239151 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:20.233445 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 01 16:02:20.349420 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:20.335612 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 01 16:02:20.349420 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:20.335699 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" podUID="41d1f840-a59c-4f46-84b0-d04c3cd1d500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 01 16:02:20.386764 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:20.386730 4363 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 01 16:02:20.411056 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:20.410924 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" podUID="74344814-e7cb-4b6f-b295-bb441f192023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 01 16:02:20.566319 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:20.566197 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:02:20.566929 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:20.566898 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:20.713050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:20.661338 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": context deadline exceeded" Nov 01 16:02:20.713050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:20.687998 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 16:02:20.713050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:20.688079 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" podUID="07f5c387-a86b-4118-bf29-dad1f686db58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 16:02:20.849113 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:20.841827 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 16:02:20.849113 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:20.841883 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" podUID="c10680c0-ac33-4914-931e-15c3728484bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 16:02:20.849113 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:20.842511 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 01 16:02:20.849113 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:20.842552 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" podUID="74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 01 16:02:20.889061 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:20.867879 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:20.889061 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:20.868781 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:02:20.889061 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:20.869792 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:21.235788 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:21.235682 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:21.336279 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:21.336239 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 01 16:02:21.336584 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:21.336535 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" podUID="41d1f840-a59c-4f46-84b0-d04c3cd1d500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 01 16:02:21.353584 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:21.353555 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:02:21.379770 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:21.379743 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:02:21.382061 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:21.382035 4363 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 01 16:02:21.382331 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:21.382307 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" podUID="74344814-e7cb-4b6f-b295-bb441f192023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 01 16:02:21.388118 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:21.382665 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:02:21.605105 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:21.580148 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:21.656339 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:21.625138 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 16:02:21.771037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:21.761326 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" podUID="07f5c387-a86b-4118-bf29-dad1f686db58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 16:02:21.771037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:21.656599 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 16:02:21.829202 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:21.823501 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 16:02:21.829202 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:21.823593 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" podUID="c10680c0-ac33-4914-931e-15c3728484bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 16:02:21.829202 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:21.829145 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 01 16:02:21.829202 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:21.829194 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" podUID="74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 01 16:02:22.027099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:22.011508 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:02:22.027099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:22.011923 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:02:22.027099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:22.012207 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:02:22.250289 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:22.250228 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:22.347040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:22.344101 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 01 16:02:22.347040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:22.344205 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" podUID="41d1f840-a59c-4f46-84b0-d04c3cd1d500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 01 16:02:22.391042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:22.379312 4363 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 01 16:02:22.425146 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:22.425096 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" podUID="74344814-e7cb-4b6f-b295-bb441f192023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 01 16:02:22.583043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:22.582100 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:22.619096 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:22.618984 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 16:02:22.619341 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:22.619309 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" podUID="07f5c387-a86b-4118-bf29-dad1f686db58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 16:02:22.660437 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:22.660392 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:22.820698 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:22.820214 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 16:02:22.820698 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:22.820277 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" podUID="c10680c0-ac33-4914-931e-15c3728484bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 16:02:22.897090 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:22.873941 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 01 16:02:22.897090 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:22.874007 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" podUID="74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 01 16:02:23.044042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:23.013676 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:23.045166 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:23.039227 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:02:23.045634 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:23.039261 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:02:23.301103 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:23.264183 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:23.411084 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:23.405068 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 01 16:02:23.411084 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:23.405151 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" podUID="41d1f840-a59c-4f46-84b0-d04c3cd1d500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 01 16:02:23.411084 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:23.406462 4363 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 01 16:02:23.411084 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:23.406504 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" podUID="74344814-e7cb-4b6f-b295-bb441f192023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 01 16:02:23.713043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:23.652603 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 16:02:23.713043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:23.652667 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" podUID="07f5c387-a86b-4118-bf29-dad1f686db58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 16:02:23.713043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:23.653095 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:02:23.713043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:23.653360 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:02:23.738040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:23.729187 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:02:23.934290 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:23.934241 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 16:02:23.934496 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:23.934343 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" podUID="c10680c0-ac33-4914-931e-15c3728484bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 16:02:24.039137 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:24.039032 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 01 16:02:24.039137 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:24.039094 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" podUID="74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 01 16:02:24.392090 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:24.072906 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:24.393229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:24.078338 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:02:24.394456 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:24.078373 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:02:24.436073 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:24.290034 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 01 16:02:24.437009 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:24.290077 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded" Nov 01 16:02:24.437216 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:24.369807 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 01 16:02:24.437450 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:24.437384 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" podUID="41d1f840-a59c-4f46-84b0-d04c3cd1d500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 01 16:02:24.612938 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:24.612885 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 16:02:24.613261 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:24.613214 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" podUID="07f5c387-a86b-4118-bf29-dad1f686db58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 16:02:24.703211 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:24.703102 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:24.742106 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:24.742060 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:24.876044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:24.857720 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 16:02:24.876044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:24.857798 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" podUID="c10680c0-ac33-4914-931e-15c3728484bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 16:02:24.978126 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:24.963750 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 01 16:02:24.978126 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:24.963818 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" podUID="74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 01 16:02:24.996192 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:24.996135 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:25.011823 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:25.004153 4363 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 01 16:02:25.026168 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:25.026121 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" podUID="74344814-e7cb-4b6f-b295-bb441f192023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 01 16:02:25.102925 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:25.102875 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:25.394042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:25.393540 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:02:25.409071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:25.399090 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:25.422843 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:25.422805 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 01 16:02:25.428180 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:25.428133 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" podUID="41d1f840-a59c-4f46-84b0-d04c3cd1d500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 01 16:02:25.428906 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:25.428878 4363 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 01 16:02:25.429139 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:25.429102 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" podUID="74344814-e7cb-4b6f-b295-bb441f192023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 01 16:02:25.438845 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:25.438807 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:25.462946 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:25.462902 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:02:25.569390 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:25.569357 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:02:25.604043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:25.598138 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 16:02:25.604043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:25.598200 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" podUID="07f5c387-a86b-4118-bf29-dad1f686db58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 16:02:25.818112 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:25.805214 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 16:02:25.844047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:25.819934 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 16:02:25.844047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:25.819989 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" podUID="c10680c0-ac33-4914-931e-15c3728484bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 16:02:25.854090 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:25.846139 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:25.888703 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:25.857217 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:02:25.967742 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:25.956846 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 01 16:02:25.967742 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:25.956906 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" podUID="74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 01 16:02:26.139328 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:26.139298 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:02:26.160049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:26.158673 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 16:02:26.335921 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:26.322417 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 01 16:02:26.335921 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:26.322480 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" podUID="41d1f840-a59c-4f46-84b0-d04c3cd1d500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 01 16:02:26.400943 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:26.400842 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:26.520653 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:26.518246 4363 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 01 16:02:26.520653 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:26.518310 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" podUID="74344814-e7cb-4b6f-b295-bb441f192023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 01 16:02:26.575053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:26.574331 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:02:26.630041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:26.629332 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 16:02:26.630041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:26.629391 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" podUID="07f5c387-a86b-4118-bf29-dad1f686db58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 16:02:26.831155 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:26.829468 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 16:02:26.831155 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:26.829525 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" podUID="c10680c0-ac33-4914-931e-15c3728484bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 16:02:26.831155 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:26.829925 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 01 16:02:26.831155 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:26.829962 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" podUID="74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 01 16:02:27.144051 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:27.136572 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:27.148682 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:27.145122 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:27.331070 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:27.322491 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 01 16:02:27.331070 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:27.322586 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" podUID="41d1f840-a59c-4f46-84b0-d04c3cd1d500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 01 16:02:27.385864 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:27.385828 4363 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 01 16:02:27.386179 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:27.386148 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" podUID="74344814-e7cb-4b6f-b295-bb441f192023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 01 16:02:27.418654 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:27.418541 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:27.587884 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:27.587451 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 16:02:27.587884 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:27.587509 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" podUID="07f5c387-a86b-4118-bf29-dad1f686db58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 16:02:27.735553 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:27.735455 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:02:28.096188 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:28.094593 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:02:28.100541 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:28.100508 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" Nov 01 16:02:28.131045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:28.109475 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" Nov 01 16:02:28.213041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:28.208789 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" Nov 01 16:02:28.491606 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:28.491515 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 01 16:02:28.526050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:28.503671 4363 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 01 16:02:28.526311 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:28.526277 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" podUID="74344814-e7cb-4b6f-b295-bb441f192023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 01 16:02:28.527639 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:28.527609 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" podUID="41d1f840-a59c-4f46-84b0-d04c3cd1d500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 01 16:02:28.587426 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:28.587388 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 16:02:28.600063 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:28.599972 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" podUID="07f5c387-a86b-4118-bf29-dad1f686db58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 16:02:28.765823 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:28.765734 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" Nov 01 16:02:28.768177 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:28.768156 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" Nov 01 16:02:28.787421 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:28.787385 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" Nov 01 16:02:29.334042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:29.333241 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" Nov 01 16:02:29.384220 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:29.384182 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" Nov 01 16:02:29.592396 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:29.592368 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" Nov 01 16:02:29.945094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:29.944171 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" Nov 01 16:02:29.984639 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:29.984397 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" Nov 01 16:02:29.985325 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:29.985266 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" Nov 01 16:02:30.022598 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:30.022475 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" Nov 01 16:02:30.652601 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:30.652568 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:02:30.652891 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:30.652850 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" podUID="07f5c387-a86b-4118-bf29-dad1f686db58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:31.280042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:31.279440 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:31.422047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:31.415061 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" Nov 01 16:02:31.449364 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:31.449333 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" Nov 01 16:02:31.469039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:31.465113 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:31.850825 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:31.850680 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:31.886126 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:31.881674 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:02:32.034151 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:32.034099 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:32.291101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:32.290104 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:32.291101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:32.290777 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 16:02:32.490278 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:32.490235 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:32.925041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:32.907511 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:32.981126 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:32.945918 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:02:32.981126 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:32.948323 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:02:33.052042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:33.050976 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:33.185942 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:33.185854 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:02:33.204817 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:33.204786 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:02:33.205144 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:33.205126 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:02:33.303038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:33.296906 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:33.523119 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:33.514510 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:33.523119 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:33.515279 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:02:33.953143 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:33.926655 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:02:33.979102 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:33.946591 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:02:34.012476 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:34.009875 4363 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:02:34.012830 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:34.012795 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" podUID="1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:34.111042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:34.063214 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:34.267130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:34.263651 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 16:02:34.280053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:34.267367 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:34.565237 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:34.565133 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:02:34.597099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:34.594916 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:02:34.802558 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:34.802512 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:34.937366 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:34.937331 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:02:34.999051 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:34.985170 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:35.222082 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:35.221974 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:02:35.251640 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:35.251608 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:02:43.839215 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:43.839181 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj"] Nov 01 16:02:43.845874 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:43.845849 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" Nov 01 16:02:43.872916 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:43.872875 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj"] Nov 01 16:02:43.957175 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:43.957135 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4b55ba5-b1af-492d-abe2-1baa85947260-istio-data\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj\" (UID: \"d4b55ba5-b1af-492d-abe2-1baa85947260\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" Nov 01 16:02:43.957383 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:43.957354 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srx75\" (UniqueName: \"kubernetes.io/projected/d4b55ba5-b1af-492d-abe2-1baa85947260-kube-api-access-srx75\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj\" (UID: \"d4b55ba5-b1af-492d-abe2-1baa85947260\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" Nov 01 16:02:43.957461 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:43.957444 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4b55ba5-b1af-492d-abe2-1baa85947260-istio-envoy\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj\" (UID: \"d4b55ba5-b1af-492d-abe2-1baa85947260\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" Nov 01 16:02:43.957563 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:43.957540 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4b55ba5-b1af-492d-abe2-1baa85947260-istio-token\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj\" (UID: \"d4b55ba5-b1af-492d-abe2-1baa85947260\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" Nov 01 16:02:43.957614 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:43.957602 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b55ba5-b1af-492d-abe2-1baa85947260-workload-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj\" (UID: \"d4b55ba5-b1af-492d-abe2-1baa85947260\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" Nov 01 16:02:43.957657 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:43.957649 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b55ba5-b1af-492d-abe2-1baa85947260-credential-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj\" (UID: \"d4b55ba5-b1af-492d-abe2-1baa85947260\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" Nov 01 16:02:43.957770 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:43.957752 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4b55ba5-b1af-492d-abe2-1baa85947260-workload-certs\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj\" (UID: \"d4b55ba5-b1af-492d-abe2-1baa85947260\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" Nov 01 16:02:43.957891 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:43.957874 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4b55ba5-b1af-492d-abe2-1baa85947260-istiod-ca-cert\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj\" (UID: \"d4b55ba5-b1af-492d-abe2-1baa85947260\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" Nov 01 16:02:43.957951 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:43.957938 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4b55ba5-b1af-492d-abe2-1baa85947260-istio-podinfo\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj\" (UID: \"d4b55ba5-b1af-492d-abe2-1baa85947260\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" Nov 01 16:02:44.042008 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.041974 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc"] Nov 01 16:02:44.046336 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.046313 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" Nov 01 16:02:44.061991 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.060160 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b55ba5-b1af-492d-abe2-1baa85947260-workload-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj\" (UID: \"d4b55ba5-b1af-492d-abe2-1baa85947260\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" Nov 01 16:02:44.061991 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.060308 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b55ba5-b1af-492d-abe2-1baa85947260-credential-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj\" (UID: \"d4b55ba5-b1af-492d-abe2-1baa85947260\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" Nov 01 16:02:44.061991 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.060371 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4b55ba5-b1af-492d-abe2-1baa85947260-workload-certs\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj\" (UID: \"d4b55ba5-b1af-492d-abe2-1baa85947260\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" Nov 01 16:02:44.061991 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.060477 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4b55ba5-b1af-492d-abe2-1baa85947260-istiod-ca-cert\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj\" (UID: \"d4b55ba5-b1af-492d-abe2-1baa85947260\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" Nov 01 16:02:44.061991 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.060569 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4b55ba5-b1af-492d-abe2-1baa85947260-istio-podinfo\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj\" (UID: \"d4b55ba5-b1af-492d-abe2-1baa85947260\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" Nov 01 16:02:44.061991 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.060613 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4b55ba5-b1af-492d-abe2-1baa85947260-istio-data\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj\" (UID: \"d4b55ba5-b1af-492d-abe2-1baa85947260\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" Nov 01 16:02:44.061991 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.060721 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srx75\" (UniqueName: \"kubernetes.io/projected/d4b55ba5-b1af-492d-abe2-1baa85947260-kube-api-access-srx75\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj\" (UID: \"d4b55ba5-b1af-492d-abe2-1baa85947260\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" Nov 01 16:02:44.061991 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.060770 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4b55ba5-b1af-492d-abe2-1baa85947260-istio-envoy\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj\" (UID: \"d4b55ba5-b1af-492d-abe2-1baa85947260\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" Nov 01 16:02:44.061991 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.060822 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4b55ba5-b1af-492d-abe2-1baa85947260-istio-token\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj\" (UID: \"d4b55ba5-b1af-492d-abe2-1baa85947260\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" Nov 01 16:02:44.061991 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.061534 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b55ba5-b1af-492d-abe2-1baa85947260-workload-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj\" (UID: \"d4b55ba5-b1af-492d-abe2-1baa85947260\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" Nov 01 16:02:44.061991 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.061750 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b55ba5-b1af-492d-abe2-1baa85947260-credential-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj\" (UID: \"d4b55ba5-b1af-492d-abe2-1baa85947260\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" Nov 01 16:02:44.061991 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.061985 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4b55ba5-b1af-492d-abe2-1baa85947260-workload-certs\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj\" (UID: \"d4b55ba5-b1af-492d-abe2-1baa85947260\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" Nov 01 16:02:44.062943 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.062621 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4b55ba5-b1af-492d-abe2-1baa85947260-istiod-ca-cert\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj\" (UID: \"d4b55ba5-b1af-492d-abe2-1baa85947260\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" Nov 01 16:02:44.063616 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.063578 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4b55ba5-b1af-492d-abe2-1baa85947260-istio-data\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj\" (UID: \"d4b55ba5-b1af-492d-abe2-1baa85947260\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" Nov 01 16:02:44.079041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.078961 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4b55ba5-b1af-492d-abe2-1baa85947260-istio-podinfo\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj\" (UID: \"d4b55ba5-b1af-492d-abe2-1baa85947260\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" Nov 01 16:02:44.087476 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.087441 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4b55ba5-b1af-492d-abe2-1baa85947260-istio-token\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj\" (UID: \"d4b55ba5-b1af-492d-abe2-1baa85947260\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" Nov 01 16:02:44.096770 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.096742 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4b55ba5-b1af-492d-abe2-1baa85947260-istio-envoy\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj\" (UID: \"d4b55ba5-b1af-492d-abe2-1baa85947260\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" Nov 01 16:02:44.099123 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.098951 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc"] Nov 01 16:02:44.101700 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.101647 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srx75\" (UniqueName: \"kubernetes.io/projected/d4b55ba5-b1af-492d-abe2-1baa85947260-kube-api-access-srx75\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj\" (UID: \"d4b55ba5-b1af-492d-abe2-1baa85947260\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" Nov 01 16:02:44.162846 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.162811 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfe59fad-a9c8-4073-81c6-11f4eda63086-istio-envoy\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc\" (UID: \"bfe59fad-a9c8-4073-81c6-11f4eda63086\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" Nov 01 16:02:44.163057 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.162871 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8695\" (UniqueName: \"kubernetes.io/projected/bfe59fad-a9c8-4073-81c6-11f4eda63086-kube-api-access-m8695\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc\" (UID: \"bfe59fad-a9c8-4073-81c6-11f4eda63086\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" Nov 01 16:02:44.163150 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.163051 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfe59fad-a9c8-4073-81c6-11f4eda63086-istio-token\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc\" (UID: \"bfe59fad-a9c8-4073-81c6-11f4eda63086\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" Nov 01 16:02:44.163150 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.163130 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfe59fad-a9c8-4073-81c6-11f4eda63086-istiod-ca-cert\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc\" (UID: \"bfe59fad-a9c8-4073-81c6-11f4eda63086\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" Nov 01 16:02:44.163277 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.163259 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfe59fad-a9c8-4073-81c6-11f4eda63086-istio-data\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc\" (UID: \"bfe59fad-a9c8-4073-81c6-11f4eda63086\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" Nov 01 16:02:44.163398 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.163376 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe59fad-a9c8-4073-81c6-11f4eda63086-workload-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc\" (UID: \"bfe59fad-a9c8-4073-81c6-11f4eda63086\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" Nov 01 16:02:44.163447 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.163427 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfe59fad-a9c8-4073-81c6-11f4eda63086-istio-podinfo\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc\" (UID: \"bfe59fad-a9c8-4073-81c6-11f4eda63086\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" Nov 01 16:02:44.163503 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.163489 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfe59fad-a9c8-4073-81c6-11f4eda63086-workload-certs\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc\" (UID: \"bfe59fad-a9c8-4073-81c6-11f4eda63086\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" Nov 01 16:02:44.163663 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.163645 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe59fad-a9c8-4073-81c6-11f4eda63086-credential-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc\" (UID: \"bfe59fad-a9c8-4073-81c6-11f4eda63086\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" Nov 01 16:02:44.187951 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.187927 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" Nov 01 16:02:44.271005 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.267771 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe59fad-a9c8-4073-81c6-11f4eda63086-credential-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc\" (UID: \"bfe59fad-a9c8-4073-81c6-11f4eda63086\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" Nov 01 16:02:44.271005 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.267832 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfe59fad-a9c8-4073-81c6-11f4eda63086-istio-envoy\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc\" (UID: \"bfe59fad-a9c8-4073-81c6-11f4eda63086\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" Nov 01 16:02:44.271005 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.267882 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8695\" (UniqueName: \"kubernetes.io/projected/bfe59fad-a9c8-4073-81c6-11f4eda63086-kube-api-access-m8695\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc\" (UID: \"bfe59fad-a9c8-4073-81c6-11f4eda63086\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" Nov 01 16:02:44.271005 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.267938 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfe59fad-a9c8-4073-81c6-11f4eda63086-istio-token\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc\" (UID: \"bfe59fad-a9c8-4073-81c6-11f4eda63086\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" Nov 01 16:02:44.271005 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.267993 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfe59fad-a9c8-4073-81c6-11f4eda63086-istiod-ca-cert\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc\" (UID: \"bfe59fad-a9c8-4073-81c6-11f4eda63086\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" Nov 01 16:02:44.271005 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.268146 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfe59fad-a9c8-4073-81c6-11f4eda63086-istio-data\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc\" (UID: \"bfe59fad-a9c8-4073-81c6-11f4eda63086\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" Nov 01 16:02:44.271005 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.268249 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe59fad-a9c8-4073-81c6-11f4eda63086-workload-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc\" (UID: \"bfe59fad-a9c8-4073-81c6-11f4eda63086\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" Nov 01 16:02:44.271005 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.268296 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfe59fad-a9c8-4073-81c6-11f4eda63086-istio-podinfo\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc\" (UID: \"bfe59fad-a9c8-4073-81c6-11f4eda63086\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" Nov 01 16:02:44.271005 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.268331 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfe59fad-a9c8-4073-81c6-11f4eda63086-workload-certs\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc\" (UID: \"bfe59fad-a9c8-4073-81c6-11f4eda63086\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" Nov 01 16:02:44.271005 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.268813 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfe59fad-a9c8-4073-81c6-11f4eda63086-workload-certs\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc\" (UID: \"bfe59fad-a9c8-4073-81c6-11f4eda63086\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" Nov 01 16:02:44.271005 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.269173 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe59fad-a9c8-4073-81c6-11f4eda63086-credential-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc\" (UID: \"bfe59fad-a9c8-4073-81c6-11f4eda63086\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" Nov 01 16:02:44.271005 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.270629 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfe59fad-a9c8-4073-81c6-11f4eda63086-istio-data\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc\" (UID: \"bfe59fad-a9c8-4073-81c6-11f4eda63086\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" Nov 01 16:02:44.272097 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.271879 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfe59fad-a9c8-4073-81c6-11f4eda63086-istio-envoy\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc\" (UID: \"bfe59fad-a9c8-4073-81c6-11f4eda63086\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" Nov 01 16:02:44.277434 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.275539 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfe59fad-a9c8-4073-81c6-11f4eda63086-istio-podinfo\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc\" (UID: \"bfe59fad-a9c8-4073-81c6-11f4eda63086\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" Nov 01 16:02:44.279460 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.279376 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8695\" (UniqueName: \"kubernetes.io/projected/bfe59fad-a9c8-4073-81c6-11f4eda63086-kube-api-access-m8695\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc\" (UID: \"bfe59fad-a9c8-4073-81c6-11f4eda63086\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" Nov 01 16:02:44.281535 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.281154 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfe59fad-a9c8-4073-81c6-11f4eda63086-istio-token\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc\" (UID: \"bfe59fad-a9c8-4073-81c6-11f4eda63086\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" Nov 01 16:02:44.283837 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.282890 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe59fad-a9c8-4073-81c6-11f4eda63086-workload-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc\" (UID: \"bfe59fad-a9c8-4073-81c6-11f4eda63086\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" Nov 01 16:02:44.292360 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.292329 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfe59fad-a9c8-4073-81c6-11f4eda63086-istiod-ca-cert\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc\" (UID: \"bfe59fad-a9c8-4073-81c6-11f4eda63086\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" Nov 01 16:02:44.373842 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.373759 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" Nov 01 16:02:44.557865 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.550165 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj"] Nov 01 16:02:44.570332 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.570291 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:44.570450 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.570427 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:44.570507 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.570479 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:44.823045 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:02:44.823003 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfe59fad_a9c8_4073_81c6_11f4eda63086.slice/crio-b37c6e1b05585e753da187a9a3d8f887e92fa32347df50eed90e6c315104a94c WatchSource:0}: Error finding container b37c6e1b05585e753da187a9a3d8f887e92fa32347df50eed90e6c315104a94c: Status 404 returned error can't find the container with id b37c6e1b05585e753da187a9a3d8f887e92fa32347df50eed90e6c315104a94c Nov 01 16:02:44.823664 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.823628 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc"] Nov 01 16:02:44.825817 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.825783 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:44.825948 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.825858 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:44.825948 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:44.825901 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:45.607091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:45.596204 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" event={"ID":"d4b55ba5-b1af-492d-abe2-1baa85947260","Type":"ContainerStarted","Data":"b7529db873a8893bcca0e736639bd1bd76e4cc7538633cae8d240320c5cc1fce"} Nov 01 16:02:45.607091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:45.596287 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" event={"ID":"d4b55ba5-b1af-492d-abe2-1baa85947260","Type":"ContainerStarted","Data":"68a1fccdbb906aa5efc7d432639854440eec39c75ca1b92dc4dd9961de2b04e1"} Nov 01 16:02:45.607700 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:45.607249 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" event={"ID":"bfe59fad-a9c8-4073-81c6-11f4eda63086","Type":"ContainerStarted","Data":"c472bea28321ed20bfc49b0b7f82215ac7384e18f1ac96ffcc1fceb0eba005d3"} Nov 01 16:02:45.607700 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:45.607325 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" event={"ID":"bfe59fad-a9c8-4073-81c6-11f4eda63086","Type":"ContainerStarted","Data":"b37c6e1b05585e753da187a9a3d8f887e92fa32347df50eed90e6c315104a94c"} Nov 01 16:02:45.625194 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:45.625129 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" podStartSLOduration=2.625109965 podStartE2EDuration="2.625109965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:02:45.62368069 +0000 UTC m=+898.955418341" watchObservedRunningTime="2026-11-01 16:02:45.625109965 +0000 UTC m=+898.956847617" Nov 01 16:02:45.658166 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:45.658112 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" podStartSLOduration=1.658101082 podStartE2EDuration="1.658101082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:02:45.654152602 +0000 UTC m=+898.985890264" watchObservedRunningTime="2026-11-01 16:02:45.658101082 +0000 UTC m=+898.989838732" Nov 01 16:02:46.196953 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:46.196917 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" Nov 01 16:02:46.197391 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:46.197363 4363 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 16:02:46.197565 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:46.197541 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" podUID="d4b55ba5-b1af-492d-abe2-1baa85947260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 16:02:46.374622 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:46.374595 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" Nov 01 16:02:46.376043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:46.376002 4363 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 16:02:46.376186 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:46.376081 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" podUID="bfe59fad-a9c8-4073-81c6-11f4eda63086" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 16:02:46.973403 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:46.973373 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtwhw_8254aafa-dd5b-4150-87e4-d165dab6bcb5/ovn-acl-logging/0.log" Nov 01 16:02:47.031053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:47.018308 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtwhw_8254aafa-dd5b-4150-87e4-d165dab6bcb5/ovn-acl-logging/0.log" Nov 01 16:02:47.209037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:47.195502 4363 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 16:02:47.209037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:47.195559 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" podUID="d4b55ba5-b1af-492d-abe2-1baa85947260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 16:02:47.396038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:47.395480 4363 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 16:02:47.396038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:47.395546 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" podUID="bfe59fad-a9c8-4073-81c6-11f4eda63086" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 16:02:48.192041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:48.188468 4363 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 16:02:48.192041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:48.188541 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" podUID="d4b55ba5-b1af-492d-abe2-1baa85947260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 16:02:48.375010 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:48.374972 4363 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 16:02:48.375215 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:48.375069 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" podUID="bfe59fad-a9c8-4073-81c6-11f4eda63086" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 16:02:48.447178 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:02:48.445432 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:02:49.205040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:49.200047 4363 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 16:02:49.205040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:49.200116 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" podUID="d4b55ba5-b1af-492d-abe2-1baa85947260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 16:02:49.374162 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:49.374127 4363 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 16:02:49.374328 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:49.374179 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" podUID="bfe59fad-a9c8-4073-81c6-11f4eda63086" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 16:02:50.190468 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:50.190423 4363 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 16:02:50.190769 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:50.190736 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" podUID="d4b55ba5-b1af-492d-abe2-1baa85947260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 16:02:50.402037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:50.378143 4363 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 16:02:50.402037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:50.378205 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" podUID="bfe59fad-a9c8-4073-81c6-11f4eda63086" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 16:02:51.193039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:51.189111 4363 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 16:02:51.193039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:51.189180 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" podUID="d4b55ba5-b1af-492d-abe2-1baa85947260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 16:02:51.377046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:51.376139 4363 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 16:02:51.377046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:51.376266 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" podUID="bfe59fad-a9c8-4073-81c6-11f4eda63086" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 16:02:52.191491 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:52.191453 4363 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 16:02:52.192207 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:52.192173 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" podUID="d4b55ba5-b1af-492d-abe2-1baa85947260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 16:02:52.394121 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:52.394088 4363 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 16:02:52.394371 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:52.394342 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" podUID="bfe59fad-a9c8-4073-81c6-11f4eda63086" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 16:02:53.199072 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:53.199038 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" Nov 01 16:02:53.202663 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:53.202638 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" Nov 01 16:02:53.214548 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:53.214526 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" Nov 01 16:02:53.393134 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:53.393092 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" Nov 01 16:02:53.394375 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:53.394350 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" Nov 01 16:02:53.395114 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:02:53.395081 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" Nov 01 16:03:00.983349 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:00.983294 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:03:00.984685 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:00.984647 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:03:01.065267 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:01.065222 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:01.231163 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:01.231110 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:01.261207 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:01.261105 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:01.390162 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:01.390111 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:01.739247 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:01.739199 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:02.029103 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:02.013257 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:02.029103 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:02.020131 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:03:02.103989 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:02.054126 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:03:02.103989 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:02.054775 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:03:02.103989 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:02.076171 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:02.103989 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:02.076808 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:03:02.266853 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:02.256378 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:02.266853 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:02.257067 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:03:02.287102 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:02.267234 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:02.287102 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:02.280298 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 16:03:02.819221 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:02.794793 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:02.819221 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:02.795340 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 16:03:03.072114 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:03.015387 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:03.144047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:03.072126 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:03.464091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:03.085258 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:03.589767 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:03.274162 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:03:03.736105 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:03.277449 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 01 16:03:03.819314 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:03.819262 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:03:03.876045 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:03:03.861274 4363 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.065s" Nov 01 16:03:04.146146 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:04.136317 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:03:04.290037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:04.283342 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:04.430106 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:04.429833 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:04.593040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:04.590608 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:04.710268 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:04.710164 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 01 16:03:04.711839 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:04.710536 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:04.928532 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:04.893818 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:03:05.029704 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:05.029582 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:05.057127 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:05.057068 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:05.148277 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:05.137324 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:05.448578 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:05.448500 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:05.610073 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:05.607921 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:05.725274 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:05.725149 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 01 16:03:05.734164 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:05.734119 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:05.977743 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:05.977642 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:03:06.149270 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:06.149228 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:06.452473 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:06.452427 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:03:06.630042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:06.625528 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:06.643367 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:06.643321 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:06.692315 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:06.692272 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:06.692616 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:06.692581 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" podUID="3b704396-3411-40af-95fe-4b67a41ae69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:06.744096 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:06.741654 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:06.744096 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:06.741737 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" podUID="edcab30a-6e4e-4019-9fd7-96fe7b085711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:06.750041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:06.744398 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:03:06.759946 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:06.750295 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:07.007205 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:07.007097 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:07.180406 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:07.180361 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:03:07.506043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:07.485176 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:07.506043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:07.485245 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" podUID="408aa757-573f-4cd4-b81d-c26c9af252f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:07.506043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:07.498278 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:03:07.596043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:07.566390 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:07.596043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:07.566456 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" podUID="af073176-cb1d-4eaf-9b6c-43787f6a7227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:07.641078 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:07.625344 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:07.641078 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:07.625405 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" podUID="6daeee34-6958-40f5-a360-9a58b3bd51d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:07.641078 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:07.625844 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:07.776157 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:07.755167 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:07.776157 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:07.763127 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:07.926077 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:07.921285 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:07.926077 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:07.921345 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" podUID="ff3d8641-74e7-496e-8b67-fd5db4a94e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:07.968047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:07.967150 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:07.968047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:07.967220 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" podUID="b3d831af-96a0-4040-8568-e092380a88c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:08.062223 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:08.062122 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:08.071603 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:08.071108 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:08.071603 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:08.071171 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" podUID="674b7965-e091-422f-b128-35635859883f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:08.193168 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:08.193120 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:03:08.434544 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:08.434505 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:08.434848 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:08.434812 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" podUID="05ec5955-5241-423b-8161-1ac7608f4fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:08.499150 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:08.499105 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:08.687102 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:08.669118 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:03:09.031102 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:09.017597 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:09.031102 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:09.019070 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 01 16:03:09.031102 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:09.024401 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:09.031102 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:09.024451 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" podUID="ef2387a8-1f76-4653-9047-0f5824e6cfbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:09.092041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:09.091358 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:09.369062 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:09.358878 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:03:09.429040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:09.426007 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:09.429040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:09.426103 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" podUID="241f70eb-0584-4e76-9616-152a84654bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:09.509043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:09.507987 4363 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:09.509043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:09.508092 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" podUID="bfe59fad-a9c8-4073-81c6-11f4eda63086" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:09.647135 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:09.589525 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:09.670710 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:09.669635 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:10.038876 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:10.038761 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:10.147298 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:10.078307 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:10.176067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:10.165632 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:10.492042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:10.489838 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:03:10.492042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:10.490505 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": context deadline exceeded" Nov 01 16:03:10.561038 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:03:10.509549 4363 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.491s" Nov 01 16:03:10.709244 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:10.709187 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:10.750048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:10.749935 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:03:11.221040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:11.188265 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:11.253039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:11.227776 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:11.253039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:11.251964 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:03:11.259936 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:11.259912 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:03:11.333707 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:11.333149 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:03:11.334186 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:11.334159 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 16:03:11.338240 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:11.337954 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:03:11.339918 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:11.339896 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:03:11.349049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:11.349008 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 16:03:18.314086 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:03:18.314005 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:03:28.633532 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:28.633489 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr"] Nov 01 16:03:28.661793 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:28.661752 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr"] Nov 01 16:03:28.662318 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:28.662296 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" Nov 01 16:03:28.750127 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:28.750092 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9247119e-2d9d-4817-a84c-b48159236aed-istio-envoy\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr\" (UID: \"9247119e-2d9d-4817-a84c-b48159236aed\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" Nov 01 16:03:28.750290 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:28.750160 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9247119e-2d9d-4817-a84c-b48159236aed-istio-token\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr\" (UID: \"9247119e-2d9d-4817-a84c-b48159236aed\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" Nov 01 16:03:28.750290 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:28.750266 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9247119e-2d9d-4817-a84c-b48159236aed-workload-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr\" (UID: \"9247119e-2d9d-4817-a84c-b48159236aed\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" Nov 01 16:03:28.750407 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:28.750372 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9247119e-2d9d-4817-a84c-b48159236aed-credential-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr\" (UID: \"9247119e-2d9d-4817-a84c-b48159236aed\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" Nov 01 16:03:28.750466 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:28.750408 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9247119e-2d9d-4817-a84c-b48159236aed-istio-data\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr\" (UID: \"9247119e-2d9d-4817-a84c-b48159236aed\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" Nov 01 16:03:28.750521 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:28.750484 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9247119e-2d9d-4817-a84c-b48159236aed-istio-podinfo\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr\" (UID: \"9247119e-2d9d-4817-a84c-b48159236aed\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" Nov 01 16:03:28.750635 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:28.750620 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9247119e-2d9d-4817-a84c-b48159236aed-workload-certs\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr\" (UID: \"9247119e-2d9d-4817-a84c-b48159236aed\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" Nov 01 16:03:28.750699 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:28.750666 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9247119e-2d9d-4817-a84c-b48159236aed-istiod-ca-cert\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr\" (UID: \"9247119e-2d9d-4817-a84c-b48159236aed\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" Nov 01 16:03:28.750780 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:28.750767 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8fkl\" (UniqueName: \"kubernetes.io/projected/9247119e-2d9d-4817-a84c-b48159236aed-kube-api-access-w8fkl\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr\" (UID: \"9247119e-2d9d-4817-a84c-b48159236aed\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" Nov 01 16:03:28.852089 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:28.852052 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8fkl\" (UniqueName: \"kubernetes.io/projected/9247119e-2d9d-4817-a84c-b48159236aed-kube-api-access-w8fkl\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr\" (UID: \"9247119e-2d9d-4817-a84c-b48159236aed\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" Nov 01 16:03:28.852260 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:28.852171 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9247119e-2d9d-4817-a84c-b48159236aed-istio-envoy\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr\" (UID: \"9247119e-2d9d-4817-a84c-b48159236aed\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" Nov 01 16:03:28.852260 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:28.852204 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9247119e-2d9d-4817-a84c-b48159236aed-istio-token\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr\" (UID: \"9247119e-2d9d-4817-a84c-b48159236aed\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" Nov 01 16:03:28.852260 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:28.852226 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9247119e-2d9d-4817-a84c-b48159236aed-workload-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr\" (UID: \"9247119e-2d9d-4817-a84c-b48159236aed\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" Nov 01 16:03:28.852380 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:28.852290 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9247119e-2d9d-4817-a84c-b48159236aed-credential-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr\" (UID: \"9247119e-2d9d-4817-a84c-b48159236aed\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" Nov 01 16:03:28.852380 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:28.852309 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9247119e-2d9d-4817-a84c-b48159236aed-istio-data\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr\" (UID: \"9247119e-2d9d-4817-a84c-b48159236aed\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" Nov 01 16:03:28.852380 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:28.852328 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9247119e-2d9d-4817-a84c-b48159236aed-istio-podinfo\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr\" (UID: \"9247119e-2d9d-4817-a84c-b48159236aed\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" Nov 01 16:03:28.852545 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:28.852472 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9247119e-2d9d-4817-a84c-b48159236aed-workload-certs\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr\" (UID: \"9247119e-2d9d-4817-a84c-b48159236aed\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" Nov 01 16:03:28.852545 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:28.852520 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9247119e-2d9d-4817-a84c-b48159236aed-istiod-ca-cert\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr\" (UID: \"9247119e-2d9d-4817-a84c-b48159236aed\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" Nov 01 16:03:28.852818 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:28.852784 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9247119e-2d9d-4817-a84c-b48159236aed-istio-data\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr\" (UID: \"9247119e-2d9d-4817-a84c-b48159236aed\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" Nov 01 16:03:28.852944 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:28.852814 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9247119e-2d9d-4817-a84c-b48159236aed-workload-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr\" (UID: \"9247119e-2d9d-4817-a84c-b48159236aed\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" Nov 01 16:03:28.852944 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:28.852865 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9247119e-2d9d-4817-a84c-b48159236aed-credential-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr\" (UID: \"9247119e-2d9d-4817-a84c-b48159236aed\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" Nov 01 16:03:28.852944 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:28.852895 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9247119e-2d9d-4817-a84c-b48159236aed-workload-certs\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr\" (UID: \"9247119e-2d9d-4817-a84c-b48159236aed\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" Nov 01 16:03:28.853311 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:28.853294 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9247119e-2d9d-4817-a84c-b48159236aed-istiod-ca-cert\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr\" (UID: \"9247119e-2d9d-4817-a84c-b48159236aed\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" Nov 01 16:03:28.854794 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:28.854767 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9247119e-2d9d-4817-a84c-b48159236aed-istio-envoy\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr\" (UID: \"9247119e-2d9d-4817-a84c-b48159236aed\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" Nov 01 16:03:28.854966 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:28.854949 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9247119e-2d9d-4817-a84c-b48159236aed-istio-podinfo\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr\" (UID: \"9247119e-2d9d-4817-a84c-b48159236aed\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" Nov 01 16:03:28.859415 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:28.859395 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8fkl\" (UniqueName: \"kubernetes.io/projected/9247119e-2d9d-4817-a84c-b48159236aed-kube-api-access-w8fkl\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr\" (UID: \"9247119e-2d9d-4817-a84c-b48159236aed\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" Nov 01 16:03:28.859851 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:28.859837 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9247119e-2d9d-4817-a84c-b48159236aed-istio-token\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr\" (UID: \"9247119e-2d9d-4817-a84c-b48159236aed\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" Nov 01 16:03:29.019386 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:29.019297 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" Nov 01 16:03:29.311457 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:29.311394 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr"] Nov 01 16:03:29.311822 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:03:29.311797 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9247119e_2d9d_4817_a84c_b48159236aed.slice/crio-23f255fabb151c52c879a9f6b3920f73c0b459eb9bb435ea694dea0ca0e77063 WatchSource:0}: Error finding container 23f255fabb151c52c879a9f6b3920f73c0b459eb9bb435ea694dea0ca0e77063: Status 404 returned error can't find the container with id 23f255fabb151c52c879a9f6b3920f73c0b459eb9bb435ea694dea0ca0e77063 Nov 01 16:03:29.315085 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:29.315011 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:03:29.315200 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:29.315113 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:03:29.315200 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:29.315142 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:03:29.331231 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:29.331204 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" event={"ID":"9247119e-2d9d-4817-a84c-b48159236aed","Type":"ContainerStarted","Data":"23f255fabb151c52c879a9f6b3920f73c0b459eb9bb435ea694dea0ca0e77063"} Nov 01 16:03:30.476048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:30.474594 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" event={"ID":"9247119e-2d9d-4817-a84c-b48159236aed","Type":"ContainerStarted","Data":"6c6000120d9be4d34bec2684d7bc5fea5c34c5cb8d5b29f9933293430c2a2064"} Nov 01 16:03:30.538733 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:30.537878 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" podStartSLOduration=2.537857606 podStartE2EDuration="2.537857606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:03:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:03:30.534308357 +0000 UTC m=+943.866046013" watchObservedRunningTime="2026-11-01 16:03:30.537857606 +0000 UTC m=+943.869595257" Nov 01 16:03:31.020219 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:31.020180 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" Nov 01 16:03:31.022131 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:31.022100 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 16:03:31.022265 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:31.022164 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" podUID="9247119e-2d9d-4817-a84c-b48159236aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 16:03:32.029458 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:32.029422 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 16:03:32.030091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:32.030056 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" podUID="9247119e-2d9d-4817-a84c-b48159236aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 16:03:33.020319 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:33.020278 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 16:03:33.020506 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:33.020343 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" podUID="9247119e-2d9d-4817-a84c-b48159236aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 16:03:34.032041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:34.031479 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 16:03:34.032041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:34.031587 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" podUID="9247119e-2d9d-4817-a84c-b48159236aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 16:03:34.714069 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:34.712188 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:34.741069 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:34.717329 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:03:34.997098 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:34.989112 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:35.031049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:35.005144 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:03:35.031049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:35.023610 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:35.031049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:35.025053 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 16:03:35.031049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:35.025104 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" podUID="9247119e-2d9d-4817-a84c-b48159236aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 16:03:35.164530 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:35.138219 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:03:35.464052 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:35.461715 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:35.633054 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:35.632170 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:35.714726 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:35.713373 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:35.730046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:35.728449 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 16:03:35.732891 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:35.732855 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:03:35.745352 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:35.745294 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:03:36.120953 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:36.120906 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:36.140823 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:36.140735 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 16:03:36.187953 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:36.161429 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:36.519100 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:36.385989 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:03:36.824498 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:36.551392 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:36.824498 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:36.601323 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:36.824498 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:36.715639 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 01 16:03:36.894857 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:36.890636 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:36.895251 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:36.895210 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" podUID="edcab30a-6e4e-4019-9fd7-96fe7b085711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:36.903040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:36.899160 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 16:03:36.938184 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:36.938128 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" podUID="9247119e-2d9d-4817-a84c-b48159236aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 16:03:36.942067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:36.739175 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:37.323143 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:37.323096 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-qncjz" podUID="7c0c2696-09a8-4726-98ca-d8331d86e830" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:37.329038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:37.325164 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:03:37.664132 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:36.981769 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k" podUID="336c4cd8-6126-4431-930e-57f9b7ccf09d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:37.664785 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:37.496050 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-frphn" podUID="fa3886b7-e5e1-4f64-93d4-a097f446a835" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:37.665249 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:37.521641 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 16:03:37.665427 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:37.665391 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" podUID="9247119e-2d9d-4817-a84c-b48159236aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 16:03:37.665537 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:37.620063 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:03:37.665724 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:37.665708 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:03:37.710041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:37.145127 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:37.740043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:37.183161 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:37.779163 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:37.766175 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:37.779163 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:37.766251 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" podUID="408aa757-573f-4cd4-b81d-c26c9af252f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:38.244322 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:38.244289 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 16:03:38.254608 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:38.254568 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" podUID="9247119e-2d9d-4817-a84c-b48159236aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 16:03:38.292235 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:38.292193 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:03:38.308226 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:38.308178 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:38.337895 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:38.337857 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:03:38.409846 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:38.409804 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:38.419475 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:38.419431 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" podUID="b3d831af-96a0-4040-8568-e092380a88c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:38.508111 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:38.507999 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:38.508374 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:38.508335 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" podUID="674b7965-e091-422f-b128-35635859883f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:38.652043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:38.640410 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:03:38.736166 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:38.736113 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:38.738516 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:38.738481 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:38.765248 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:38.765139 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:38.765248 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:38.765206 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" podUID="ef2387a8-1f76-4653-9047-0f5824e6cfbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:38.816176 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:38.816132 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:38.816477 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:38.816443 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" podUID="05ec5955-5241-423b-8161-1ac7608f4fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:38.932184 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:38.932144 4363 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:38.932484 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:38.932446 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" podUID="241f70eb-0584-4e76-9616-152a84654bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:39.022461 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:39.022359 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 16:03:39.022727 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:39.022697 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" podUID="9247119e-2d9d-4817-a84c-b48159236aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 16:03:39.201248 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:39.201205 4363 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:39.201576 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:39.201543 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" podUID="07c0b57d-ed0c-4366-9716-256714694541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:39.297103 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:39.296149 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:03:39.323069 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:39.318985 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:39.325005 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:39.324974 4363 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:39.325253 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:39.325223 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" podUID="d4b55ba5-b1af-492d-abe2-1baa85947260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:39.386063 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:39.384610 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:39.464241 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:39.464105 4363 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:39.464547 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:39.464496 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" podUID="bfe59fad-a9c8-4073-81c6-11f4eda63086" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:39.653175 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:39.653125 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:39.741790 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:39.741700 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:03:39.742282 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:39.742240 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:40.022591 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:40.022489 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 16:03:40.022881 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:40.022833 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" podUID="9247119e-2d9d-4817-a84c-b48159236aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 16:03:40.300309 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:40.300119 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:40.320159 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:40.320110 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:40.385165 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:40.385116 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:40.425340 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:40.425281 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:40.656050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:40.654340 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:40.751042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:40.743109 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:03:40.751042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:40.744173 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:03:41.096104 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:41.044386 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 16:03:41.096104 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:41.044460 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" podUID="9247119e-2d9d-4817-a84c-b48159236aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 16:03:41.330510 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:41.330462 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:41.355219 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:41.344194 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:03:41.391219 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:41.391164 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:41.412763 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:41.412702 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:41.667119 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:41.666626 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:41.794206 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:41.794154 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:03:41.819281 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:41.819236 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:03:42.028101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:42.022555 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 16:03:42.028101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:42.022632 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" podUID="9247119e-2d9d-4817-a84c-b48159236aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 16:03:42.332816 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:42.331730 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:42.350975 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:42.349898 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:42.411042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:42.395098 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:42.667158 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:42.667101 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:03:42.802636 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:42.802102 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:42.821124 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:42.820112 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:42.919161 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:42.919055 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:42.919475 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:42.919435 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" podUID="4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:43.029194 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:43.029151 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 16:03:43.029488 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:43.029455 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" podUID="9247119e-2d9d-4817-a84c-b48159236aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 16:03:43.374079 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:43.361121 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:43.375485 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:43.374542 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:43.430577 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:43.399682 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:03:43.715213 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:43.715105 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:03:43.744167 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:43.744125 4363 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:43.744540 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:43.744495 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" podUID="4070ca0f-1230-4eed-992c-b3f6bce6a4bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:43.811047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:43.810545 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:43.847044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:43.830255 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:44.080095 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:43.989667 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:03:44.214044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:44.197102 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" Nov 01 16:03:44.214044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:44.101102 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:44.214044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:44.198236 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" podUID="9427e75b-2800-4bee-8711-b1e1306d8f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:44.214044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:44.157071 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 16:03:44.214044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:44.205260 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" podUID="9247119e-2d9d-4817-a84c-b48159236aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 16:03:44.214044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:44.174102 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:44.214044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:44.205532 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" podUID="c10680c0-ac33-4914-931e-15c3728484bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:44.272041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:44.271974 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:44.448043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:44.422753 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:44.469255 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:44.449412 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:44.732111 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:44.727618 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:03:44.871046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:44.837919 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:44.908830 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:44.877468 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:03:44.956467 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:44.956429 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:44.956741 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:44.956713 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" podUID="74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:45.047100 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:45.045446 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:45.059759 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:45.059723 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 16:03:45.060038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:45.059992 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" podUID="9247119e-2d9d-4817-a84c-b48159236aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 16:03:45.067153 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:45.064894 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:45.308985 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:45.308881 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:45.469187 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:45.469131 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:45.496762 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:45.496703 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:46.075704 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:46.049228 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:03:46.177260 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:46.075908 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:46.177260 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:46.176984 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" podUID="41d1f840-a59c-4f46-84b0-d04c3cd1d500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:46.177260 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:46.075975 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 16:03:46.177260 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:46.177089 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" podUID="9247119e-2d9d-4817-a84c-b48159236aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 16:03:46.180359 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:46.134442 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:03:46.344106 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:46.342448 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:46.344106 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:46.343253 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:03:46.686256 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:46.686153 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 01 16:03:47.070384 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:47.059362 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:47.097359 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:47.097315 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:03:47.098662 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:47.098626 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": context deadline exceeded" Nov 01 16:03:47.325595 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:47.325481 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:03:47.354732 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:47.354679 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:03:47.356066 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:47.356030 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:47.738171 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:47.724683 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 01 16:03:47.780040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:47.778462 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 16:03:47.780040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:47.778537 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" podUID="9247119e-2d9d-4817-a84c-b48159236aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 16:03:48.065279 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:48.065183 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-qncjz" podUID="7c0c2696-09a8-4726-98ca-d8331d86e830" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:48.113037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:48.110227 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:48.113037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:48.110891 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 16:03:48.113037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:48.110954 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" podUID="9247119e-2d9d-4817-a84c-b48159236aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 16:03:48.114228 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:48.114200 4363 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.012677064s: [/var/lib/containers/storage/overlay/7706096fa11bf79e433a9e587130a505c0c6c519b9eaa4e06ab68cf96457f770/diff /var/log/pods/kube-system_global-pull-secret-syncer-ckz9h_c7838987-12dc-47bc-9b39-a87ffd11ddaf/global-pull-secret-syncer/0.log]; will not log again for this container unless duration exceeds 2s Nov 01 16:03:48.114412 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:48.114394 4363 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.012876778s: [/var/lib/containers/storage/overlay/218d75f260863573f27ed05fc441d5baecfe167fccfe69c3d824dce309be6509/diff ]; will not log again for this container unless duration exceeds 2s Nov 01 16:03:48.207520 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:03:48.207484 4363 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.376s" Nov 01 16:03:48.207796 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:48.207779 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 16:03:48.207946 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:48.207935 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:03:48.208081 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:48.208069 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 16:03:48.208182 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:48.208174 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:03:48.226346 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:48.224085 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:03:48.226346 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:48.225857 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:03:48.359283 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:03:48.359212 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:03:49.019858 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:49.019824 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 16:03:49.020098 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:49.019876 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" podUID="9247119e-2d9d-4817-a84c-b48159236aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 16:03:50.510285 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:50.510246 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" Nov 01 16:03:50.512822 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:50.512796 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" Nov 01 16:03:51.442011 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:51.441975 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8624bae9-f4ffc7b67-pcczf"] Nov 01 16:03:51.447888 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:51.447857 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8624bae9-f4ffc7b67-pcczf" Nov 01 16:03:51.466379 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:51.466335 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8624bae9-f4ffc7b67-pcczf"] Nov 01 16:03:51.469228 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:51.469203 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8624bae9\"" Nov 01 16:03:51.469908 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:51.469498 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8624bae9-dockercfg-vz8mz\"" Nov 01 16:03:51.546040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:51.545747 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65l5s\" (UniqueName: \"kubernetes.io/projected/2d567652-04f7-416f-b812-85a8ed997863-kube-api-access-65l5s\") pod \"payload-pre-processing-e2e-ait-8624bae9-f4ffc7b67-pcczf\" (UID: \"2d567652-04f7-416f-b812-85a8ed997863\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8624bae9-f4ffc7b67-pcczf" Nov 01 16:03:51.546918 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:51.546766 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d567652-04f7-416f-b812-85a8ed997863-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8624bae9-f4ffc7b67-pcczf\" (UID: \"2d567652-04f7-416f-b812-85a8ed997863\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8624bae9-f4ffc7b67-pcczf" Nov 01 16:03:51.679064 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:51.663893 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d567652-04f7-416f-b812-85a8ed997863-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8624bae9-f4ffc7b67-pcczf\" (UID: \"2d567652-04f7-416f-b812-85a8ed997863\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8624bae9-f4ffc7b67-pcczf" Nov 01 16:03:51.679064 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:51.663985 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65l5s\" (UniqueName: \"kubernetes.io/projected/2d567652-04f7-416f-b812-85a8ed997863-kube-api-access-65l5s\") pod \"payload-pre-processing-e2e-ait-8624bae9-f4ffc7b67-pcczf\" (UID: \"2d567652-04f7-416f-b812-85a8ed997863\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8624bae9-f4ffc7b67-pcczf" Nov 01 16:03:51.687050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:51.685584 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d567652-04f7-416f-b812-85a8ed997863-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8624bae9-f4ffc7b67-pcczf\" (UID: \"2d567652-04f7-416f-b812-85a8ed997863\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8624bae9-f4ffc7b67-pcczf" Nov 01 16:03:51.741098 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:51.734888 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65l5s\" (UniqueName: \"kubernetes.io/projected/2d567652-04f7-416f-b812-85a8ed997863-kube-api-access-65l5s\") pod \"payload-pre-processing-e2e-ait-8624bae9-f4ffc7b67-pcczf\" (UID: \"2d567652-04f7-416f-b812-85a8ed997863\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8624bae9-f4ffc7b67-pcczf" Nov 01 16:03:51.784045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:51.781297 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8624bae9-f4ffc7b67-pcczf" Nov 01 16:03:52.397156 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:52.397103 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8624bae9-f4ffc7b67-pcczf"] Nov 01 16:03:53.153379 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:53.153349 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3664624a-64bd765999-zjrqf"] Nov 01 16:03:53.157781 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:53.157764 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3664624a-64bd765999-zjrqf" Nov 01 16:03:53.167548 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:53.167526 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-3664624a\"" Nov 01 16:03:53.168689 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:53.168669 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-3664624a-dockercfg-l6lt8\"" Nov 01 16:03:53.174290 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:53.174269 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3664624a-64bd765999-zjrqf"] Nov 01 16:03:53.189557 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:53.189531 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3664624a-68966c9b49-8hqvg"] Nov 01 16:03:53.210800 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:53.210776 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3664624a-68966c9b49-8hqvg"] Nov 01 16:03:53.210959 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:53.210859 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-3664624a-68966c9b49-8hqvg" Nov 01 16:03:53.252692 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:53.252660 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdcvb\" (UniqueName: \"kubernetes.io/projected/07ec09d0-8bfa-44a9-9667-1f4e5daa5c40-kube-api-access-fdcvb\") pod \"payload-processing-e2e-migrate-3664624a-68966c9b49-8hqvg\" (UID: \"07ec09d0-8bfa-44a9-9667-1f4e5daa5c40\") " pod="openshift-ingress/payload-processing-e2e-migrate-3664624a-68966c9b49-8hqvg" Nov 01 16:03:53.254283 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:53.254252 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c49f02cc-3d34-4350-9261-320ae44d8fd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-3664624a-64bd765999-zjrqf\" (UID: \"c49f02cc-3d34-4350-9261-320ae44d8fd4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3664624a-64bd765999-zjrqf" Nov 01 16:03:53.254656 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:53.254613 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07ec09d0-8bfa-44a9-9667-1f4e5daa5c40-plugins-config-volume\") pod \"payload-processing-e2e-migrate-3664624a-68966c9b49-8hqvg\" (UID: \"07ec09d0-8bfa-44a9-9667-1f4e5daa5c40\") " pod="openshift-ingress/payload-processing-e2e-migrate-3664624a-68966c9b49-8hqvg" Nov 01 16:03:53.254787 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:53.254693 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6jcj\" (UniqueName: \"kubernetes.io/projected/c49f02cc-3d34-4350-9261-320ae44d8fd4-kube-api-access-w6jcj\") pod \"payload-pre-processing-e2e-migrate-3664624a-64bd765999-zjrqf\" (UID: \"c49f02cc-3d34-4350-9261-320ae44d8fd4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3664624a-64bd765999-zjrqf" Nov 01 16:03:53.379045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:53.378105 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdcvb\" (UniqueName: \"kubernetes.io/projected/07ec09d0-8bfa-44a9-9667-1f4e5daa5c40-kube-api-access-fdcvb\") pod \"payload-processing-e2e-migrate-3664624a-68966c9b49-8hqvg\" (UID: \"07ec09d0-8bfa-44a9-9667-1f4e5daa5c40\") " pod="openshift-ingress/payload-processing-e2e-migrate-3664624a-68966c9b49-8hqvg" Nov 01 16:03:53.379045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:53.378494 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c49f02cc-3d34-4350-9261-320ae44d8fd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-3664624a-64bd765999-zjrqf\" (UID: \"c49f02cc-3d34-4350-9261-320ae44d8fd4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3664624a-64bd765999-zjrqf" Nov 01 16:03:53.379045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:53.378589 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07ec09d0-8bfa-44a9-9667-1f4e5daa5c40-plugins-config-volume\") pod \"payload-processing-e2e-migrate-3664624a-68966c9b49-8hqvg\" (UID: \"07ec09d0-8bfa-44a9-9667-1f4e5daa5c40\") " pod="openshift-ingress/payload-processing-e2e-migrate-3664624a-68966c9b49-8hqvg" Nov 01 16:03:53.379045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:53.378628 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6jcj\" (UniqueName: \"kubernetes.io/projected/c49f02cc-3d34-4350-9261-320ae44d8fd4-kube-api-access-w6jcj\") pod \"payload-pre-processing-e2e-migrate-3664624a-64bd765999-zjrqf\" (UID: \"c49f02cc-3d34-4350-9261-320ae44d8fd4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3664624a-64bd765999-zjrqf" Nov 01 16:03:53.392496 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:53.380451 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c49f02cc-3d34-4350-9261-320ae44d8fd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-3664624a-64bd765999-zjrqf\" (UID: \"c49f02cc-3d34-4350-9261-320ae44d8fd4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3664624a-64bd765999-zjrqf" Nov 01 16:03:53.392730 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:53.380632 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07ec09d0-8bfa-44a9-9667-1f4e5daa5c40-plugins-config-volume\") pod \"payload-processing-e2e-migrate-3664624a-68966c9b49-8hqvg\" (UID: \"07ec09d0-8bfa-44a9-9667-1f4e5daa5c40\") " pod="openshift-ingress/payload-processing-e2e-migrate-3664624a-68966c9b49-8hqvg" Nov 01 16:03:53.420472 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:53.420393 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6jcj\" (UniqueName: \"kubernetes.io/projected/c49f02cc-3d34-4350-9261-320ae44d8fd4-kube-api-access-w6jcj\") pod \"payload-pre-processing-e2e-migrate-3664624a-64bd765999-zjrqf\" (UID: \"c49f02cc-3d34-4350-9261-320ae44d8fd4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3664624a-64bd765999-zjrqf" Nov 01 16:03:53.444687 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:53.444653 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdcvb\" (UniqueName: \"kubernetes.io/projected/07ec09d0-8bfa-44a9-9667-1f4e5daa5c40-kube-api-access-fdcvb\") pod \"payload-processing-e2e-migrate-3664624a-68966c9b49-8hqvg\" (UID: \"07ec09d0-8bfa-44a9-9667-1f4e5daa5c40\") " pod="openshift-ingress/payload-processing-e2e-migrate-3664624a-68966c9b49-8hqvg" Nov 01 16:03:53.475338 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:53.475307 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3664624a-64bd765999-zjrqf" Nov 01 16:03:53.563087 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:53.557199 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-3664624a-68966c9b49-8hqvg" Nov 01 16:03:53.563087 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:53.559813 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8624bae9-f4ffc7b67-pcczf" event={"ID":"2d567652-04f7-416f-b812-85a8ed997863","Type":"ContainerStarted","Data":"8e7aa4a828dd04366f642c4cc5b3a009fa5883f726b2739391253385fa2062e8"} Nov 01 16:03:53.563087 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:53.559914 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8624bae9-f4ffc7b67-pcczf" event={"ID":"2d567652-04f7-416f-b812-85a8ed997863","Type":"ContainerStarted","Data":"bde67bf21bd0155614ee587fb8c4b78dac40e94b039d8708cf0724d9313c6d66"} Nov 01 16:03:53.578736 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:53.576571 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8624bae9-f4ffc7b67-pcczf" Nov 01 16:03:53.712108 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:53.707716 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-8624bae9-f4ffc7b67-pcczf" podStartSLOduration=2.707692833 podStartE2EDuration="2.707692833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:03:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:03:53.599616156 +0000 UTC m=+966.931353807" watchObservedRunningTime="2026-11-01 16:03:53.707692833 +0000 UTC m=+967.039430487" Nov 01 16:03:54.173395 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:54.173370 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3664624a-68966c9b49-8hqvg"] Nov 01 16:03:54.174465 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:03:54.174441 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07ec09d0_8bfa_44a9_9667_1f4e5daa5c40.slice/crio-8817bb252093128fc2f43f957daf9bd0812309aa657e1b68c89311cd2373dc76 WatchSource:0}: Error finding container 8817bb252093128fc2f43f957daf9bd0812309aa657e1b68c89311cd2373dc76: Status 404 returned error can't find the container with id 8817bb252093128fc2f43f957daf9bd0812309aa657e1b68c89311cd2373dc76 Nov 01 16:03:54.285371 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:03:54.285339 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc49f02cc_3d34_4350_9261_320ae44d8fd4.slice/crio-832e264aba94e32eb06bc4f5132cd7fb3f1a73a7b7f2db175a354b221e81e7f1 WatchSource:0}: Error finding container 832e264aba94e32eb06bc4f5132cd7fb3f1a73a7b7f2db175a354b221e81e7f1: Status 404 returned error can't find the container with id 832e264aba94e32eb06bc4f5132cd7fb3f1a73a7b7f2db175a354b221e81e7f1 Nov 01 16:03:54.286152 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:54.286112 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3664624a-64bd765999-zjrqf"] Nov 01 16:03:54.571100 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:54.570985 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-3664624a-68966c9b49-8hqvg" event={"ID":"07ec09d0-8bfa-44a9-9667-1f4e5daa5c40","Type":"ContainerStarted","Data":"4739f1cd00525f4dd9b40a081380b9c6cf72f27263a7bd56f824baecf7801dca"} Nov 01 16:03:54.571279 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:54.571103 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-3664624a-68966c9b49-8hqvg" Nov 01 16:03:54.571279 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:54.571134 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-3664624a-68966c9b49-8hqvg" event={"ID":"07ec09d0-8bfa-44a9-9667-1f4e5daa5c40","Type":"ContainerStarted","Data":"8817bb252093128fc2f43f957daf9bd0812309aa657e1b68c89311cd2373dc76"} Nov 01 16:03:54.573845 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:54.573818 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3664624a-64bd765999-zjrqf" event={"ID":"c49f02cc-3d34-4350-9261-320ae44d8fd4","Type":"ContainerStarted","Data":"eecb5518af735a0be57cb22c508b4ddf3635a14e8c494d59a69f9311a794e94c"} Nov 01 16:03:54.573992 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:54.573862 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3664624a-64bd765999-zjrqf" Nov 01 16:03:54.573992 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:54.573880 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3664624a-64bd765999-zjrqf" event={"ID":"c49f02cc-3d34-4350-9261-320ae44d8fd4","Type":"ContainerStarted","Data":"832e264aba94e32eb06bc4f5132cd7fb3f1a73a7b7f2db175a354b221e81e7f1"} Nov 01 16:03:54.608845 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:54.608798 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-3664624a-68966c9b49-8hqvg" podStartSLOduration=1.608784089 podStartE2EDuration="1.608784089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:03:54.604141608 +0000 UTC m=+967.935879259" watchObservedRunningTime="2026-11-01 16:03:54.608784089 +0000 UTC m=+967.940521739" Nov 01 16:03:54.667745 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:54.667688 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3664624a-64bd765999-zjrqf" podStartSLOduration=1.6676691959999999 podStartE2EDuration="1.667669196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:03:54.665429838 +0000 UTC m=+967.997167489" watchObservedRunningTime="2026-11-01 16:03:54.667669196 +0000 UTC m=+967.999406846" Nov 01 16:03:58.328946 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:58.327656 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8624bae9-655cf6c8-wngkz"] Nov 01 16:03:58.335626 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:58.333881 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8624bae9-655cf6c8-wngkz" Nov 01 16:03:58.356467 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:58.345648 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8624bae9-serving-cert\"" Nov 01 16:03:58.356467 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:58.347432 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8624bae9-655cf6c8-wngkz"] Nov 01 16:03:58.390452 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:58.390420 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e688a3b-8de5-45bf-9475-942fd0d99c70-openshift-service-ca\") pod \"maas-api-e2e-ait-8624bae9-655cf6c8-wngkz\" (UID: \"2e688a3b-8de5-45bf-9475-942fd0d99c70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8624bae9-655cf6c8-wngkz" Nov 01 16:03:58.390643 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:58.390540 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s6zx\" (UniqueName: \"kubernetes.io/projected/2e688a3b-8de5-45bf-9475-942fd0d99c70-kube-api-access-2s6zx\") pod \"maas-api-e2e-ait-8624bae9-655cf6c8-wngkz\" (UID: \"2e688a3b-8de5-45bf-9475-942fd0d99c70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8624bae9-655cf6c8-wngkz" Nov 01 16:03:58.390643 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:58.390559 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e688a3b-8de5-45bf-9475-942fd0d99c70-metrics-certs\") pod \"maas-api-e2e-ait-8624bae9-655cf6c8-wngkz\" (UID: \"2e688a3b-8de5-45bf-9475-942fd0d99c70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8624bae9-655cf6c8-wngkz" Nov 01 16:03:58.390772 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:58.390690 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e688a3b-8de5-45bf-9475-942fd0d99c70-maas-api-tls\") pod \"maas-api-e2e-ait-8624bae9-655cf6c8-wngkz\" (UID: \"2e688a3b-8de5-45bf-9475-942fd0d99c70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8624bae9-655cf6c8-wngkz" Nov 01 16:03:58.492128 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:58.492088 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e688a3b-8de5-45bf-9475-942fd0d99c70-maas-api-tls\") pod \"maas-api-e2e-ait-8624bae9-655cf6c8-wngkz\" (UID: \"2e688a3b-8de5-45bf-9475-942fd0d99c70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8624bae9-655cf6c8-wngkz" Nov 01 16:03:58.492304 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:58.492279 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e688a3b-8de5-45bf-9475-942fd0d99c70-openshift-service-ca\") pod \"maas-api-e2e-ait-8624bae9-655cf6c8-wngkz\" (UID: \"2e688a3b-8de5-45bf-9475-942fd0d99c70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8624bae9-655cf6c8-wngkz" Nov 01 16:03:58.492393 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:58.492380 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s6zx\" (UniqueName: \"kubernetes.io/projected/2e688a3b-8de5-45bf-9475-942fd0d99c70-kube-api-access-2s6zx\") pod \"maas-api-e2e-ait-8624bae9-655cf6c8-wngkz\" (UID: \"2e688a3b-8de5-45bf-9475-942fd0d99c70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8624bae9-655cf6c8-wngkz" Nov 01 16:03:58.492430 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:58.492403 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e688a3b-8de5-45bf-9475-942fd0d99c70-metrics-certs\") pod \"maas-api-e2e-ait-8624bae9-655cf6c8-wngkz\" (UID: \"2e688a3b-8de5-45bf-9475-942fd0d99c70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8624bae9-655cf6c8-wngkz" Nov 01 16:03:58.493075 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:58.493008 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e688a3b-8de5-45bf-9475-942fd0d99c70-openshift-service-ca\") pod \"maas-api-e2e-ait-8624bae9-655cf6c8-wngkz\" (UID: \"2e688a3b-8de5-45bf-9475-942fd0d99c70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8624bae9-655cf6c8-wngkz" Nov 01 16:03:58.494569 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:58.494546 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e688a3b-8de5-45bf-9475-942fd0d99c70-maas-api-tls\") pod \"maas-api-e2e-ait-8624bae9-655cf6c8-wngkz\" (UID: \"2e688a3b-8de5-45bf-9475-942fd0d99c70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8624bae9-655cf6c8-wngkz" Nov 01 16:03:58.494739 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:58.494722 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e688a3b-8de5-45bf-9475-942fd0d99c70-metrics-certs\") pod \"maas-api-e2e-ait-8624bae9-655cf6c8-wngkz\" (UID: \"2e688a3b-8de5-45bf-9475-942fd0d99c70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8624bae9-655cf6c8-wngkz" Nov 01 16:03:58.506341 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:58.506318 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s6zx\" (UniqueName: \"kubernetes.io/projected/2e688a3b-8de5-45bf-9475-942fd0d99c70-kube-api-access-2s6zx\") pod \"maas-api-e2e-ait-8624bae9-655cf6c8-wngkz\" (UID: \"2e688a3b-8de5-45bf-9475-942fd0d99c70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8624bae9-655cf6c8-wngkz" Nov 01 16:03:58.672044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:58.671438 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8624bae9-655cf6c8-wngkz" Nov 01 16:03:59.359103 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:59.356577 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8624bae9-655cf6c8-wngkz"] Nov 01 16:03:59.369537 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:59.368764 4363 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 16:03:59.803407 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:59.803365 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8624bae9-655cf6c8-wngkz" event={"ID":"2e688a3b-8de5-45bf-9475-942fd0d99c70","Type":"ContainerStarted","Data":"0c31fdba60d5445a8683fd214018126dd4423d8305f5d3445577a63eacdf2e62"} Nov 01 16:03:59.814180 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:59.814137 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ec6894c9-57bc8d4f9-2dkbw"] Nov 01 16:03:59.832372 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:59.832341 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ec6894c9-57bc8d4f9-2dkbw"] Nov 01 16:03:59.832589 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:59.832466 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec6894c9-57bc8d4f9-2dkbw" Nov 01 16:03:59.839670 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:59.839532 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ec6894c9\"" Nov 01 16:03:59.839670 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:59.839637 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ec6894c9-dockercfg-z2hk2\"" Nov 01 16:03:59.843476 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:59.843450 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ec6894c9-77f7f5dc8f-hstfs"] Nov 01 16:03:59.850178 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:59.850159 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ec6894c9-77f7f5dc8f-hstfs" Nov 01 16:03:59.858763 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:59.856300 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ec6894c9-77f7f5dc8f-hstfs"] Nov 01 16:03:59.988122 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:59.987485 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rm7x\" (UniqueName: \"kubernetes.io/projected/f2a05cd5-e01f-43a1-a318-f3dd5285a970-kube-api-access-8rm7x\") pod \"payload-processing-e2e-ait-ec6894c9-77f7f5dc8f-hstfs\" (UID: \"f2a05cd5-e01f-43a1-a318-f3dd5285a970\") " pod="openshift-ingress/payload-processing-e2e-ait-ec6894c9-77f7f5dc8f-hstfs" Nov 01 16:03:59.988122 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:59.987612 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2a05cd5-e01f-43a1-a318-f3dd5285a970-plugins-config-volume\") pod \"payload-processing-e2e-ait-ec6894c9-77f7f5dc8f-hstfs\" (UID: \"f2a05cd5-e01f-43a1-a318-f3dd5285a970\") " pod="openshift-ingress/payload-processing-e2e-ait-ec6894c9-77f7f5dc8f-hstfs" Nov 01 16:03:59.988122 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:59.987664 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bd0daf3-1545-45b7-a481-d9439a83939c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ec6894c9-57bc8d4f9-2dkbw\" (UID: \"6bd0daf3-1545-45b7-a481-d9439a83939c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ec6894c9-57bc8d4f9-2dkbw" Nov 01 16:03:59.988122 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:03:59.987734 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmm4v\" (UniqueName: \"kubernetes.io/projected/6bd0daf3-1545-45b7-a481-d9439a83939c-kube-api-access-jmm4v\") pod \"payload-pre-processing-e2e-ait-ec6894c9-57bc8d4f9-2dkbw\" (UID: \"6bd0daf3-1545-45b7-a481-d9439a83939c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ec6894c9-57bc8d4f9-2dkbw" Nov 01 16:04:00.094160 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:00.091405 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2a05cd5-e01f-43a1-a318-f3dd5285a970-plugins-config-volume\") pod \"payload-processing-e2e-ait-ec6894c9-77f7f5dc8f-hstfs\" (UID: \"f2a05cd5-e01f-43a1-a318-f3dd5285a970\") " pod="openshift-ingress/payload-processing-e2e-ait-ec6894c9-77f7f5dc8f-hstfs" Nov 01 16:04:00.094160 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:00.091486 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bd0daf3-1545-45b7-a481-d9439a83939c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ec6894c9-57bc8d4f9-2dkbw\" (UID: \"6bd0daf3-1545-45b7-a481-d9439a83939c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ec6894c9-57bc8d4f9-2dkbw" Nov 01 16:04:00.094160 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:00.091571 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmm4v\" (UniqueName: \"kubernetes.io/projected/6bd0daf3-1545-45b7-a481-d9439a83939c-kube-api-access-jmm4v\") pod \"payload-pre-processing-e2e-ait-ec6894c9-57bc8d4f9-2dkbw\" (UID: \"6bd0daf3-1545-45b7-a481-d9439a83939c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ec6894c9-57bc8d4f9-2dkbw" Nov 01 16:04:00.094160 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:00.091850 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rm7x\" (UniqueName: \"kubernetes.io/projected/f2a05cd5-e01f-43a1-a318-f3dd5285a970-kube-api-access-8rm7x\") pod \"payload-processing-e2e-ait-ec6894c9-77f7f5dc8f-hstfs\" (UID: \"f2a05cd5-e01f-43a1-a318-f3dd5285a970\") " pod="openshift-ingress/payload-processing-e2e-ait-ec6894c9-77f7f5dc8f-hstfs" Nov 01 16:04:00.094160 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:00.093087 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bd0daf3-1545-45b7-a481-d9439a83939c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ec6894c9-57bc8d4f9-2dkbw\" (UID: \"6bd0daf3-1545-45b7-a481-d9439a83939c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ec6894c9-57bc8d4f9-2dkbw" Nov 01 16:04:00.189116 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:00.186999 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2a05cd5-e01f-43a1-a318-f3dd5285a970-plugins-config-volume\") pod \"payload-processing-e2e-ait-ec6894c9-77f7f5dc8f-hstfs\" (UID: \"f2a05cd5-e01f-43a1-a318-f3dd5285a970\") " pod="openshift-ingress/payload-processing-e2e-ait-ec6894c9-77f7f5dc8f-hstfs" Nov 01 16:04:00.347113 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:00.346977 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmm4v\" (UniqueName: \"kubernetes.io/projected/6bd0daf3-1545-45b7-a481-d9439a83939c-kube-api-access-jmm4v\") pod \"payload-pre-processing-e2e-ait-ec6894c9-57bc8d4f9-2dkbw\" (UID: \"6bd0daf3-1545-45b7-a481-d9439a83939c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ec6894c9-57bc8d4f9-2dkbw" Nov 01 16:04:00.348002 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:00.347976 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rm7x\" (UniqueName: \"kubernetes.io/projected/f2a05cd5-e01f-43a1-a318-f3dd5285a970-kube-api-access-8rm7x\") pod \"payload-processing-e2e-ait-ec6894c9-77f7f5dc8f-hstfs\" (UID: \"f2a05cd5-e01f-43a1-a318-f3dd5285a970\") " pod="openshift-ingress/payload-processing-e2e-ait-ec6894c9-77f7f5dc8f-hstfs" Nov 01 16:04:00.468473 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:00.468444 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec6894c9-57bc8d4f9-2dkbw" Nov 01 16:04:00.475753 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:00.475716 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ec6894c9-77f7f5dc8f-hstfs" Nov 01 16:04:00.764354 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:00.764325 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ec6894c9-77f7f5dc8f-hstfs"] Nov 01 16:04:00.765229 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:04:00.765198 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2a05cd5_e01f_43a1_a318_f3dd5285a970.slice/crio-7b2927ce2bae03d6933e9cbda927fd7f4c7f06c6e53b2a5ab14bd8f34d02d5e1 WatchSource:0}: Error finding container 7b2927ce2bae03d6933e9cbda927fd7f4c7f06c6e53b2a5ab14bd8f34d02d5e1: Status 404 returned error can't find the container with id 7b2927ce2bae03d6933e9cbda927fd7f4c7f06c6e53b2a5ab14bd8f34d02d5e1 Nov 01 16:04:00.768395 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:04:00.768369 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bd0daf3_1545_45b7_a481_d9439a83939c.slice/crio-c04dc07da02c856ee2845d2bb2409eb04669a3ba80608348f4999244440290f5 WatchSource:0}: Error finding container c04dc07da02c856ee2845d2bb2409eb04669a3ba80608348f4999244440290f5: Status 404 returned error can't find the container with id c04dc07da02c856ee2845d2bb2409eb04669a3ba80608348f4999244440290f5 Nov 01 16:04:00.770081 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:00.770058 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ec6894c9-57bc8d4f9-2dkbw"] Nov 01 16:04:00.960064 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:00.959915 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8624bae9-655cf6c8-wngkz" podStartSLOduration=2.05650729 podStartE2EDuration="2.959893605s" totalImagesPullingTime="903.386315ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:03:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:04:00.954755758 +0000 UTC m=+974.286493409" watchObservedRunningTime="2026-11-01 16:04:00.959893605 +0000 UTC m=+974.291631258" Nov 01 16:04:01.005293 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:01.005079 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8624bae9-655cf6c8-wngkz" event={"ID":"2e688a3b-8de5-45bf-9475-942fd0d99c70","Type":"ContainerStarted","Data":"ed9c2fcf7f0ec3cf38ec61e63f1e3bc9e840bf909c49ee1d66a68090b55e3029"} Nov 01 16:04:01.005293 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:01.005147 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ec6894c9-77f7f5dc8f-hstfs" event={"ID":"f2a05cd5-e01f-43a1-a318-f3dd5285a970","Type":"ContainerStarted","Data":"7b2927ce2bae03d6933e9cbda927fd7f4c7f06c6e53b2a5ab14bd8f34d02d5e1"} Nov 01 16:04:01.005293 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:01.005182 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8624bae9-655cf6c8-wngkz" Nov 01 16:04:01.005293 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:01.005205 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec6894c9-57bc8d4f9-2dkbw" event={"ID":"6bd0daf3-1545-45b7-a481-d9439a83939c","Type":"ContainerStarted","Data":"c04dc07da02c856ee2845d2bb2409eb04669a3ba80608348f4999244440290f5"} Nov 01 16:04:02.019410 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:02.019322 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ec6894c9-77f7f5dc8f-hstfs" event={"ID":"f2a05cd5-e01f-43a1-a318-f3dd5285a970","Type":"ContainerStarted","Data":"6874e0bcec4ef004520933f45bc0e3356d5118c9a0175b821d0a7a79140fe7cb"} Nov 01 16:04:02.021290 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:02.021223 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ec6894c9-77f7f5dc8f-hstfs" Nov 01 16:04:02.029758 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:02.029712 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec6894c9-57bc8d4f9-2dkbw" event={"ID":"6bd0daf3-1545-45b7-a481-d9439a83939c","Type":"ContainerStarted","Data":"7503c03cd6c44ddbdeb436463ff12f34c1f6d76bc1ef8478d541ebcc7d8b2086"} Nov 01 16:04:02.030928 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:02.030902 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec6894c9-57bc8d4f9-2dkbw" Nov 01 16:04:02.051998 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:02.051936 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ec6894c9-77f7f5dc8f-hstfs" podStartSLOduration=3.05191653 podStartE2EDuration="3.05191653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:03:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:04:02.050443223 +0000 UTC m=+975.382180874" watchObservedRunningTime="2026-11-01 16:04:02.05191653 +0000 UTC m=+975.383654180" Nov 01 16:04:02.069318 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:02.069242 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec6894c9-57bc8d4f9-2dkbw" podStartSLOduration=3.069221368 podStartE2EDuration="3.069221368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:03:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:04:02.06843909 +0000 UTC m=+975.400176741" watchObservedRunningTime="2026-11-01 16:04:02.069221368 +0000 UTC m=+975.400959095" Nov 01 16:04:04.576741 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:04.576715 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8624bae9-f4ffc7b67-pcczf" Nov 01 16:04:05.639414 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:05.639372 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3664624a-64bd765999-zjrqf" Nov 01 16:04:05.640373 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:05.639750 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-3664624a-68966c9b49-8hqvg" Nov 01 16:04:06.414685 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:06.414648 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ec6894c9-555b99b646-qgw8c"] Nov 01 16:04:06.421619 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:06.420292 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec6894c9-555b99b646-qgw8c" Nov 01 16:04:06.433390 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:06.429246 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ec6894c9-serving-cert\"" Nov 01 16:04:06.433679 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:06.433650 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ec6894c9-555b99b646-qgw8c"] Nov 01 16:04:06.498536 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:06.498501 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba-openshift-service-ca\") pod \"maas-api-e2e-ait-ec6894c9-555b99b646-qgw8c\" (UID: \"1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec6894c9-555b99b646-qgw8c" Nov 01 16:04:06.498768 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:06.498674 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba-maas-api-tls\") pod \"maas-api-e2e-ait-ec6894c9-555b99b646-qgw8c\" (UID: \"1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec6894c9-555b99b646-qgw8c" Nov 01 16:04:06.498768 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:06.498759 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba-metrics-certs\") pod \"maas-api-e2e-ait-ec6894c9-555b99b646-qgw8c\" (UID: \"1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec6894c9-555b99b646-qgw8c" Nov 01 16:04:06.498931 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:06.498873 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9mql\" (UniqueName: \"kubernetes.io/projected/1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba-kube-api-access-n9mql\") pod \"maas-api-e2e-ait-ec6894c9-555b99b646-qgw8c\" (UID: \"1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec6894c9-555b99b646-qgw8c" Nov 01 16:04:06.614152 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:06.612680 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba-openshift-service-ca\") pod \"maas-api-e2e-ait-ec6894c9-555b99b646-qgw8c\" (UID: \"1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec6894c9-555b99b646-qgw8c" Nov 01 16:04:06.635102 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:06.631239 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba-maas-api-tls\") pod \"maas-api-e2e-ait-ec6894c9-555b99b646-qgw8c\" (UID: \"1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec6894c9-555b99b646-qgw8c" Nov 01 16:04:06.635102 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:06.631395 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba-metrics-certs\") pod \"maas-api-e2e-ait-ec6894c9-555b99b646-qgw8c\" (UID: \"1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec6894c9-555b99b646-qgw8c" Nov 01 16:04:06.635102 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:06.631563 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9mql\" (UniqueName: \"kubernetes.io/projected/1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba-kube-api-access-n9mql\") pod \"maas-api-e2e-ait-ec6894c9-555b99b646-qgw8c\" (UID: \"1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec6894c9-555b99b646-qgw8c" Nov 01 16:04:06.643074 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:06.618471 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba-openshift-service-ca\") pod \"maas-api-e2e-ait-ec6894c9-555b99b646-qgw8c\" (UID: \"1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec6894c9-555b99b646-qgw8c" Nov 01 16:04:06.678086 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:06.677814 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba-metrics-certs\") pod \"maas-api-e2e-ait-ec6894c9-555b99b646-qgw8c\" (UID: \"1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec6894c9-555b99b646-qgw8c" Nov 01 16:04:06.678827 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:06.678798 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba-maas-api-tls\") pod \"maas-api-e2e-ait-ec6894c9-555b99b646-qgw8c\" (UID: \"1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec6894c9-555b99b646-qgw8c" Nov 01 16:04:06.697745 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:06.697712 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9mql\" (UniqueName: \"kubernetes.io/projected/1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba-kube-api-access-n9mql\") pod \"maas-api-e2e-ait-ec6894c9-555b99b646-qgw8c\" (UID: \"1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec6894c9-555b99b646-qgw8c" Nov 01 16:04:06.757590 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:06.757553 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec6894c9-555b99b646-qgw8c" Nov 01 16:04:07.203038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:07.189642 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8624bae9-655cf6c8-wngkz" Nov 01 16:04:07.285576 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:07.285542 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ec6894c9-555b99b646-qgw8c"] Nov 01 16:04:08.203539 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:08.203501 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec6894c9-555b99b646-qgw8c" event={"ID":"1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba","Type":"ContainerStarted","Data":"6762f9b5bc68af93027a81b9a07540275d3e4060a052c26f564542e081daf39a"} Nov 01 16:04:08.204006 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:08.203566 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec6894c9-555b99b646-qgw8c" event={"ID":"1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba","Type":"ContainerStarted","Data":"e400ce344447f1f48b090e32e37e78f4642f027c3160dd6b2c9094ac8d603b5b"} Nov 01 16:04:08.204006 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:08.203619 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec6894c9-555b99b646-qgw8c" Nov 01 16:04:08.229656 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:08.229612 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec6894c9-555b99b646-qgw8c" podStartSLOduration=1.77274112 podStartE2EDuration="2.229597361s" totalImagesPullingTime="456.856241ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:04:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:04:08.226148191 +0000 UTC m=+981.557885841" watchObservedRunningTime="2026-11-01 16:04:08.229597361 +0000 UTC m=+981.561335011" Nov 01 16:04:11.403184 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:11.403134 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:11.452514 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:11.425812 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:11.805099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:11.669363 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:11.805099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:11.755178 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:11.805099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:11.755437 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:04:11.997457 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:11.997413 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:12.333148 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:12.333102 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:12.430299 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:12.427579 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:12.430299 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:12.428174 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:04:12.772039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:12.771420 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:12.772039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:12.771557 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:04:12.826046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:12.774696 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:04:12.826046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:12.775586 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:12.826046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:12.776936 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 16:04:12.826046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:12.776990 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:04:13.429094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:13.429038 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:13.433096 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:13.433054 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:13.514038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:13.510187 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:04:13.809415 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:13.809315 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:04:13.988845 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:13.977846 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" podUID="4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:13.988845 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:13.863037 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:13.988845 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:13.862959 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:13.988845 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:13.863071 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:04:13.988845 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:13.967493 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:13.988845 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:13.972108 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k" podUID="336c4cd8-6126-4431-930e-57f9b7ccf09d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:13.988845 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:13.984362 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 16:04:14.393970 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:14.380703 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec6894c9-57bc8d4f9-2dkbw" Nov 01 16:04:14.465870 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:14.439782 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:14.516329 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:14.439782 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:14.517335 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:14.456189 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ec6894c9-77f7f5dc8f-hstfs" Nov 01 16:04:14.517335 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:14.516978 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec6894c9-555b99b646-qgw8c" Nov 01 16:04:14.569161 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:14.569111 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:15.091009 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:15.090964 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:04:15.099136 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:15.099095 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:04:15.112064 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:15.110570 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:04:15.112447 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:15.112420 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 01 16:04:15.141598 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:15.141552 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:15.239059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:15.233610 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:04:15.239059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:15.233687 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" podUID="c10680c0-ac33-4914-931e-15c3728484bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:15.249045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:15.248099 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": context deadline exceeded" Nov 01 16:04:15.416041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:15.390407 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:04:15.416041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:15.390472 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" podUID="74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:15.651044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:15.641168 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:15.651044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:15.641519 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:16.117257 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:16.117194 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:16.121157 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:16.121117 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:16.135076 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:16.123239 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:16.135076 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:16.123539 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:04:16.246243 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:16.246189 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:04:16.246578 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:16.246543 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" podUID="41d1f840-a59c-4f46-84b0-d04c3cd1d500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:16.644042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:16.643504 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:16.649316 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:16.649271 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:16.689204 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:16.689151 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:17.152043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:17.124713 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:17.155111 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:17.132493 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:04:17.155111 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:17.133346 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:17.155111 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:17.133589 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:17.648049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:17.647116 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:17.672049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:17.670522 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:18.184054 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:18.169361 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:18.184054 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:18.173174 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 01 16:04:18.184054 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:18.174073 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:04:18.184054 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:18.174779 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:04:18.686045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:18.678156 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:18.702052 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:18.698100 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:19.190056 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:19.179606 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:19.190056 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:19.180253 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:04:19.190056 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:19.181567 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 01 16:04:19.190056 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:19.181834 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:19.190056 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:04:19.182877 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:04:19.836588 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:19.835078 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:04:19.868186 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:19.837666 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:04:19.912056 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:19.905691 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:04:20.193190 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:20.192754 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:04:20.213366 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:20.213320 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:20.427071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:20.403446 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:20.427071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:20.423708 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:20.912397 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:20.912350 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:04:20.945810 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:20.928292 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:04:21.050953 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:21.050900 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:04:21.205875 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:21.205763 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:21.232245 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:21.232200 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 01 16:04:21.399187 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:21.399119 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:21.408142 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:21.408096 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:21.549106 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:21.546611 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:04:21.651983 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:21.651573 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:04:21.651983 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:21.651644 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" podUID="3b704396-3411-40af-95fe-4b67a41ae69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:21.963046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:21.959849 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:04:21.963046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:21.960248 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:04:22.103724 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:22.103678 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:22.217335 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:22.217236 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:22.258042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:22.244925 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:22.329938 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:22.329907 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 16:04:22.413177 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:22.413125 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:22.564093 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:22.551130 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:23.059041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:22.993560 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:23.178148 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:23.178084 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:04:23.221679 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:23.221629 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:23.387063 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:23.386674 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 16:04:23.414187 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:23.414140 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:23.459951 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:23.459919 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:04:24.081503 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:24.081459 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:04:24.211071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:24.211007 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:04:24.252681 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:24.252624 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:24.388932 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:24.388902 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:04:24.406934 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:24.406420 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:04:31.402899 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:31.402860 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8624bae9-f4ffc7b67-pcczf"] Nov 01 16:04:33.118228 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:33.118191 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3664624a-68966c9b49-8hqvg"] Nov 01 16:04:33.142686 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:33.142640 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3664624a-64bd765999-zjrqf"] Nov 01 16:04:36.409077 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:36.409031 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-8624bae9-f4ffc7b67-pcczf" podUID="2d567652-04f7-416f-b812-85a8ed997863" containerName="payload-pre-processing" containerID="cri-o://8e7aa4a828dd04366f642c4cc5b3a009fa5883f726b2739391253385fa2062e8" gracePeriod=25 Nov 01 16:04:36.473763 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:36.473722 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8624bae9-655cf6c8-wngkz"] Nov 01 16:04:36.474073 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:36.473996 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8624bae9-655cf6c8-wngkz" podUID="2e688a3b-8de5-45bf-9475-942fd0d99c70" containerName="maas-api" containerID="cri-o://ed9c2fcf7f0ec3cf38ec61e63f1e3bc9e840bf909c49ee1d66a68090b55e3029" gracePeriod=30 Nov 01 16:04:37.084064 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:37.082797 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8624bae9-655cf6c8-wngkz" podUID="2e688a3b-8de5-45bf-9475-942fd0d99c70" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.85:8443/health\": dial tcp 10.134.0.85:8443: connect: connection refused" Nov 01 16:04:37.216042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:37.215556 4363 generic.go:356] "Generic (PLEG): container finished" podID="2e688a3b-8de5-45bf-9475-942fd0d99c70" containerID="ed9c2fcf7f0ec3cf38ec61e63f1e3bc9e840bf909c49ee1d66a68090b55e3029" exitCode=0 Nov 01 16:04:37.302043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:37.285636 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8624bae9-655cf6c8-wngkz" event={"ID":"2e688a3b-8de5-45bf-9475-942fd0d99c70","Type":"ContainerDied","Data":"ed9c2fcf7f0ec3cf38ec61e63f1e3bc9e840bf909c49ee1d66a68090b55e3029"} Nov 01 16:04:37.370688 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:37.370633 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8624bae9-f4ffc7b67-pcczf" event={"ID":"2d567652-04f7-416f-b812-85a8ed997863","Type":"ContainerDied","Data":"8e7aa4a828dd04366f642c4cc5b3a009fa5883f726b2739391253385fa2062e8"} Nov 01 16:04:37.370969 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:37.370366 4363 generic.go:356] "Generic (PLEG): container finished" podID="2d567652-04f7-416f-b812-85a8ed997863" containerID="8e7aa4a828dd04366f642c4cc5b3a009fa5883f726b2739391253385fa2062e8" exitCode=0 Nov 01 16:04:37.755469 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:37.755434 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8624bae9-f4ffc7b67-pcczf" Nov 01 16:04:37.759838 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:37.759817 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8624bae9-655cf6c8-wngkz" Nov 01 16:04:37.915098 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:37.912878 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e688a3b-8de5-45bf-9475-942fd0d99c70-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e688a3b-8de5-45bf-9475-942fd0d99c70-maas-api-tls\") pod \"2e688a3b-8de5-45bf-9475-942fd0d99c70\" (UID: \"2e688a3b-8de5-45bf-9475-942fd0d99c70\") " Nov 01 16:04:37.915098 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:37.913005 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d567652-04f7-416f-b812-85a8ed997863-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d567652-04f7-416f-b812-85a8ed997863-plugins-config-volume\") pod \"2d567652-04f7-416f-b812-85a8ed997863\" (UID: \"2d567652-04f7-416f-b812-85a8ed997863\") " Nov 01 16:04:37.915098 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:37.913053 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e688a3b-8de5-45bf-9475-942fd0d99c70-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e688a3b-8de5-45bf-9475-942fd0d99c70-openshift-service-ca\") pod \"2e688a3b-8de5-45bf-9475-942fd0d99c70\" (UID: \"2e688a3b-8de5-45bf-9475-942fd0d99c70\") " Nov 01 16:04:37.915098 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:37.913198 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d567652-04f7-416f-b812-85a8ed997863-kube-api-access-65l5s\" (UniqueName: \"kubernetes.io/projected/2d567652-04f7-416f-b812-85a8ed997863-kube-api-access-65l5s\") pod \"2d567652-04f7-416f-b812-85a8ed997863\" (UID: \"2d567652-04f7-416f-b812-85a8ed997863\") " Nov 01 16:04:37.915098 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:37.913336 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e688a3b-8de5-45bf-9475-942fd0d99c70-kube-api-access-2s6zx\" (UniqueName: \"kubernetes.io/projected/2e688a3b-8de5-45bf-9475-942fd0d99c70-kube-api-access-2s6zx\") pod \"2e688a3b-8de5-45bf-9475-942fd0d99c70\" (UID: \"2e688a3b-8de5-45bf-9475-942fd0d99c70\") " Nov 01 16:04:37.915098 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:37.913400 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e688a3b-8de5-45bf-9475-942fd0d99c70-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e688a3b-8de5-45bf-9475-942fd0d99c70-metrics-certs\") pod \"2e688a3b-8de5-45bf-9475-942fd0d99c70\" (UID: \"2e688a3b-8de5-45bf-9475-942fd0d99c70\") " Nov 01 16:04:37.923376 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:37.921100 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d567652-04f7-416f-b812-85a8ed997863-plugins-config-volume" pod "2d567652-04f7-416f-b812-85a8ed997863" (UID: "2d567652-04f7-416f-b812-85a8ed997863"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:04:37.924447 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:37.924420 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e688a3b-8de5-45bf-9475-942fd0d99c70-openshift-service-ca" pod "2e688a3b-8de5-45bf-9475-942fd0d99c70" (UID: "2e688a3b-8de5-45bf-9475-942fd0d99c70"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:04:37.926416 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:37.926387 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d567652-04f7-416f-b812-85a8ed997863-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:37.926538 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:37.926420 4363 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e688a3b-8de5-45bf-9475-942fd0d99c70-openshift-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:37.927519 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:37.927496 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e688a3b-8de5-45bf-9475-942fd0d99c70-kube-api-access-2s6zx" pod "2e688a3b-8de5-45bf-9475-942fd0d99c70" (UID: "2e688a3b-8de5-45bf-9475-942fd0d99c70"). InnerVolumeSpecName "kube-api-access-2s6zx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:04:37.927743 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:37.927718 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e688a3b-8de5-45bf-9475-942fd0d99c70-maas-api-tls" pod "2e688a3b-8de5-45bf-9475-942fd0d99c70" (UID: "2e688a3b-8de5-45bf-9475-942fd0d99c70"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:04:37.928010 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:37.927991 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d567652-04f7-416f-b812-85a8ed997863-kube-api-access-65l5s" pod "2d567652-04f7-416f-b812-85a8ed997863" (UID: "2d567652-04f7-416f-b812-85a8ed997863"). InnerVolumeSpecName "kube-api-access-65l5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:04:37.930797 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:37.930771 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e688a3b-8de5-45bf-9475-942fd0d99c70-metrics-certs" pod "2e688a3b-8de5-45bf-9475-942fd0d99c70" (UID: "2e688a3b-8de5-45bf-9475-942fd0d99c70"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:04:38.027676 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.027636 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65l5s\" (UniqueName: \"kubernetes.io/projected/2d567652-04f7-416f-b812-85a8ed997863-kube-api-access-65l5s\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:38.027676 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.027665 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s6zx\" (UniqueName: \"kubernetes.io/projected/2e688a3b-8de5-45bf-9475-942fd0d99c70-kube-api-access-2s6zx\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:38.027676 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.027675 4363 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e688a3b-8de5-45bf-9475-942fd0d99c70-metrics-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:38.027676 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.027685 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e688a3b-8de5-45bf-9475-942fd0d99c70-maas-api-tls\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:38.059775 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.059742 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54"] Nov 01 16:04:38.060649 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.060628 4363 cpu_manager.go:414] "containerMap: removing container" podUID="2d567652-04f7-416f-b812-85a8ed997863" containerName="payload-pre-processing" Nov 01 16:04:38.060649 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.060649 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d567652-04f7-416f-b812-85a8ed997863" containerName="payload-pre-processing" Nov 01 16:04:38.060761 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.060662 4363 cpu_manager.go:414] "containerMap: removing container" podUID="2e688a3b-8de5-45bf-9475-942fd0d99c70" containerName="maas-api" Nov 01 16:04:38.060761 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.060667 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e688a3b-8de5-45bf-9475-942fd0d99c70" containerName="maas-api" Nov 01 16:04:38.060852 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.060788 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d567652-04f7-416f-b812-85a8ed997863" containerName="payload-pre-processing" Nov 01 16:04:38.060852 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.060795 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e688a3b-8de5-45bf-9475-942fd0d99c70" containerName="maas-api" Nov 01 16:04:38.065046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.064952 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54" Nov 01 16:04:38.071445 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.071419 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-474cab5e-openshift-default-dockercfg-rgl9g\"" Nov 01 16:04:38.077606 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.077579 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54"] Nov 01 16:04:38.120103 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.120058 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-3664624a-68966c9b49-8hqvg" podUID="07ec09d0-8bfa-44a9-9667-1f4e5daa5c40" containerName="payload-processing" containerID="cri-o://4739f1cd00525f4dd9b40a081380b9c6cf72f27263a7bd56f824baecf7801dca" gracePeriod=25 Nov 01 16:04:38.144243 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.144211 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3664624a-64bd765999-zjrqf" podUID="c49f02cc-3d34-4350-9261-320ae44d8fd4" containerName="payload-pre-processing" containerID="cri-o://eecb5518af735a0be57cb22c508b4ddf3635a14e8c494d59a69f9311a794e94c" gracePeriod=25 Nov 01 16:04:38.254039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.231440 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4504fd0-ac35-4f52-8403-beb7c42daa96-istio-data\") pod \"e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54\" (UID: \"a4504fd0-ac35-4f52-8403-beb7c42daa96\") " pod="openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54" Nov 01 16:04:38.254493 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.254461 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4504fd0-ac35-4f52-8403-beb7c42daa96-credential-socket\") pod \"e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54\" (UID: \"a4504fd0-ac35-4f52-8403-beb7c42daa96\") " pod="openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54" Nov 01 16:04:38.254750 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.254731 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4504fd0-ac35-4f52-8403-beb7c42daa96-istio-token\") pod \"e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54\" (UID: \"a4504fd0-ac35-4f52-8403-beb7c42daa96\") " pod="openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54" Nov 01 16:04:38.254982 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.254966 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4504fd0-ac35-4f52-8403-beb7c42daa96-istiod-ca-cert\") pod \"e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54\" (UID: \"a4504fd0-ac35-4f52-8403-beb7c42daa96\") " pod="openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54" Nov 01 16:04:38.255353 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.255332 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4504fd0-ac35-4f52-8403-beb7c42daa96-workload-socket\") pod \"e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54\" (UID: \"a4504fd0-ac35-4f52-8403-beb7c42daa96\") " pod="openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54" Nov 01 16:04:38.255543 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.255526 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4504fd0-ac35-4f52-8403-beb7c42daa96-istio-envoy\") pod \"e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54\" (UID: \"a4504fd0-ac35-4f52-8403-beb7c42daa96\") " pod="openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54" Nov 01 16:04:38.255698 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.255681 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4504fd0-ac35-4f52-8403-beb7c42daa96-istio-podinfo\") pod \"e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54\" (UID: \"a4504fd0-ac35-4f52-8403-beb7c42daa96\") " pod="openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54" Nov 01 16:04:38.255947 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.255930 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4504fd0-ac35-4f52-8403-beb7c42daa96-workload-certs\") pod \"e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54\" (UID: \"a4504fd0-ac35-4f52-8403-beb7c42daa96\") " pod="openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54" Nov 01 16:04:38.256127 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.256110 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chrdj\" (UniqueName: \"kubernetes.io/projected/a4504fd0-ac35-4f52-8403-beb7c42daa96-kube-api-access-chrdj\") pod \"e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54\" (UID: \"a4504fd0-ac35-4f52-8403-beb7c42daa96\") " pod="openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54" Nov 01 16:04:38.371333 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.370356 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4504fd0-ac35-4f52-8403-beb7c42daa96-credential-socket\") pod \"e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54\" (UID: \"a4504fd0-ac35-4f52-8403-beb7c42daa96\") " pod="openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54" Nov 01 16:04:38.371333 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.370428 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4504fd0-ac35-4f52-8403-beb7c42daa96-istio-token\") pod \"e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54\" (UID: \"a4504fd0-ac35-4f52-8403-beb7c42daa96\") " pod="openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54" Nov 01 16:04:38.371333 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.370509 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4504fd0-ac35-4f52-8403-beb7c42daa96-istiod-ca-cert\") pod \"e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54\" (UID: \"a4504fd0-ac35-4f52-8403-beb7c42daa96\") " pod="openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54" Nov 01 16:04:38.371333 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.370635 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4504fd0-ac35-4f52-8403-beb7c42daa96-workload-socket\") pod \"e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54\" (UID: \"a4504fd0-ac35-4f52-8403-beb7c42daa96\") " pod="openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54" Nov 01 16:04:38.371333 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.370664 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4504fd0-ac35-4f52-8403-beb7c42daa96-istio-envoy\") pod \"e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54\" (UID: \"a4504fd0-ac35-4f52-8403-beb7c42daa96\") " pod="openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54" Nov 01 16:04:38.371333 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.370685 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4504fd0-ac35-4f52-8403-beb7c42daa96-istio-podinfo\") pod \"e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54\" (UID: \"a4504fd0-ac35-4f52-8403-beb7c42daa96\") " pod="openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54" Nov 01 16:04:38.371333 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.370770 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4504fd0-ac35-4f52-8403-beb7c42daa96-workload-certs\") pod \"e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54\" (UID: \"a4504fd0-ac35-4f52-8403-beb7c42daa96\") " pod="openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54" Nov 01 16:04:38.371333 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.370799 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chrdj\" (UniqueName: \"kubernetes.io/projected/a4504fd0-ac35-4f52-8403-beb7c42daa96-kube-api-access-chrdj\") pod \"e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54\" (UID: \"a4504fd0-ac35-4f52-8403-beb7c42daa96\") " pod="openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54" Nov 01 16:04:38.371333 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.370888 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4504fd0-ac35-4f52-8403-beb7c42daa96-istio-data\") pod \"e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54\" (UID: \"a4504fd0-ac35-4f52-8403-beb7c42daa96\") " pod="openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54" Nov 01 16:04:38.371803 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.371370 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4504fd0-ac35-4f52-8403-beb7c42daa96-istio-data\") pod \"e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54\" (UID: \"a4504fd0-ac35-4f52-8403-beb7c42daa96\") " pod="openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54" Nov 01 16:04:38.371803 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.371790 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4504fd0-ac35-4f52-8403-beb7c42daa96-credential-socket\") pod \"e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54\" (UID: \"a4504fd0-ac35-4f52-8403-beb7c42daa96\") " pod="openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54" Nov 01 16:04:38.376036 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.372730 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4504fd0-ac35-4f52-8403-beb7c42daa96-istiod-ca-cert\") pod \"e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54\" (UID: \"a4504fd0-ac35-4f52-8403-beb7c42daa96\") " pod="openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54" Nov 01 16:04:38.376036 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.372976 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4504fd0-ac35-4f52-8403-beb7c42daa96-workload-socket\") pod \"e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54\" (UID: \"a4504fd0-ac35-4f52-8403-beb7c42daa96\") " pod="openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54" Nov 01 16:04:38.376036 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.374329 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4504fd0-ac35-4f52-8403-beb7c42daa96-workload-certs\") pod \"e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54\" (UID: \"a4504fd0-ac35-4f52-8403-beb7c42daa96\") " pod="openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54" Nov 01 16:04:38.412084 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.401518 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4504fd0-ac35-4f52-8403-beb7c42daa96-istio-token\") pod \"e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54\" (UID: \"a4504fd0-ac35-4f52-8403-beb7c42daa96\") " pod="openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54" Nov 01 16:04:38.420100 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.419967 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4504fd0-ac35-4f52-8403-beb7c42daa96-istio-envoy\") pod \"e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54\" (UID: \"a4504fd0-ac35-4f52-8403-beb7c42daa96\") " pod="openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54" Nov 01 16:04:38.448322 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.448235 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4504fd0-ac35-4f52-8403-beb7c42daa96-istio-podinfo\") pod \"e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54\" (UID: \"a4504fd0-ac35-4f52-8403-beb7c42daa96\") " pod="openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54" Nov 01 16:04:38.458050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.449249 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chrdj\" (UniqueName: \"kubernetes.io/projected/a4504fd0-ac35-4f52-8403-beb7c42daa96-kube-api-access-chrdj\") pod \"e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54\" (UID: \"a4504fd0-ac35-4f52-8403-beb7c42daa96\") " pod="openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54" Nov 01 16:04:38.490877 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.490637 4363 generic.go:356] "Generic (PLEG): container finished" podID="c49f02cc-3d34-4350-9261-320ae44d8fd4" containerID="eecb5518af735a0be57cb22c508b4ddf3635a14e8c494d59a69f9311a794e94c" exitCode=0 Nov 01 16:04:38.490877 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.490809 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3664624a-64bd765999-zjrqf" event={"ID":"c49f02cc-3d34-4350-9261-320ae44d8fd4","Type":"ContainerDied","Data":"eecb5518af735a0be57cb22c508b4ddf3635a14e8c494d59a69f9311a794e94c"} Nov 01 16:04:38.510286 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.510248 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8624bae9-655cf6c8-wngkz" event={"ID":"2e688a3b-8de5-45bf-9475-942fd0d99c70","Type":"ContainerDied","Data":"0c31fdba60d5445a8683fd214018126dd4423d8305f5d3445577a63eacdf2e62"} Nov 01 16:04:38.510567 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.510544 4363 scope.go:122] "RemoveContainer" containerID="ed9c2fcf7f0ec3cf38ec61e63f1e3bc9e840bf909c49ee1d66a68090b55e3029" Nov 01 16:04:38.510839 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.510814 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8624bae9-655cf6c8-wngkz" Nov 01 16:04:38.535045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.529545 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8624bae9-f4ffc7b67-pcczf" event={"ID":"2d567652-04f7-416f-b812-85a8ed997863","Type":"ContainerDied","Data":"bde67bf21bd0155614ee587fb8c4b78dac40e94b039d8708cf0724d9313c6d66"} Nov 01 16:04:38.535045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.529668 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8624bae9-f4ffc7b67-pcczf" Nov 01 16:04:38.674708 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.674576 4363 scope.go:122] "RemoveContainer" containerID="8e7aa4a828dd04366f642c4cc5b3a009fa5883f726b2739391253385fa2062e8" Nov 01 16:04:38.685650 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.683296 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54" Nov 01 16:04:38.697767 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.697711 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8624bae9-655cf6c8-wngkz"] Nov 01 16:04:38.743230 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.739189 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8624bae9-655cf6c8-wngkz"] Nov 01 16:04:38.747142 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.747112 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8624bae9-f4ffc7b67-pcczf"] Nov 01 16:04:38.755166 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.755135 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8624bae9-f4ffc7b67-pcczf"] Nov 01 16:04:38.821230 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.820083 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d567652-04f7-416f-b812-85a8ed997863" path="/var/lib/kubelet/pods/2d567652-04f7-416f-b812-85a8ed997863/volumes" Nov 01 16:04:38.821230 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.821008 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e688a3b-8de5-45bf-9475-942fd0d99c70" path="/var/lib/kubelet/pods/2e688a3b-8de5-45bf-9475-942fd0d99c70/volumes" Nov 01 16:04:38.876809 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:38.876775 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3664624a-64bd765999-zjrqf" Nov 01 16:04:39.003832 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:39.003738 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c49f02cc-3d34-4350-9261-320ae44d8fd4-kube-api-access-w6jcj\" (UniqueName: \"kubernetes.io/projected/c49f02cc-3d34-4350-9261-320ae44d8fd4-kube-api-access-w6jcj\") pod \"c49f02cc-3d34-4350-9261-320ae44d8fd4\" (UID: \"c49f02cc-3d34-4350-9261-320ae44d8fd4\") " Nov 01 16:04:39.008704 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:39.008673 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c49f02cc-3d34-4350-9261-320ae44d8fd4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c49f02cc-3d34-4350-9261-320ae44d8fd4-plugins-config-volume\") pod \"c49f02cc-3d34-4350-9261-320ae44d8fd4\" (UID: \"c49f02cc-3d34-4350-9261-320ae44d8fd4\") " Nov 01 16:04:39.040374 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:39.009330 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c49f02cc-3d34-4350-9261-320ae44d8fd4-plugins-config-volume" pod "c49f02cc-3d34-4350-9261-320ae44d8fd4" (UID: "c49f02cc-3d34-4350-9261-320ae44d8fd4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:04:39.081595 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:39.075912 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c49f02cc-3d34-4350-9261-320ae44d8fd4-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:39.097321 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:39.094642 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c49f02cc-3d34-4350-9261-320ae44d8fd4-kube-api-access-w6jcj" pod "c49f02cc-3d34-4350-9261-320ae44d8fd4" (UID: "c49f02cc-3d34-4350-9261-320ae44d8fd4"). InnerVolumeSpecName "kube-api-access-w6jcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:04:39.100928 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:39.100778 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54"] Nov 01 16:04:39.105218 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:39.104624 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:04:39.105218 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:39.104713 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:04:39.105218 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:39.104755 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:04:39.145420 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:39.145397 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-3664624a-68966c9b49-8hqvg" Nov 01 16:04:39.178217 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:39.178190 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6jcj\" (UniqueName: \"kubernetes.io/projected/c49f02cc-3d34-4350-9261-320ae44d8fd4-kube-api-access-w6jcj\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:39.279565 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:39.279476 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07ec09d0-8bfa-44a9-9667-1f4e5daa5c40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07ec09d0-8bfa-44a9-9667-1f4e5daa5c40-plugins-config-volume\") pod \"07ec09d0-8bfa-44a9-9667-1f4e5daa5c40\" (UID: \"07ec09d0-8bfa-44a9-9667-1f4e5daa5c40\") " Nov 01 16:04:39.279565 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:39.279553 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07ec09d0-8bfa-44a9-9667-1f4e5daa5c40-kube-api-access-fdcvb\" (UniqueName: \"kubernetes.io/projected/07ec09d0-8bfa-44a9-9667-1f4e5daa5c40-kube-api-access-fdcvb\") pod \"07ec09d0-8bfa-44a9-9667-1f4e5daa5c40\" (UID: \"07ec09d0-8bfa-44a9-9667-1f4e5daa5c40\") " Nov 01 16:04:39.280161 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:39.280128 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07ec09d0-8bfa-44a9-9667-1f4e5daa5c40-plugins-config-volume" pod "07ec09d0-8bfa-44a9-9667-1f4e5daa5c40" (UID: "07ec09d0-8bfa-44a9-9667-1f4e5daa5c40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:04:39.280743 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:39.280718 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07ec09d0-8bfa-44a9-9667-1f4e5daa5c40-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:39.282307 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:39.282279 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07ec09d0-8bfa-44a9-9667-1f4e5daa5c40-kube-api-access-fdcvb" pod "07ec09d0-8bfa-44a9-9667-1f4e5daa5c40" (UID: "07ec09d0-8bfa-44a9-9667-1f4e5daa5c40"). InnerVolumeSpecName "kube-api-access-fdcvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:04:39.383147 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:39.383108 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdcvb\" (UniqueName: \"kubernetes.io/projected/07ec09d0-8bfa-44a9-9667-1f4e5daa5c40-kube-api-access-fdcvb\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:39.546797 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:39.546687 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54" event={"ID":"a4504fd0-ac35-4f52-8403-beb7c42daa96","Type":"ContainerStarted","Data":"9a427625ec052056690996064900fc5e0eff4838367e779abb16c2fd23883b40"} Nov 01 16:04:39.547099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:39.547076 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54" event={"ID":"a4504fd0-ac35-4f52-8403-beb7c42daa96","Type":"ContainerStarted","Data":"6a768033148c0b8460589a7b3000ca9f7d8e4109f7d53e8e25a7fbfbe76396fb"} Nov 01 16:04:39.551123 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:39.549844 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3664624a-64bd765999-zjrqf" event={"ID":"c49f02cc-3d34-4350-9261-320ae44d8fd4","Type":"ContainerDied","Data":"832e264aba94e32eb06bc4f5132cd7fb3f1a73a7b7f2db175a354b221e81e7f1"} Nov 01 16:04:39.551123 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:39.549900 4363 scope.go:122] "RemoveContainer" containerID="eecb5518af735a0be57cb22c508b4ddf3635a14e8c494d59a69f9311a794e94c" Nov 01 16:04:39.551123 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:39.550008 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3664624a-64bd765999-zjrqf" Nov 01 16:04:39.558978 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:39.558947 4363 generic.go:356] "Generic (PLEG): container finished" podID="07ec09d0-8bfa-44a9-9667-1f4e5daa5c40" containerID="4739f1cd00525f4dd9b40a081380b9c6cf72f27263a7bd56f824baecf7801dca" exitCode=0 Nov 01 16:04:39.559132 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:39.559067 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-3664624a-68966c9b49-8hqvg" event={"ID":"07ec09d0-8bfa-44a9-9667-1f4e5daa5c40","Type":"ContainerDied","Data":"4739f1cd00525f4dd9b40a081380b9c6cf72f27263a7bd56f824baecf7801dca"} Nov 01 16:04:39.559132 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:39.559112 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-3664624a-68966c9b49-8hqvg" event={"ID":"07ec09d0-8bfa-44a9-9667-1f4e5daa5c40","Type":"ContainerDied","Data":"8817bb252093128fc2f43f957daf9bd0812309aa657e1b68c89311cd2373dc76"} Nov 01 16:04:39.559258 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:39.559178 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-3664624a-68966c9b49-8hqvg" Nov 01 16:04:39.568062 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:39.568044 4363 scope.go:122] "RemoveContainer" containerID="4739f1cd00525f4dd9b40a081380b9c6cf72f27263a7bd56f824baecf7801dca" Nov 01 16:04:39.575885 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:39.575835 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54" podStartSLOduration=1.575819372 podStartE2EDuration="1.575819372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:04:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:04:39.569543728 +0000 UTC m=+1012.901281378" watchObservedRunningTime="2026-11-01 16:04:39.575819372 +0000 UTC m=+1012.907557022" Nov 01 16:04:39.579664 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:39.579646 4363 scope.go:122] "RemoveContainer" containerID="4739f1cd00525f4dd9b40a081380b9c6cf72f27263a7bd56f824baecf7801dca" Nov 01 16:04:39.579966 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:04:39.579944 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4739f1cd00525f4dd9b40a081380b9c6cf72f27263a7bd56f824baecf7801dca\": container with ID starting with 4739f1cd00525f4dd9b40a081380b9c6cf72f27263a7bd56f824baecf7801dca not found: ID does not exist" containerID="4739f1cd00525f4dd9b40a081380b9c6cf72f27263a7bd56f824baecf7801dca" Nov 01 16:04:39.580069 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:39.579975 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4739f1cd00525f4dd9b40a081380b9c6cf72f27263a7bd56f824baecf7801dca"} err="failed to get container status \"4739f1cd00525f4dd9b40a081380b9c6cf72f27263a7bd56f824baecf7801dca\": rpc error: code = NotFound desc = could not find container \"4739f1cd00525f4dd9b40a081380b9c6cf72f27263a7bd56f824baecf7801dca\": container with ID starting with 4739f1cd00525f4dd9b40a081380b9c6cf72f27263a7bd56f824baecf7801dca not found: ID does not exist" Nov 01 16:04:39.597210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:39.597183 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3664624a-64bd765999-zjrqf"] Nov 01 16:04:39.604025 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:39.603990 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3664624a-64bd765999-zjrqf"] Nov 01 16:04:39.610607 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:39.610584 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3664624a-68966c9b49-8hqvg"] Nov 01 16:04:39.616185 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:39.616165 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3664624a-68966c9b49-8hqvg"] Nov 01 16:04:39.656067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:39.656042 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ec6894c9-77f7f5dc8f-hstfs"] Nov 01 16:04:39.668595 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:39.668570 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ec6894c9-57bc8d4f9-2dkbw"] Nov 01 16:04:39.684396 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:39.684371 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54" Nov 01 16:04:40.707650 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:40.707328 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54" Nov 01 16:04:40.886397 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:40.886353 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07ec09d0-8bfa-44a9-9667-1f4e5daa5c40" path="/var/lib/kubelet/pods/07ec09d0-8bfa-44a9-9667-1f4e5daa5c40/volumes" Nov 01 16:04:40.892689 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:40.890508 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c49f02cc-3d34-4350-9261-320ae44d8fd4" path="/var/lib/kubelet/pods/c49f02cc-3d34-4350-9261-320ae44d8fd4/volumes" Nov 01 16:04:41.530568 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:41.530531 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9"] Nov 01 16:04:41.609934 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:41.609897 4363 cpu_manager.go:414] "containerMap: removing container" podUID="07ec09d0-8bfa-44a9-9667-1f4e5daa5c40" containerName="payload-processing" Nov 01 16:04:41.611989 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:41.611964 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07ec09d0-8bfa-44a9-9667-1f4e5daa5c40" containerName="payload-processing" Nov 01 16:04:41.612906 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:41.612890 4363 cpu_manager.go:414] "containerMap: removing container" podUID="c49f02cc-3d34-4350-9261-320ae44d8fd4" containerName="payload-pre-processing" Nov 01 16:04:41.613061 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:41.613046 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c49f02cc-3d34-4350-9261-320ae44d8fd4" containerName="payload-pre-processing" Nov 01 16:04:41.613423 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:41.613407 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="c49f02cc-3d34-4350-9261-320ae44d8fd4" containerName="payload-pre-processing" Nov 01 16:04:41.613530 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:41.613520 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="07ec09d0-8bfa-44a9-9667-1f4e5daa5c40" containerName="payload-processing" Nov 01 16:04:42.073645 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:42.010309 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9"] Nov 01 16:04:42.073645 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:42.010387 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54" Nov 01 16:04:42.073645 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:42.010487 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9" Nov 01 16:04:42.119065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:42.106049 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54" Nov 01 16:04:42.120634 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:42.120602 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-9fd01d20-openshift-default-dockercfg-xtkj5\"" Nov 01 16:04:42.124300 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:42.124264 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86b37143-20ce-45d3-90d7-45b451012c14-istiod-ca-cert\") pod \"e2e-derive-9fd01d20-openshift-default-744446d849-cstp9\" (UID: \"86b37143-20ce-45d3-90d7-45b451012c14\") " pod="openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9" Nov 01 16:04:42.127122 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:42.127097 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n6x4\" (UniqueName: \"kubernetes.io/projected/86b37143-20ce-45d3-90d7-45b451012c14-kube-api-access-5n6x4\") pod \"e2e-derive-9fd01d20-openshift-default-744446d849-cstp9\" (UID: \"86b37143-20ce-45d3-90d7-45b451012c14\") " pod="openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9" Nov 01 16:04:42.127312 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:42.127294 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86b37143-20ce-45d3-90d7-45b451012c14-istio-token\") pod \"e2e-derive-9fd01d20-openshift-default-744446d849-cstp9\" (UID: \"86b37143-20ce-45d3-90d7-45b451012c14\") " pod="openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9" Nov 01 16:04:42.127451 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:42.127434 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86b37143-20ce-45d3-90d7-45b451012c14-istio-envoy\") pod \"e2e-derive-9fd01d20-openshift-default-744446d849-cstp9\" (UID: \"86b37143-20ce-45d3-90d7-45b451012c14\") " pod="openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9" Nov 01 16:04:42.127585 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:42.127569 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b37143-20ce-45d3-90d7-45b451012c14-credential-socket\") pod \"e2e-derive-9fd01d20-openshift-default-744446d849-cstp9\" (UID: \"86b37143-20ce-45d3-90d7-45b451012c14\") " pod="openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9" Nov 01 16:04:42.127819 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:42.127802 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86b37143-20ce-45d3-90d7-45b451012c14-workload-certs\") pod \"e2e-derive-9fd01d20-openshift-default-744446d849-cstp9\" (UID: \"86b37143-20ce-45d3-90d7-45b451012c14\") " pod="openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9" Nov 01 16:04:42.127964 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:42.127947 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86b37143-20ce-45d3-90d7-45b451012c14-istio-data\") pod \"e2e-derive-9fd01d20-openshift-default-744446d849-cstp9\" (UID: \"86b37143-20ce-45d3-90d7-45b451012c14\") " pod="openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9" Nov 01 16:04:42.128100 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:42.128082 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86b37143-20ce-45d3-90d7-45b451012c14-istio-podinfo\") pod \"e2e-derive-9fd01d20-openshift-default-744446d849-cstp9\" (UID: \"86b37143-20ce-45d3-90d7-45b451012c14\") " pod="openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9" Nov 01 16:04:42.128323 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:42.128303 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b37143-20ce-45d3-90d7-45b451012c14-workload-socket\") pod \"e2e-derive-9fd01d20-openshift-default-744446d849-cstp9\" (UID: \"86b37143-20ce-45d3-90d7-45b451012c14\") " pod="openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9" Nov 01 16:04:42.284055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:42.268385 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86b37143-20ce-45d3-90d7-45b451012c14-istiod-ca-cert\") pod \"e2e-derive-9fd01d20-openshift-default-744446d849-cstp9\" (UID: \"86b37143-20ce-45d3-90d7-45b451012c14\") " pod="openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9" Nov 01 16:04:42.284569 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:42.284532 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n6x4\" (UniqueName: \"kubernetes.io/projected/86b37143-20ce-45d3-90d7-45b451012c14-kube-api-access-5n6x4\") pod \"e2e-derive-9fd01d20-openshift-default-744446d849-cstp9\" (UID: \"86b37143-20ce-45d3-90d7-45b451012c14\") " pod="openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9" Nov 01 16:04:42.284784 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:42.284763 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86b37143-20ce-45d3-90d7-45b451012c14-istio-token\") pod \"e2e-derive-9fd01d20-openshift-default-744446d849-cstp9\" (UID: \"86b37143-20ce-45d3-90d7-45b451012c14\") " pod="openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9" Nov 01 16:04:42.284914 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:42.284901 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86b37143-20ce-45d3-90d7-45b451012c14-istio-envoy\") pod \"e2e-derive-9fd01d20-openshift-default-744446d849-cstp9\" (UID: \"86b37143-20ce-45d3-90d7-45b451012c14\") " pod="openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9" Nov 01 16:04:42.285053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:42.285039 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b37143-20ce-45d3-90d7-45b451012c14-credential-socket\") pod \"e2e-derive-9fd01d20-openshift-default-744446d849-cstp9\" (UID: \"86b37143-20ce-45d3-90d7-45b451012c14\") " pod="openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9" Nov 01 16:04:42.285345 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:42.285330 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86b37143-20ce-45d3-90d7-45b451012c14-workload-certs\") pod \"e2e-derive-9fd01d20-openshift-default-744446d849-cstp9\" (UID: \"86b37143-20ce-45d3-90d7-45b451012c14\") " pod="openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9" Nov 01 16:04:42.285489 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:42.285476 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86b37143-20ce-45d3-90d7-45b451012c14-istio-data\") pod \"e2e-derive-9fd01d20-openshift-default-744446d849-cstp9\" (UID: \"86b37143-20ce-45d3-90d7-45b451012c14\") " pod="openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9" Nov 01 16:04:42.285586 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:42.285573 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86b37143-20ce-45d3-90d7-45b451012c14-istio-podinfo\") pod \"e2e-derive-9fd01d20-openshift-default-744446d849-cstp9\" (UID: \"86b37143-20ce-45d3-90d7-45b451012c14\") " pod="openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9" Nov 01 16:04:42.285786 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:42.285773 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b37143-20ce-45d3-90d7-45b451012c14-workload-socket\") pod \"e2e-derive-9fd01d20-openshift-default-744446d849-cstp9\" (UID: \"86b37143-20ce-45d3-90d7-45b451012c14\") " pod="openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9" Nov 01 16:04:42.286474 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:42.286453 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b37143-20ce-45d3-90d7-45b451012c14-workload-socket\") pod \"e2e-derive-9fd01d20-openshift-default-744446d849-cstp9\" (UID: \"86b37143-20ce-45d3-90d7-45b451012c14\") " pod="openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9" Nov 01 16:04:42.287760 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:42.287736 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86b37143-20ce-45d3-90d7-45b451012c14-workload-certs\") pod \"e2e-derive-9fd01d20-openshift-default-744446d849-cstp9\" (UID: \"86b37143-20ce-45d3-90d7-45b451012c14\") " pod="openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9" Nov 01 16:04:42.296105 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:42.292246 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b37143-20ce-45d3-90d7-45b451012c14-credential-socket\") pod \"e2e-derive-9fd01d20-openshift-default-744446d849-cstp9\" (UID: \"86b37143-20ce-45d3-90d7-45b451012c14\") " pod="openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9" Nov 01 16:04:42.333249 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:42.333138 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86b37143-20ce-45d3-90d7-45b451012c14-istio-data\") pod \"e2e-derive-9fd01d20-openshift-default-744446d849-cstp9\" (UID: \"86b37143-20ce-45d3-90d7-45b451012c14\") " pod="openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9" Nov 01 16:04:42.365318 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:42.365272 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86b37143-20ce-45d3-90d7-45b451012c14-istiod-ca-cert\") pod \"e2e-derive-9fd01d20-openshift-default-744446d849-cstp9\" (UID: \"86b37143-20ce-45d3-90d7-45b451012c14\") " pod="openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9" Nov 01 16:04:42.441049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:42.440392 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n6x4\" (UniqueName: \"kubernetes.io/projected/86b37143-20ce-45d3-90d7-45b451012c14-kube-api-access-5n6x4\") pod \"e2e-derive-9fd01d20-openshift-default-744446d849-cstp9\" (UID: \"86b37143-20ce-45d3-90d7-45b451012c14\") " pod="openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9" Nov 01 16:04:42.441837 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:42.441808 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86b37143-20ce-45d3-90d7-45b451012c14-istio-podinfo\") pod \"e2e-derive-9fd01d20-openshift-default-744446d849-cstp9\" (UID: \"86b37143-20ce-45d3-90d7-45b451012c14\") " pod="openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9" Nov 01 16:04:42.526175 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:42.458623 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86b37143-20ce-45d3-90d7-45b451012c14-istio-token\") pod \"e2e-derive-9fd01d20-openshift-default-744446d849-cstp9\" (UID: \"86b37143-20ce-45d3-90d7-45b451012c14\") " pod="openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9" Nov 01 16:04:42.526175 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:42.461870 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86b37143-20ce-45d3-90d7-45b451012c14-istio-envoy\") pod \"e2e-derive-9fd01d20-openshift-default-744446d849-cstp9\" (UID: \"86b37143-20ce-45d3-90d7-45b451012c14\") " pod="openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9" Nov 01 16:04:42.758072 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:42.757586 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9" Nov 01 16:04:43.040096 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:42.995802 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-474cab5e-79b66887b7-brn4j"] Nov 01 16:04:43.073054 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:43.046091 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-474cab5e-79b66887b7-brn4j" Nov 01 16:04:43.073054 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:43.054415 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-474cab5e\"" Nov 01 16:04:43.117039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:43.105555 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72cpp\" (UniqueName: \"kubernetes.io/projected/a49dcb4b-99a0-4939-9e0e-c844ce2e8d83-kube-api-access-72cpp\") pod \"payload-pre-processing-e2e-ait-474cab5e-79b66887b7-brn4j\" (UID: \"a49dcb4b-99a0-4939-9e0e-c844ce2e8d83\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-474cab5e-79b66887b7-brn4j" Nov 01 16:04:43.117039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:43.105973 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a49dcb4b-99a0-4939-9e0e-c844ce2e8d83-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-474cab5e-79b66887b7-brn4j\" (UID: \"a49dcb4b-99a0-4939-9e0e-c844ce2e8d83\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-474cab5e-79b66887b7-brn4j" Nov 01 16:04:43.204114 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:43.204076 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-474cab5e-79b66887b7-brn4j"] Nov 01 16:04:43.278352 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:43.278311 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a49dcb4b-99a0-4939-9e0e-c844ce2e8d83-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-474cab5e-79b66887b7-brn4j\" (UID: \"a49dcb4b-99a0-4939-9e0e-c844ce2e8d83\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-474cab5e-79b66887b7-brn4j" Nov 01 16:04:43.278625 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:43.278603 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72cpp\" (UniqueName: \"kubernetes.io/projected/a49dcb4b-99a0-4939-9e0e-c844ce2e8d83-kube-api-access-72cpp\") pod \"payload-pre-processing-e2e-ait-474cab5e-79b66887b7-brn4j\" (UID: \"a49dcb4b-99a0-4939-9e0e-c844ce2e8d83\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-474cab5e-79b66887b7-brn4j" Nov 01 16:04:43.294621 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:43.294500 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a49dcb4b-99a0-4939-9e0e-c844ce2e8d83-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-474cab5e-79b66887b7-brn4j\" (UID: \"a49dcb4b-99a0-4939-9e0e-c844ce2e8d83\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-474cab5e-79b66887b7-brn4j" Nov 01 16:04:43.306098 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:43.306060 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-474cab5e-85488d9c5c-btq6b"] Nov 01 16:04:43.578193 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:43.578064 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72cpp\" (UniqueName: \"kubernetes.io/projected/a49dcb4b-99a0-4939-9e0e-c844ce2e8d83-kube-api-access-72cpp\") pod \"payload-pre-processing-e2e-ait-474cab5e-79b66887b7-brn4j\" (UID: \"a49dcb4b-99a0-4939-9e0e-c844ce2e8d83\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-474cab5e-79b66887b7-brn4j" Nov 01 16:04:43.589214 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:43.581117 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-474cab5e-85488d9c5c-btq6b"] Nov 01 16:04:43.589214 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:43.581227 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-474cab5e-85488d9c5c-btq6b" Nov 01 16:04:43.600041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:43.596300 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-474cab5e-dockercfg-vns96\"" Nov 01 16:04:43.696185 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:43.674897 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/519e328e-0e0a-4fc3-b608-b3ff05e24c88-plugins-config-volume\") pod \"payload-processing-e2e-ait-474cab5e-85488d9c5c-btq6b\" (UID: \"519e328e-0e0a-4fc3-b608-b3ff05e24c88\") " pod="openshift-ingress/payload-processing-e2e-ait-474cab5e-85488d9c5c-btq6b" Nov 01 16:04:43.696185 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:43.675039 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rhpp\" (UniqueName: \"kubernetes.io/projected/519e328e-0e0a-4fc3-b608-b3ff05e24c88-kube-api-access-4rhpp\") pod \"payload-processing-e2e-ait-474cab5e-85488d9c5c-btq6b\" (UID: \"519e328e-0e0a-4fc3-b608-b3ff05e24c88\") " pod="openshift-ingress/payload-processing-e2e-ait-474cab5e-85488d9c5c-btq6b" Nov 01 16:04:43.777117 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:43.766244 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-474cab5e-79b66887b7-brn4j" Nov 01 16:04:43.777282 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:43.777242 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/519e328e-0e0a-4fc3-b608-b3ff05e24c88-plugins-config-volume\") pod \"payload-processing-e2e-ait-474cab5e-85488d9c5c-btq6b\" (UID: \"519e328e-0e0a-4fc3-b608-b3ff05e24c88\") " pod="openshift-ingress/payload-processing-e2e-ait-474cab5e-85488d9c5c-btq6b" Nov 01 16:04:43.778361 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:43.777390 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rhpp\" (UniqueName: \"kubernetes.io/projected/519e328e-0e0a-4fc3-b608-b3ff05e24c88-kube-api-access-4rhpp\") pod \"payload-processing-e2e-ait-474cab5e-85488d9c5c-btq6b\" (UID: \"519e328e-0e0a-4fc3-b608-b3ff05e24c88\") " pod="openshift-ingress/payload-processing-e2e-ait-474cab5e-85488d9c5c-btq6b" Nov 01 16:04:43.792593 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:43.792560 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/519e328e-0e0a-4fc3-b608-b3ff05e24c88-plugins-config-volume\") pod \"payload-processing-e2e-ait-474cab5e-85488d9c5c-btq6b\" (UID: \"519e328e-0e0a-4fc3-b608-b3ff05e24c88\") " pod="openshift-ingress/payload-processing-e2e-ait-474cab5e-85488d9c5c-btq6b" Nov 01 16:04:43.867785 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:43.867715 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rhpp\" (UniqueName: \"kubernetes.io/projected/519e328e-0e0a-4fc3-b608-b3ff05e24c88-kube-api-access-4rhpp\") pod \"payload-processing-e2e-ait-474cab5e-85488d9c5c-btq6b\" (UID: \"519e328e-0e0a-4fc3-b608-b3ff05e24c88\") " pod="openshift-ingress/payload-processing-e2e-ait-474cab5e-85488d9c5c-btq6b" Nov 01 16:04:43.987536 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:43.983808 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-474cab5e-85488d9c5c-btq6b" Nov 01 16:04:44.199354 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:44.199316 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9"] Nov 01 16:04:44.219462 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:44.219421 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:04:44.219683 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:44.219662 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:04:44.219822 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:44.219806 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:04:44.416678 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:04:44.416642 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda49dcb4b_99a0_4939_9e0e_c844ce2e8d83.slice/crio-11dcf92525217c75424df0fde26c75b8908cfe265dc5cb6d0ed03bf84dce596d WatchSource:0}: Error finding container 11dcf92525217c75424df0fde26c75b8908cfe265dc5cb6d0ed03bf84dce596d: Status 404 returned error can't find the container with id 11dcf92525217c75424df0fde26c75b8908cfe265dc5cb6d0ed03bf84dce596d Nov 01 16:04:44.417432 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:44.417396 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-474cab5e-79b66887b7-brn4j"] Nov 01 16:04:44.461810 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:44.461778 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-474cab5e-85488d9c5c-btq6b"] Nov 01 16:04:44.657270 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:44.657229 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ec6894c9-77f7f5dc8f-hstfs" podUID="f2a05cd5-e01f-43a1-a318-f3dd5285a970" containerName="payload-processing" containerID="cri-o://6874e0bcec4ef004520933f45bc0e3356d5118c9a0175b821d0a7a79140fe7cb" gracePeriod=25 Nov 01 16:04:44.668886 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:44.668823 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec6894c9-57bc8d4f9-2dkbw" podUID="6bd0daf3-1545-45b7-a481-d9439a83939c" containerName="payload-pre-processing" containerID="cri-o://7503c03cd6c44ddbdeb436463ff12f34c1f6d76bc1ef8478d541ebcc7d8b2086" gracePeriod=25 Nov 01 16:04:45.043809 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.043110 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ec6894c9-77f7f5dc8f-hstfs" Nov 01 16:04:45.053007 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.052981 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2a05cd5-e01f-43a1-a318-f3dd5285a970-kube-api-access-8rm7x\" (UniqueName: \"kubernetes.io/projected/f2a05cd5-e01f-43a1-a318-f3dd5285a970-kube-api-access-8rm7x\") pod \"f2a05cd5-e01f-43a1-a318-f3dd5285a970\" (UID: \"f2a05cd5-e01f-43a1-a318-f3dd5285a970\") " Nov 01 16:04:45.053220 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.053095 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2a05cd5-e01f-43a1-a318-f3dd5285a970-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2a05cd5-e01f-43a1-a318-f3dd5285a970-plugins-config-volume\") pod \"f2a05cd5-e01f-43a1-a318-f3dd5285a970\" (UID: \"f2a05cd5-e01f-43a1-a318-f3dd5285a970\") " Nov 01 16:04:45.053803 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.053763 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2a05cd5-e01f-43a1-a318-f3dd5285a970-plugins-config-volume" pod "f2a05cd5-e01f-43a1-a318-f3dd5285a970" (UID: "f2a05cd5-e01f-43a1-a318-f3dd5285a970"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:04:45.055331 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.055306 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2a05cd5-e01f-43a1-a318-f3dd5285a970-kube-api-access-8rm7x" pod "f2a05cd5-e01f-43a1-a318-f3dd5285a970" (UID: "f2a05cd5-e01f-43a1-a318-f3dd5285a970"). InnerVolumeSpecName "kube-api-access-8rm7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:04:45.084544 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.084509 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec6894c9-57bc8d4f9-2dkbw" Nov 01 16:04:45.154240 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.154213 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bd0daf3-1545-45b7-a481-d9439a83939c-kube-api-access-jmm4v\" (UniqueName: \"kubernetes.io/projected/6bd0daf3-1545-45b7-a481-d9439a83939c-kube-api-access-jmm4v\") pod \"6bd0daf3-1545-45b7-a481-d9439a83939c\" (UID: \"6bd0daf3-1545-45b7-a481-d9439a83939c\") " Nov 01 16:04:45.154433 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.154343 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bd0daf3-1545-45b7-a481-d9439a83939c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bd0daf3-1545-45b7-a481-d9439a83939c-plugins-config-volume\") pod \"6bd0daf3-1545-45b7-a481-d9439a83939c\" (UID: \"6bd0daf3-1545-45b7-a481-d9439a83939c\") " Nov 01 16:04:45.154811 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.154779 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bd0daf3-1545-45b7-a481-d9439a83939c-plugins-config-volume" pod "6bd0daf3-1545-45b7-a481-d9439a83939c" (UID: "6bd0daf3-1545-45b7-a481-d9439a83939c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:04:45.155410 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.155387 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rm7x\" (UniqueName: \"kubernetes.io/projected/f2a05cd5-e01f-43a1-a318-f3dd5285a970-kube-api-access-8rm7x\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:45.155473 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.155414 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2a05cd5-e01f-43a1-a318-f3dd5285a970-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:45.155473 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.155424 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bd0daf3-1545-45b7-a481-d9439a83939c-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:45.156382 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.156365 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bd0daf3-1545-45b7-a481-d9439a83939c-kube-api-access-jmm4v" pod "6bd0daf3-1545-45b7-a481-d9439a83939c" (UID: "6bd0daf3-1545-45b7-a481-d9439a83939c"). InnerVolumeSpecName "kube-api-access-jmm4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:04:45.190693 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.190605 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-474cab5e-79b66887b7-brn4j" event={"ID":"a49dcb4b-99a0-4939-9e0e-c844ce2e8d83","Type":"ContainerStarted","Data":"71d8eeaa23a90d433a76703e16ba90954a6fe8159ed71c8cb35299edd6b17cf1"} Nov 01 16:04:45.190693 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.190669 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-474cab5e-79b66887b7-brn4j" event={"ID":"a49dcb4b-99a0-4939-9e0e-c844ce2e8d83","Type":"ContainerStarted","Data":"11dcf92525217c75424df0fde26c75b8908cfe265dc5cb6d0ed03bf84dce596d"} Nov 01 16:04:45.190883 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.190823 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-474cab5e-79b66887b7-brn4j" Nov 01 16:04:45.192668 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.192637 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9" event={"ID":"86b37143-20ce-45d3-90d7-45b451012c14","Type":"ContainerStarted","Data":"c34de563140d0a4a76025d99491d588ee2ae1f61c1a504e992e9a67b10e71baf"} Nov 01 16:04:45.192805 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.192689 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9" event={"ID":"86b37143-20ce-45d3-90d7-45b451012c14","Type":"ContainerStarted","Data":"a911f0e4ff312b96c99ad3246f33d8082a5738e22eec9068aeb36282a835bc5b"} Nov 01 16:04:45.194578 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.194552 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-474cab5e-85488d9c5c-btq6b" event={"ID":"519e328e-0e0a-4fc3-b608-b3ff05e24c88","Type":"ContainerStarted","Data":"8f4b2e60d6bd6963643aa6f757d6a338f275c163b358a254dc70db22d5e6f410"} Nov 01 16:04:45.194722 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.194596 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-474cab5e-85488d9c5c-btq6b" event={"ID":"519e328e-0e0a-4fc3-b608-b3ff05e24c88","Type":"ContainerStarted","Data":"b01c24439ec68b708b48afaf0cd80210a8fcccebf125914015daf86b0a8f7d7b"} Nov 01 16:04:45.194722 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.194639 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-474cab5e-85488d9c5c-btq6b" Nov 01 16:04:45.195967 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.195942 4363 generic.go:356] "Generic (PLEG): container finished" podID="f2a05cd5-e01f-43a1-a318-f3dd5285a970" containerID="6874e0bcec4ef004520933f45bc0e3356d5118c9a0175b821d0a7a79140fe7cb" exitCode=0 Nov 01 16:04:45.196150 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.195983 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ec6894c9-77f7f5dc8f-hstfs" Nov 01 16:04:45.196150 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.196032 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ec6894c9-77f7f5dc8f-hstfs" event={"ID":"f2a05cd5-e01f-43a1-a318-f3dd5285a970","Type":"ContainerDied","Data":"6874e0bcec4ef004520933f45bc0e3356d5118c9a0175b821d0a7a79140fe7cb"} Nov 01 16:04:45.196150 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.196080 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ec6894c9-77f7f5dc8f-hstfs" event={"ID":"f2a05cd5-e01f-43a1-a318-f3dd5285a970","Type":"ContainerDied","Data":"7b2927ce2bae03d6933e9cbda927fd7f4c7f06c6e53b2a5ab14bd8f34d02d5e1"} Nov 01 16:04:45.196150 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.196108 4363 scope.go:122] "RemoveContainer" containerID="6874e0bcec4ef004520933f45bc0e3356d5118c9a0175b821d0a7a79140fe7cb" Nov 01 16:04:45.197590 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.197567 4363 generic.go:356] "Generic (PLEG): container finished" podID="6bd0daf3-1545-45b7-a481-d9439a83939c" containerID="7503c03cd6c44ddbdeb436463ff12f34c1f6d76bc1ef8478d541ebcc7d8b2086" exitCode=0 Nov 01 16:04:45.197761 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.197745 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec6894c9-57bc8d4f9-2dkbw" event={"ID":"6bd0daf3-1545-45b7-a481-d9439a83939c","Type":"ContainerDied","Data":"7503c03cd6c44ddbdeb436463ff12f34c1f6d76bc1ef8478d541ebcc7d8b2086"} Nov 01 16:04:45.197830 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.197787 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec6894c9-57bc8d4f9-2dkbw" event={"ID":"6bd0daf3-1545-45b7-a481-d9439a83939c","Type":"ContainerDied","Data":"c04dc07da02c856ee2845d2bb2409eb04669a3ba80608348f4999244440290f5"} Nov 01 16:04:45.197890 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.197850 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec6894c9-57bc8d4f9-2dkbw" Nov 01 16:04:45.209509 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.208800 4363 scope.go:122] "RemoveContainer" containerID="6874e0bcec4ef004520933f45bc0e3356d5118c9a0175b821d0a7a79140fe7cb" Nov 01 16:04:45.210894 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.210857 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-474cab5e-79b66887b7-brn4j" podStartSLOduration=3.210846011 podStartE2EDuration="3.210846011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:04:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:04:45.205430677 +0000 UTC m=+1018.537168328" watchObservedRunningTime="2026-11-01 16:04:45.210846011 +0000 UTC m=+1018.542583660" Nov 01 16:04:45.211830 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:04:45.211419 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6874e0bcec4ef004520933f45bc0e3356d5118c9a0175b821d0a7a79140fe7cb\": container with ID starting with 6874e0bcec4ef004520933f45bc0e3356d5118c9a0175b821d0a7a79140fe7cb not found: ID does not exist" containerID="6874e0bcec4ef004520933f45bc0e3356d5118c9a0175b821d0a7a79140fe7cb" Nov 01 16:04:45.211830 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.211455 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6874e0bcec4ef004520933f45bc0e3356d5118c9a0175b821d0a7a79140fe7cb"} err="failed to get container status \"6874e0bcec4ef004520933f45bc0e3356d5118c9a0175b821d0a7a79140fe7cb\": rpc error: code = NotFound desc = could not find container \"6874e0bcec4ef004520933f45bc0e3356d5118c9a0175b821d0a7a79140fe7cb\": container with ID starting with 6874e0bcec4ef004520933f45bc0e3356d5118c9a0175b821d0a7a79140fe7cb not found: ID does not exist" Nov 01 16:04:45.211830 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.211476 4363 scope.go:122] "RemoveContainer" containerID="7503c03cd6c44ddbdeb436463ff12f34c1f6d76bc1ef8478d541ebcc7d8b2086" Nov 01 16:04:45.227096 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.227074 4363 scope.go:122] "RemoveContainer" containerID="7503c03cd6c44ddbdeb436463ff12f34c1f6d76bc1ef8478d541ebcc7d8b2086" Nov 01 16:04:45.227451 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:04:45.227432 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7503c03cd6c44ddbdeb436463ff12f34c1f6d76bc1ef8478d541ebcc7d8b2086\": container with ID starting with 7503c03cd6c44ddbdeb436463ff12f34c1f6d76bc1ef8478d541ebcc7d8b2086 not found: ID does not exist" containerID="7503c03cd6c44ddbdeb436463ff12f34c1f6d76bc1ef8478d541ebcc7d8b2086" Nov 01 16:04:45.227533 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.227457 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7503c03cd6c44ddbdeb436463ff12f34c1f6d76bc1ef8478d541ebcc7d8b2086"} err="failed to get container status \"7503c03cd6c44ddbdeb436463ff12f34c1f6d76bc1ef8478d541ebcc7d8b2086\": rpc error: code = NotFound desc = could not find container \"7503c03cd6c44ddbdeb436463ff12f34c1f6d76bc1ef8478d541ebcc7d8b2086\": container with ID starting with 7503c03cd6c44ddbdeb436463ff12f34c1f6d76bc1ef8478d541ebcc7d8b2086 not found: ID does not exist" Nov 01 16:04:45.227679 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.227644 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-474cab5e-85488d9c5c-btq6b" podStartSLOduration=3.227632993 podStartE2EDuration="3.227632993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:04:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:04:45.226818073 +0000 UTC m=+1018.558555726" watchObservedRunningTime="2026-11-01 16:04:45.227632993 +0000 UTC m=+1018.559370642" Nov 01 16:04:45.248885 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.248839 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9" podStartSLOduration=4.248824045 podStartE2EDuration="4.248824045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:04:45.246867529 +0000 UTC m=+1018.578605192" watchObservedRunningTime="2026-11-01 16:04:45.248824045 +0000 UTC m=+1018.580561700" Nov 01 16:04:45.257436 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.257403 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmm4v\" (UniqueName: \"kubernetes.io/projected/6bd0daf3-1545-45b7-a481-d9439a83939c-kube-api-access-jmm4v\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:45.263449 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.263430 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ec6894c9-57bc8d4f9-2dkbw"] Nov 01 16:04:45.267442 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.267422 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ec6894c9-57bc8d4f9-2dkbw"] Nov 01 16:04:45.276993 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.276972 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ec6894c9-77f7f5dc8f-hstfs"] Nov 01 16:04:45.282484 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.282461 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ec6894c9-77f7f5dc8f-hstfs"] Nov 01 16:04:45.782439 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.782404 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9" Nov 01 16:04:45.787656 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:45.787633 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9" Nov 01 16:04:46.099675 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:46.099640 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ec6894c9-555b99b646-qgw8c"] Nov 01 16:04:46.100147 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:46.100102 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec6894c9-555b99b646-qgw8c" podUID="1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba" containerName="maas-api" containerID="cri-o://6762f9b5bc68af93027a81b9a07540275d3e4060a052c26f564542e081daf39a" gracePeriod=30 Nov 01 16:04:46.211100 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:46.211070 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9" Nov 01 16:04:46.211790 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:46.211768 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9" Nov 01 16:04:46.516041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:46.513157 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec6894c9-555b99b646-qgw8c" Nov 01 16:04:46.632920 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:46.632883 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba-metrics-certs\") pod \"1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba\" (UID: \"1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba\") " Nov 01 16:04:46.634558 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:46.634532 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba-kube-api-access-n9mql\" (UniqueName: \"kubernetes.io/projected/1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba-kube-api-access-n9mql\") pod \"1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba\" (UID: \"1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba\") " Nov 01 16:04:46.635225 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:46.635203 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba-maas-api-tls\") pod \"1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba\" (UID: \"1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba\") " Nov 01 16:04:46.635890 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:46.635820 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba-openshift-service-ca\") pod \"1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba\" (UID: \"1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba\") " Nov 01 16:04:46.647080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:46.646889 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba-openshift-service-ca" pod "1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba" (UID: "1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:04:46.648459 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:46.648361 4363 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba-openshift-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:46.676960 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:46.676926 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba-metrics-certs" pod "1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba" (UID: "1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:04:46.698505 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:46.698447 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba-maas-api-tls" pod "1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba" (UID: "1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:04:46.712757 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:46.712719 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba-kube-api-access-n9mql" pod "1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba" (UID: "1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba"). InnerVolumeSpecName "kube-api-access-n9mql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:04:46.768802 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:46.768768 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba-maas-api-tls\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:46.910327 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:46.910232 4363 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba-metrics-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:46.913773 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:46.913731 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9mql\" (UniqueName: \"kubernetes.io/projected/1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba-kube-api-access-n9mql\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:46.990243 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:46.990210 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bd0daf3-1545-45b7-a481-d9439a83939c" path="/var/lib/kubelet/pods/6bd0daf3-1545-45b7-a481-d9439a83939c/volumes" Nov 01 16:04:46.990833 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:46.990809 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2a05cd5-e01f-43a1-a318-f3dd5285a970" path="/var/lib/kubelet/pods/f2a05cd5-e01f-43a1-a318-f3dd5285a970/volumes" Nov 01 16:04:47.262570 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:47.262298 4363 generic.go:356] "Generic (PLEG): container finished" podID="1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba" containerID="6762f9b5bc68af93027a81b9a07540275d3e4060a052c26f564542e081daf39a" exitCode=0 Nov 01 16:04:47.264396 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:47.264344 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec6894c9-555b99b646-qgw8c" Nov 01 16:04:47.265091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:47.264566 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec6894c9-555b99b646-qgw8c" event={"ID":"1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba","Type":"ContainerDied","Data":"6762f9b5bc68af93027a81b9a07540275d3e4060a052c26f564542e081daf39a"} Nov 01 16:04:47.265304 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:47.265274 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec6894c9-555b99b646-qgw8c" event={"ID":"1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba","Type":"ContainerDied","Data":"e400ce344447f1f48b090e32e37e78f4642f027c3160dd6b2c9094ac8d603b5b"} Nov 01 16:04:47.265696 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:47.265631 4363 scope.go:122] "RemoveContainer" containerID="6762f9b5bc68af93027a81b9a07540275d3e4060a052c26f564542e081daf39a" Nov 01 16:04:47.305436 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:47.305077 4363 scope.go:122] "RemoveContainer" containerID="6762f9b5bc68af93027a81b9a07540275d3e4060a052c26f564542e081daf39a" Nov 01 16:04:47.305823 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:04:47.305682 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6762f9b5bc68af93027a81b9a07540275d3e4060a052c26f564542e081daf39a\": container with ID starting with 6762f9b5bc68af93027a81b9a07540275d3e4060a052c26f564542e081daf39a not found: ID does not exist" containerID="6762f9b5bc68af93027a81b9a07540275d3e4060a052c26f564542e081daf39a" Nov 01 16:04:47.305823 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:47.305719 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6762f9b5bc68af93027a81b9a07540275d3e4060a052c26f564542e081daf39a"} err="failed to get container status \"6762f9b5bc68af93027a81b9a07540275d3e4060a052c26f564542e081daf39a\": rpc error: code = NotFound desc = could not find container \"6762f9b5bc68af93027a81b9a07540275d3e4060a052c26f564542e081daf39a\": container with ID starting with 6762f9b5bc68af93027a81b9a07540275d3e4060a052c26f564542e081daf39a not found: ID does not exist" Nov 01 16:04:47.305823 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:47.305760 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ec6894c9-555b99b646-qgw8c"] Nov 01 16:04:47.311887 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:47.311858 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ec6894c9-555b99b646-qgw8c"] Nov 01 16:04:48.167361 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:48.167325 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9fd01d20-755c777847-lqtlx"] Nov 01 16:04:48.168057 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:48.168030 4363 cpu_manager.go:414] "containerMap: removing container" podUID="6bd0daf3-1545-45b7-a481-d9439a83939c" containerName="payload-pre-processing" Nov 01 16:04:48.168057 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:48.168058 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bd0daf3-1545-45b7-a481-d9439a83939c" containerName="payload-pre-processing" Nov 01 16:04:48.168166 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:48.168090 4363 cpu_manager.go:414] "containerMap: removing container" podUID="f2a05cd5-e01f-43a1-a318-f3dd5285a970" containerName="payload-processing" Nov 01 16:04:48.168166 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:48.168095 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2a05cd5-e01f-43a1-a318-f3dd5285a970" containerName="payload-processing" Nov 01 16:04:48.168166 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:48.168104 4363 cpu_manager.go:414] "containerMap: removing container" podUID="1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba" containerName="maas-api" Nov 01 16:04:48.168166 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:48.168109 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba" containerName="maas-api" Nov 01 16:04:48.168283 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:48.168229 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2a05cd5-e01f-43a1-a318-f3dd5285a970" containerName="payload-processing" Nov 01 16:04:48.168283 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:48.168237 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba" containerName="maas-api" Nov 01 16:04:48.168283 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:48.168251 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bd0daf3-1545-45b7-a481-d9439a83939c" containerName="payload-pre-processing" Nov 01 16:04:48.173642 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:48.173620 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9fd01d20-755c777847-lqtlx" Nov 01 16:04:48.180609 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:48.180582 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9fd01d20-755c777847-lqtlx"] Nov 01 16:04:48.181769 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:48.181744 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-9fd01d20-dockercfg-rs5cf\"" Nov 01 16:04:48.182697 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:48.182399 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-9fd01d20\"" Nov 01 16:04:48.232196 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:48.232175 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9fd01d20-7b479cbc8c-mt2d2"] Nov 01 16:04:48.238566 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:48.238546 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9fd01d20-7b479cbc8c-mt2d2" Nov 01 16:04:48.243327 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:48.243307 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9fd01d20-7b479cbc8c-mt2d2"] Nov 01 16:04:48.323349 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:48.323309 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z5hj\" (UniqueName: \"kubernetes.io/projected/b8f0e6f8-af57-4ae0-9f2a-f22b87e77028-kube-api-access-7z5hj\") pod \"payload-pre-processing-e2e-derive-9fd01d20-755c777847-lqtlx\" (UID: \"b8f0e6f8-af57-4ae0-9f2a-f22b87e77028\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9fd01d20-755c777847-lqtlx" Nov 01 16:04:48.323997 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:48.323975 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8f0e6f8-af57-4ae0-9f2a-f22b87e77028-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-9fd01d20-755c777847-lqtlx\" (UID: \"b8f0e6f8-af57-4ae0-9f2a-f22b87e77028\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9fd01d20-755c777847-lqtlx" Nov 01 16:04:48.328712 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:04:48.328682 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:04:48.425498 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:48.425409 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbb9q\" (UniqueName: \"kubernetes.io/projected/e1fff68b-27c4-40e5-a6a0-61e1b8e966b6-kube-api-access-fbb9q\") pod \"payload-processing-e2e-derive-9fd01d20-7b479cbc8c-mt2d2\" (UID: \"e1fff68b-27c4-40e5-a6a0-61e1b8e966b6\") " pod="openshift-ingress/payload-processing-e2e-derive-9fd01d20-7b479cbc8c-mt2d2" Nov 01 16:04:48.425647 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:48.425553 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z5hj\" (UniqueName: \"kubernetes.io/projected/b8f0e6f8-af57-4ae0-9f2a-f22b87e77028-kube-api-access-7z5hj\") pod \"payload-pre-processing-e2e-derive-9fd01d20-755c777847-lqtlx\" (UID: \"b8f0e6f8-af57-4ae0-9f2a-f22b87e77028\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9fd01d20-755c777847-lqtlx" Nov 01 16:04:48.425809 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:48.425785 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8f0e6f8-af57-4ae0-9f2a-f22b87e77028-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-9fd01d20-755c777847-lqtlx\" (UID: \"b8f0e6f8-af57-4ae0-9f2a-f22b87e77028\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9fd01d20-755c777847-lqtlx" Nov 01 16:04:48.425894 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:48.425840 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1fff68b-27c4-40e5-a6a0-61e1b8e966b6-plugins-config-volume\") pod \"payload-processing-e2e-derive-9fd01d20-7b479cbc8c-mt2d2\" (UID: \"e1fff68b-27c4-40e5-a6a0-61e1b8e966b6\") " pod="openshift-ingress/payload-processing-e2e-derive-9fd01d20-7b479cbc8c-mt2d2" Nov 01 16:04:48.426541 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:48.426517 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8f0e6f8-af57-4ae0-9f2a-f22b87e77028-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-9fd01d20-755c777847-lqtlx\" (UID: \"b8f0e6f8-af57-4ae0-9f2a-f22b87e77028\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9fd01d20-755c777847-lqtlx" Nov 01 16:04:48.454859 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:48.454777 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z5hj\" (UniqueName: \"kubernetes.io/projected/b8f0e6f8-af57-4ae0-9f2a-f22b87e77028-kube-api-access-7z5hj\") pod \"payload-pre-processing-e2e-derive-9fd01d20-755c777847-lqtlx\" (UID: \"b8f0e6f8-af57-4ae0-9f2a-f22b87e77028\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9fd01d20-755c777847-lqtlx" Nov 01 16:04:48.522229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:48.522165 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9fd01d20-755c777847-lqtlx" Nov 01 16:04:48.542906 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:48.530338 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1fff68b-27c4-40e5-a6a0-61e1b8e966b6-plugins-config-volume\") pod \"payload-processing-e2e-derive-9fd01d20-7b479cbc8c-mt2d2\" (UID: \"e1fff68b-27c4-40e5-a6a0-61e1b8e966b6\") " pod="openshift-ingress/payload-processing-e2e-derive-9fd01d20-7b479cbc8c-mt2d2" Nov 01 16:04:48.542906 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:48.530619 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbb9q\" (UniqueName: \"kubernetes.io/projected/e1fff68b-27c4-40e5-a6a0-61e1b8e966b6-kube-api-access-fbb9q\") pod \"payload-processing-e2e-derive-9fd01d20-7b479cbc8c-mt2d2\" (UID: \"e1fff68b-27c4-40e5-a6a0-61e1b8e966b6\") " pod="openshift-ingress/payload-processing-e2e-derive-9fd01d20-7b479cbc8c-mt2d2" Nov 01 16:04:48.542906 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:48.531849 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1fff68b-27c4-40e5-a6a0-61e1b8e966b6-plugins-config-volume\") pod \"payload-processing-e2e-derive-9fd01d20-7b479cbc8c-mt2d2\" (UID: \"e1fff68b-27c4-40e5-a6a0-61e1b8e966b6\") " pod="openshift-ingress/payload-processing-e2e-derive-9fd01d20-7b479cbc8c-mt2d2" Nov 01 16:04:48.619041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:48.599276 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbb9q\" (UniqueName: \"kubernetes.io/projected/e1fff68b-27c4-40e5-a6a0-61e1b8e966b6-kube-api-access-fbb9q\") pod \"payload-processing-e2e-derive-9fd01d20-7b479cbc8c-mt2d2\" (UID: \"e1fff68b-27c4-40e5-a6a0-61e1b8e966b6\") " pod="openshift-ingress/payload-processing-e2e-derive-9fd01d20-7b479cbc8c-mt2d2" Nov 01 16:04:48.857361 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:48.857322 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba" path="/var/lib/kubelet/pods/1ac29eb9-96ee-43e4-a00e-89b2ccd4c8ba/volumes" Nov 01 16:04:48.907270 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:48.907236 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9fd01d20-7b479cbc8c-mt2d2" Nov 01 16:04:49.214369 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:49.214334 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9fd01d20-755c777847-lqtlx"] Nov 01 16:04:49.214492 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:04:49.214366 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8f0e6f8_af57_4ae0_9f2a_f22b87e77028.slice/crio-6238c89ed797d89de59f412b6d23575da2f2fc5bf0b47080f4a590ccf827f26d WatchSource:0}: Error finding container 6238c89ed797d89de59f412b6d23575da2f2fc5bf0b47080f4a590ccf827f26d: Status 404 returned error can't find the container with id 6238c89ed797d89de59f412b6d23575da2f2fc5bf0b47080f4a590ccf827f26d Nov 01 16:04:49.264444 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:49.264420 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9fd01d20-7b479cbc8c-mt2d2"] Nov 01 16:04:49.268546 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:04:49.268512 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1fff68b_27c4_40e5_a6a0_61e1b8e966b6.slice/crio-ddd97b9667e0bd58f3a363f3ad2b34b47fbe897dc457a05521f55d3bf7bd5100 WatchSource:0}: Error finding container ddd97b9667e0bd58f3a363f3ad2b34b47fbe897dc457a05521f55d3bf7bd5100: Status 404 returned error can't find the container with id ddd97b9667e0bd58f3a363f3ad2b34b47fbe897dc457a05521f55d3bf7bd5100 Nov 01 16:04:49.459295 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:49.437469 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9fd01d20-755c777847-lqtlx" event={"ID":"b8f0e6f8-af57-4ae0-9f2a-f22b87e77028","Type":"ContainerStarted","Data":"6238c89ed797d89de59f412b6d23575da2f2fc5bf0b47080f4a590ccf827f26d"} Nov 01 16:04:49.459295 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:49.438755 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-9fd01d20-755c777847-lqtlx" Nov 01 16:04:49.459295 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:49.453328 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9fd01d20-7b479cbc8c-mt2d2" event={"ID":"e1fff68b-27c4-40e5-a6a0-61e1b8e966b6","Type":"ContainerStarted","Data":"ddd97b9667e0bd58f3a363f3ad2b34b47fbe897dc457a05521f55d3bf7bd5100"} Nov 01 16:04:49.471658 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:49.471613 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-9fd01d20-755c777847-lqtlx" podStartSLOduration=1.471598645 podStartE2EDuration="1.471598645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:04:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:04:49.469342922 +0000 UTC m=+1022.801080573" watchObservedRunningTime="2026-11-01 16:04:49.471598645 +0000 UTC m=+1022.803336300" Nov 01 16:04:50.461800 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:50.461762 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9fd01d20-755c777847-lqtlx" event={"ID":"b8f0e6f8-af57-4ae0-9f2a-f22b87e77028","Type":"ContainerStarted","Data":"cc48d4b914473c156ea6333a78d391d693cbefc38f97c1d52a6af47217b49e1c"} Nov 01 16:04:50.463542 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:50.463513 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9fd01d20-7b479cbc8c-mt2d2" event={"ID":"e1fff68b-27c4-40e5-a6a0-61e1b8e966b6","Type":"ContainerStarted","Data":"7e9c94fb78110d48c999ad05c8ab0bdc384ce442052d3f84afcb132aab32487e"} Nov 01 16:04:50.463677 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:50.463614 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-9fd01d20-7b479cbc8c-mt2d2" Nov 01 16:04:50.478541 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:50.478499 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-9fd01d20-7b479cbc8c-mt2d2" podStartSLOduration=2.478486011 podStartE2EDuration="2.478486011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:04:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:04:50.477006603 +0000 UTC m=+1023.808744249" watchObservedRunningTime="2026-11-01 16:04:50.478486011 +0000 UTC m=+1023.810223661" Nov 01 16:04:51.437743 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:51.437710 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn"] Nov 01 16:04:51.443580 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:51.443555 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn" Nov 01 16:04:51.452532 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:51.452492 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn"] Nov 01 16:04:51.453695 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:51.453672 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-474cab5e-serving-cert\"" Nov 01 16:04:51.568516 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:51.563383 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-296xn\" (UniqueName: \"kubernetes.io/projected/db263bc3-5e94-4bc7-83b9-0598f567eeb7-kube-api-access-296xn\") pod \"maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn\" (UID: \"db263bc3-5e94-4bc7-83b9-0598f567eeb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn" Nov 01 16:04:51.568516 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:51.563459 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db263bc3-5e94-4bc7-83b9-0598f567eeb7-metrics-certs\") pod \"maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn\" (UID: \"db263bc3-5e94-4bc7-83b9-0598f567eeb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn" Nov 01 16:04:51.568516 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:51.563572 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db263bc3-5e94-4bc7-83b9-0598f567eeb7-maas-api-tls\") pod \"maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn\" (UID: \"db263bc3-5e94-4bc7-83b9-0598f567eeb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn" Nov 01 16:04:51.568516 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:51.563783 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db263bc3-5e94-4bc7-83b9-0598f567eeb7-openshift-service-ca\") pod \"maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn\" (UID: \"db263bc3-5e94-4bc7-83b9-0598f567eeb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn" Nov 01 16:04:51.675041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:51.666083 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db263bc3-5e94-4bc7-83b9-0598f567eeb7-maas-api-tls\") pod \"maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn\" (UID: \"db263bc3-5e94-4bc7-83b9-0598f567eeb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn" Nov 01 16:04:51.675041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:51.666345 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db263bc3-5e94-4bc7-83b9-0598f567eeb7-openshift-service-ca\") pod \"maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn\" (UID: \"db263bc3-5e94-4bc7-83b9-0598f567eeb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn" Nov 01 16:04:51.675041 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:04:51.669177 4363 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-474cab5e-serving-cert: secret "maas-api-e2e-ait-474cab5e-serving-cert" not found Nov 01 16:04:51.675041 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:04:51.669263 4363 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db263bc3-5e94-4bc7-83b9-0598f567eeb7-maas-api-tls podName:db263bc3-5e94-4bc7-83b9-0598f567eeb7 nodeName:}" failed. No retries permitted until 2026-11-01 16:04:52.169240865 +0000 UTC m=+1025.500978529 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/db263bc3-5e94-4bc7-83b9-0598f567eeb7-maas-api-tls") pod "maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn" (UID: "db263bc3-5e94-4bc7-83b9-0598f567eeb7") : secret "maas-api-e2e-ait-474cab5e-serving-cert" not found Nov 01 16:04:51.675705 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:51.675668 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-296xn\" (UniqueName: \"kubernetes.io/projected/db263bc3-5e94-4bc7-83b9-0598f567eeb7-kube-api-access-296xn\") pod \"maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn\" (UID: \"db263bc3-5e94-4bc7-83b9-0598f567eeb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn" Nov 01 16:04:51.677119 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:51.677095 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db263bc3-5e94-4bc7-83b9-0598f567eeb7-metrics-certs\") pod \"maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn\" (UID: \"db263bc3-5e94-4bc7-83b9-0598f567eeb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn" Nov 01 16:04:51.678202 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:51.678176 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db263bc3-5e94-4bc7-83b9-0598f567eeb7-openshift-service-ca\") pod \"maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn\" (UID: \"db263bc3-5e94-4bc7-83b9-0598f567eeb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn" Nov 01 16:04:51.707918 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:51.707821 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db263bc3-5e94-4bc7-83b9-0598f567eeb7-metrics-certs\") pod \"maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn\" (UID: \"db263bc3-5e94-4bc7-83b9-0598f567eeb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn" Nov 01 16:04:51.708164 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:51.707901 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-296xn\" (UniqueName: \"kubernetes.io/projected/db263bc3-5e94-4bc7-83b9-0598f567eeb7-kube-api-access-296xn\") pod \"maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn\" (UID: \"db263bc3-5e94-4bc7-83b9-0598f567eeb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn" Nov 01 16:04:52.200155 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:52.197456 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db263bc3-5e94-4bc7-83b9-0598f567eeb7-maas-api-tls\") pod \"maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn\" (UID: \"db263bc3-5e94-4bc7-83b9-0598f567eeb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn" Nov 01 16:04:52.252957 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:52.252636 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db263bc3-5e94-4bc7-83b9-0598f567eeb7-maas-api-tls\") pod \"maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn\" (UID: \"db263bc3-5e94-4bc7-83b9-0598f567eeb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn" Nov 01 16:04:52.369991 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:52.369957 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn" Nov 01 16:04:52.680348 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:52.680219 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn"] Nov 01 16:04:52.691685 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:04:52.691655 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb263bc3_5e94_4bc7_83b9_0598f567eeb7.slice/crio-737677f91b6ef276ee91d12a3b0e6ef06559444fcf4585dd5b594023204491c5 WatchSource:0}: Error finding container 737677f91b6ef276ee91d12a3b0e6ef06559444fcf4585dd5b594023204491c5: Status 404 returned error can't find the container with id 737677f91b6ef276ee91d12a3b0e6ef06559444fcf4585dd5b594023204491c5 Nov 01 16:04:53.494827 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:53.494743 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn" event={"ID":"db263bc3-5e94-4bc7-83b9-0598f567eeb7","Type":"ContainerStarted","Data":"47fd377f6cfeaa554fadf101add76eaa6e76ee1b328a9404519dd8c5989b7921"} Nov 01 16:04:53.494827 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:53.494800 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn" event={"ID":"db263bc3-5e94-4bc7-83b9-0598f567eeb7","Type":"ContainerStarted","Data":"737677f91b6ef276ee91d12a3b0e6ef06559444fcf4585dd5b594023204491c5"} Nov 01 16:04:53.495094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:53.494959 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn" Nov 01 16:04:53.515222 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:53.515178 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn" podStartSLOduration=2.016420793 podStartE2EDuration="2.515164548s" totalImagesPullingTime="498.743755ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:04:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:04:53.510450186 +0000 UTC m=+1026.842187838" watchObservedRunningTime="2026-11-01 16:04:53.515164548 +0000 UTC m=+1026.846902202" Nov 01 16:04:55.508827 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:55.505130 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9fd01d20-6f66b5869b-t5dkr"] Nov 01 16:04:55.635363 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:55.635328 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fd01d20-6f66b5869b-t5dkr" Nov 01 16:04:55.692573 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:55.670975 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-9fd01d20-serving-cert\"" Nov 01 16:04:55.692573 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:55.684004 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9fd01d20-6f66b5869b-t5dkr"] Nov 01 16:04:55.791693 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:55.791443 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3367eda-e540-4fca-a638-9571e68141eb-openshift-service-ca\") pod \"maas-api-e2e-derive-9fd01d20-6f66b5869b-t5dkr\" (UID: \"a3367eda-e540-4fca-a638-9571e68141eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fd01d20-6f66b5869b-t5dkr" Nov 01 16:04:55.791693 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:55.791514 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3367eda-e540-4fca-a638-9571e68141eb-maas-api-tls\") pod \"maas-api-e2e-derive-9fd01d20-6f66b5869b-t5dkr\" (UID: \"a3367eda-e540-4fca-a638-9571e68141eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fd01d20-6f66b5869b-t5dkr" Nov 01 16:04:55.791912 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:55.791692 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3367eda-e540-4fca-a638-9571e68141eb-metrics-certs\") pod \"maas-api-e2e-derive-9fd01d20-6f66b5869b-t5dkr\" (UID: \"a3367eda-e540-4fca-a638-9571e68141eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fd01d20-6f66b5869b-t5dkr" Nov 01 16:04:55.792556 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:55.791989 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwfbx\" (UniqueName: \"kubernetes.io/projected/a3367eda-e540-4fca-a638-9571e68141eb-kube-api-access-fwfbx\") pod \"maas-api-e2e-derive-9fd01d20-6f66b5869b-t5dkr\" (UID: \"a3367eda-e540-4fca-a638-9571e68141eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fd01d20-6f66b5869b-t5dkr" Nov 01 16:04:55.905035 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:55.895448 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3367eda-e540-4fca-a638-9571e68141eb-metrics-certs\") pod \"maas-api-e2e-derive-9fd01d20-6f66b5869b-t5dkr\" (UID: \"a3367eda-e540-4fca-a638-9571e68141eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fd01d20-6f66b5869b-t5dkr" Nov 01 16:04:55.905035 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:55.895711 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwfbx\" (UniqueName: \"kubernetes.io/projected/a3367eda-e540-4fca-a638-9571e68141eb-kube-api-access-fwfbx\") pod \"maas-api-e2e-derive-9fd01d20-6f66b5869b-t5dkr\" (UID: \"a3367eda-e540-4fca-a638-9571e68141eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fd01d20-6f66b5869b-t5dkr" Nov 01 16:04:55.905035 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:55.895827 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3367eda-e540-4fca-a638-9571e68141eb-openshift-service-ca\") pod \"maas-api-e2e-derive-9fd01d20-6f66b5869b-t5dkr\" (UID: \"a3367eda-e540-4fca-a638-9571e68141eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fd01d20-6f66b5869b-t5dkr" Nov 01 16:04:55.905035 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:55.895870 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3367eda-e540-4fca-a638-9571e68141eb-maas-api-tls\") pod \"maas-api-e2e-derive-9fd01d20-6f66b5869b-t5dkr\" (UID: \"a3367eda-e540-4fca-a638-9571e68141eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fd01d20-6f66b5869b-t5dkr" Nov 01 16:04:55.935703 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:55.935669 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3367eda-e540-4fca-a638-9571e68141eb-openshift-service-ca\") pod \"maas-api-e2e-derive-9fd01d20-6f66b5869b-t5dkr\" (UID: \"a3367eda-e540-4fca-a638-9571e68141eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fd01d20-6f66b5869b-t5dkr" Nov 01 16:04:55.959147 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:55.948744 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3367eda-e540-4fca-a638-9571e68141eb-metrics-certs\") pod \"maas-api-e2e-derive-9fd01d20-6f66b5869b-t5dkr\" (UID: \"a3367eda-e540-4fca-a638-9571e68141eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fd01d20-6f66b5869b-t5dkr" Nov 01 16:04:55.959612 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:55.943855 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3367eda-e540-4fca-a638-9571e68141eb-maas-api-tls\") pod \"maas-api-e2e-derive-9fd01d20-6f66b5869b-t5dkr\" (UID: \"a3367eda-e540-4fca-a638-9571e68141eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fd01d20-6f66b5869b-t5dkr" Nov 01 16:04:56.004043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:55.991742 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwfbx\" (UniqueName: \"kubernetes.io/projected/a3367eda-e540-4fca-a638-9571e68141eb-kube-api-access-fwfbx\") pod \"maas-api-e2e-derive-9fd01d20-6f66b5869b-t5dkr\" (UID: \"a3367eda-e540-4fca-a638-9571e68141eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fd01d20-6f66b5869b-t5dkr" Nov 01 16:04:56.119226 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:56.109374 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fd01d20-6f66b5869b-t5dkr" Nov 01 16:04:56.266156 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:56.265508 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-474cab5e-79b66887b7-brn4j" Nov 01 16:04:56.302039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:56.290182 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-474cab5e-85488d9c5c-btq6b" Nov 01 16:04:56.918291 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:56.918270 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9fd01d20-6f66b5869b-t5dkr"] Nov 01 16:04:57.880669 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:57.880629 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fd01d20-6f66b5869b-t5dkr" event={"ID":"a3367eda-e540-4fca-a638-9571e68141eb","Type":"ContainerStarted","Data":"312e2a494c1a3171477bfe17560ebea8af1fefae98cc0475813af27d0e92b863"} Nov 01 16:04:57.880878 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:57.880692 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fd01d20-6f66b5869b-t5dkr" event={"ID":"a3367eda-e540-4fca-a638-9571e68141eb","Type":"ContainerStarted","Data":"57234f431e78b9522d8fb157c07aedd123ae97d8ba1e287bc7903e71965b19a7"} Nov 01 16:04:57.880878 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:57.880806 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fd01d20-6f66b5869b-t5dkr" Nov 01 16:04:57.904113 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:57.904060 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fd01d20-6f66b5869b-t5dkr" podStartSLOduration=2.418592331 podStartE2EDuration="2.904040853s" totalImagesPullingTime="485.448522ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:04:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:04:57.898560199 +0000 UTC m=+1031.230297849" watchObservedRunningTime="2026-11-01 16:04:57.904040853 +0000 UTC m=+1031.235778501" Nov 01 16:04:59.509612 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:04:59.509580 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn" Nov 01 16:05:01.504309 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:01.504221 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-9fd01d20-755c777847-lqtlx" Nov 01 16:05:01.506423 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:01.506397 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-9fd01d20-7b479cbc8c-mt2d2" Nov 01 16:05:03.961039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:03.953592 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fd01d20-6f66b5869b-t5dkr" Nov 01 16:05:12.022904 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:12.022868 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02433184-qzbf4"] Nov 01 16:05:12.026779 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:12.026751 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02433184-qzbf4"] Nov 01 16:05:12.819842 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:12.819793 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba7a1d02-36e4-4fc8-9cc6-75606b0427bc" path="/var/lib/kubelet/pods/ba7a1d02-36e4-4fc8-9cc6-75606b0427bc/volumes" Nov 01 16:05:16.861009 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:16.860978 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9fd01d20-7b479cbc8c-mt2d2"] Nov 01 16:05:16.874090 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:16.874063 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9fd01d20-755c777847-lqtlx"] Nov 01 16:05:18.132279 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:18.130149 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:05:18.197120 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:18.197083 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-474cab5e-85488d9c5c-btq6b"] Nov 01 16:05:18.369063 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:18.357966 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-474cab5e-79b66887b7-brn4j"] Nov 01 16:05:18.604275 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:18.604232 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:05:18.643109 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:18.643067 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:05:18.929094 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:05:18.908853 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:05:19.050220 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:19.050171 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:05:19.050818 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:19.050783 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:05:19.052642 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:19.052620 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn"] Nov 01 16:05:19.052982 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:19.052954 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn" podUID="db263bc3-5e94-4bc7-83b9-0598f567eeb7" containerName="maas-api" containerID="cri-o://47fd377f6cfeaa554fadf101add76eaa6e76ee1b328a9404519dd8c5989b7921" gracePeriod=30 Nov 01 16:05:19.174398 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:19.174353 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:05:19.219047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:19.208186 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 16:05:19.474003 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:19.473688 4363 generic.go:356] "Generic (PLEG): container finished" podID="db263bc3-5e94-4bc7-83b9-0598f567eeb7" containerID="47fd377f6cfeaa554fadf101add76eaa6e76ee1b328a9404519dd8c5989b7921" exitCode=0 Nov 01 16:05:19.474003 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:19.473747 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn" event={"ID":"db263bc3-5e94-4bc7-83b9-0598f567eeb7","Type":"ContainerDied","Data":"47fd377f6cfeaa554fadf101add76eaa6e76ee1b328a9404519dd8c5989b7921"} Nov 01 16:05:19.565620 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:19.562312 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn" podUID="db263bc3-5e94-4bc7-83b9-0598f567eeb7" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.95:8443/health\": dial tcp 10.134.0.95:8443: connect: connection refused" Nov 01 16:05:19.565620 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:19.565096 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 16:05:20.066608 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:20.066581 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn" Nov 01 16:05:20.140340 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:20.140300 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db263bc3-5e94-4bc7-83b9-0598f567eeb7-kube-api-access-296xn\" (UniqueName: \"kubernetes.io/projected/db263bc3-5e94-4bc7-83b9-0598f567eeb7-kube-api-access-296xn\") pod \"db263bc3-5e94-4bc7-83b9-0598f567eeb7\" (UID: \"db263bc3-5e94-4bc7-83b9-0598f567eeb7\") " Nov 01 16:05:20.141859 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:20.141829 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db263bc3-5e94-4bc7-83b9-0598f567eeb7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db263bc3-5e94-4bc7-83b9-0598f567eeb7-openshift-service-ca\") pod \"db263bc3-5e94-4bc7-83b9-0598f567eeb7\" (UID: \"db263bc3-5e94-4bc7-83b9-0598f567eeb7\") " Nov 01 16:05:20.142106 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:20.142085 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db263bc3-5e94-4bc7-83b9-0598f567eeb7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db263bc3-5e94-4bc7-83b9-0598f567eeb7-maas-api-tls\") pod \"db263bc3-5e94-4bc7-83b9-0598f567eeb7\" (UID: \"db263bc3-5e94-4bc7-83b9-0598f567eeb7\") " Nov 01 16:05:20.142270 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:20.142252 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db263bc3-5e94-4bc7-83b9-0598f567eeb7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/db263bc3-5e94-4bc7-83b9-0598f567eeb7-metrics-certs\") pod \"db263bc3-5e94-4bc7-83b9-0598f567eeb7\" (UID: \"db263bc3-5e94-4bc7-83b9-0598f567eeb7\") " Nov 01 16:05:20.155373 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:20.153493 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db263bc3-5e94-4bc7-83b9-0598f567eeb7-openshift-service-ca" pod "db263bc3-5e94-4bc7-83b9-0598f567eeb7" (UID: "db263bc3-5e94-4bc7-83b9-0598f567eeb7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:05:20.158533 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:20.158503 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db263bc3-5e94-4bc7-83b9-0598f567eeb7-metrics-certs" pod "db263bc3-5e94-4bc7-83b9-0598f567eeb7" (UID: "db263bc3-5e94-4bc7-83b9-0598f567eeb7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:05:20.160384 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:20.160324 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db263bc3-5e94-4bc7-83b9-0598f567eeb7-kube-api-access-296xn" pod "db263bc3-5e94-4bc7-83b9-0598f567eeb7" (UID: "db263bc3-5e94-4bc7-83b9-0598f567eeb7"). InnerVolumeSpecName "kube-api-access-296xn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:05:20.163203 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:20.163146 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db263bc3-5e94-4bc7-83b9-0598f567eeb7-maas-api-tls" pod "db263bc3-5e94-4bc7-83b9-0598f567eeb7" (UID: "db263bc3-5e94-4bc7-83b9-0598f567eeb7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:05:20.172655 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:20.172628 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54"] Nov 01 16:05:20.172903 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:20.172876 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54" podUID="a4504fd0-ac35-4f52-8403-beb7c42daa96" containerName="istio-proxy" containerID="cri-o://9a427625ec052056690996064900fc5e0eff4838367e779abb16c2fd23883b40" gracePeriod=30 Nov 01 16:05:20.395183 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:20.391465 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-296xn\" (UniqueName: \"kubernetes.io/projected/db263bc3-5e94-4bc7-83b9-0598f567eeb7-kube-api-access-296xn\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:20.395183 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:20.391516 4363 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db263bc3-5e94-4bc7-83b9-0598f567eeb7-openshift-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:20.395183 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:20.391535 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db263bc3-5e94-4bc7-83b9-0598f567eeb7-maas-api-tls\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:20.395183 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:20.391550 4363 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db263bc3-5e94-4bc7-83b9-0598f567eeb7-metrics-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:20.756056 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:20.738565 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn" event={"ID":"db263bc3-5e94-4bc7-83b9-0598f567eeb7","Type":"ContainerDied","Data":"737677f91b6ef276ee91d12a3b0e6ef06559444fcf4585dd5b594023204491c5"} Nov 01 16:05:20.756056 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:20.738835 4363 scope.go:122] "RemoveContainer" containerID="47fd377f6cfeaa554fadf101add76eaa6e76ee1b328a9404519dd8c5989b7921" Nov 01 16:05:20.756056 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:20.739278 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn" Nov 01 16:05:21.141040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:21.140076 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn"] Nov 01 16:05:21.143957 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:21.143921 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-474cab5e-7f565cb696-nxvpn"] Nov 01 16:05:21.864136 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:21.862092 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-9fd01d20-7b479cbc8c-mt2d2" podUID="e1fff68b-27c4-40e5-a6a0-61e1b8e966b6" containerName="payload-processing" containerID="cri-o://7e9c94fb78110d48c999ad05c8ab0bdc384ce442052d3f84afcb132aab32487e" gracePeriod=25 Nov 01 16:05:21.885049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:21.878905 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-9fd01d20-755c777847-lqtlx" podUID="b8f0e6f8-af57-4ae0-9f2a-f22b87e77028" containerName="payload-pre-processing" containerID="cri-o://cc48d4b914473c156ea6333a78d391d693cbefc38f97c1d52a6af47217b49e1c" gracePeriod=25 Nov 01 16:05:22.065081 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:22.063908 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9fd01d20-6f66b5869b-t5dkr"] Nov 01 16:05:22.138721 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:22.138268 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fd01d20-6f66b5869b-t5dkr" podUID="a3367eda-e540-4fca-a638-9571e68141eb" containerName="maas-api" containerID="cri-o://312e2a494c1a3171477bfe17560ebea8af1fefae98cc0475813af27d0e92b863" gracePeriod=30 Nov 01 16:05:22.573846 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:22.556251 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:05:22.778043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:22.764072 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9"] Nov 01 16:05:22.778629 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:22.778574 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9" podUID="86b37143-20ce-45d3-90d7-45b451012c14" containerName="istio-proxy" containerID="cri-o://c34de563140d0a4a76025d99491d588ee2ae1f61c1a504e992e9a67b10e71baf" gracePeriod=30 Nov 01 16:05:23.056238 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:23.052116 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:05:23.216094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:23.213106 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-474cab5e-85488d9c5c-btq6b" podUID="519e328e-0e0a-4fc3-b608-b3ff05e24c88" containerName="payload-processing" containerID="cri-o://8f4b2e60d6bd6963643aa6f757d6a338f275c163b358a254dc70db22d5e6f410" gracePeriod=25 Nov 01 16:05:23.341688 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:23.341654 4363 generic.go:356] "Generic (PLEG): container finished" podID="b8f0e6f8-af57-4ae0-9f2a-f22b87e77028" containerID="cc48d4b914473c156ea6333a78d391d693cbefc38f97c1d52a6af47217b49e1c" exitCode=0 Nov 01 16:05:23.342608 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:23.342581 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db263bc3-5e94-4bc7-83b9-0598f567eeb7" path="/var/lib/kubelet/pods/db263bc3-5e94-4bc7-83b9-0598f567eeb7/volumes" Nov 01 16:05:23.345141 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:23.343250 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9fd01d20-755c777847-lqtlx" event={"ID":"b8f0e6f8-af57-4ae0-9f2a-f22b87e77028","Type":"ContainerDied","Data":"cc48d4b914473c156ea6333a78d391d693cbefc38f97c1d52a6af47217b49e1c"} Nov 01 16:05:23.360254 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:23.360206 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-474cab5e-79b66887b7-brn4j" podUID="a49dcb4b-99a0-4939-9e0e-c844ce2e8d83" containerName="payload-pre-processing" containerID="cri-o://71d8eeaa23a90d433a76703e16ba90954a6fe8159ed71c8cb35299edd6b17cf1" gracePeriod=25 Nov 01 16:05:23.375616 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:23.375581 4363 generic.go:356] "Generic (PLEG): container finished" podID="e1fff68b-27c4-40e5-a6a0-61e1b8e966b6" containerID="7e9c94fb78110d48c999ad05c8ab0bdc384ce442052d3f84afcb132aab32487e" exitCode=0 Nov 01 16:05:23.375986 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:23.375959 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9fd01d20-7b479cbc8c-mt2d2" event={"ID":"e1fff68b-27c4-40e5-a6a0-61e1b8e966b6","Type":"ContainerDied","Data":"7e9c94fb78110d48c999ad05c8ab0bdc384ce442052d3f84afcb132aab32487e"} Nov 01 16:05:23.557170 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:23.557114 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:05:23.557723 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:23.557695 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:05:23.586310 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:23.578388 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:05:23.916824 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:23.916512 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fd01d20-6f66b5869b-t5dkr" podUID="a3367eda-e540-4fca-a638-9571e68141eb" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.96:8443/health\": dial tcp 10.134.0.96:8443: connect: connection refused" Nov 01 16:05:24.009140 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:23.981971 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:05:24.117356 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:24.046081 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz"] Nov 01 16:05:24.179109 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:24.119938 4363 cpu_manager.go:414] "containerMap: removing container" podUID="db263bc3-5e94-4bc7-83b9-0598f567eeb7" containerName="maas-api" Nov 01 16:05:24.179109 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:24.119970 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db263bc3-5e94-4bc7-83b9-0598f567eeb7" containerName="maas-api" Nov 01 16:05:24.179109 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:24.120230 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="db263bc3-5e94-4bc7-83b9-0598f567eeb7" containerName="maas-api" Nov 01 16:05:24.238043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:24.236430 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" Nov 01 16:05:24.265183 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:24.263630 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-5e2d03-openshift-default-dockercfg-7zm9b\"" Nov 01 16:05:24.406619 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:24.300875 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:05:24.406619 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:24.320149 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz"] Nov 01 16:05:24.406619 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:24.405415 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19456502-09eb-4dc7-8877-395f1b5e7218-istio-data\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz\" (UID: \"19456502-09eb-4dc7-8877-395f1b5e7218\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" Nov 01 16:05:24.406619 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:24.405589 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19456502-09eb-4dc7-8877-395f1b5e7218-istio-podinfo\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz\" (UID: \"19456502-09eb-4dc7-8877-395f1b5e7218\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" Nov 01 16:05:24.406619 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:24.405702 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19456502-09eb-4dc7-8877-395f1b5e7218-istio-envoy\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz\" (UID: \"19456502-09eb-4dc7-8877-395f1b5e7218\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" Nov 01 16:05:24.406619 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:24.405941 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19456502-09eb-4dc7-8877-395f1b5e7218-credential-socket\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz\" (UID: \"19456502-09eb-4dc7-8877-395f1b5e7218\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" Nov 01 16:05:24.406619 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:24.406079 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19456502-09eb-4dc7-8877-395f1b5e7218-istiod-ca-cert\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz\" (UID: \"19456502-09eb-4dc7-8877-395f1b5e7218\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" Nov 01 16:05:24.406619 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:24.406139 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19456502-09eb-4dc7-8877-395f1b5e7218-workload-certs\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz\" (UID: \"19456502-09eb-4dc7-8877-395f1b5e7218\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" Nov 01 16:05:24.406619 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:24.406294 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19456502-09eb-4dc7-8877-395f1b5e7218-istio-token\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz\" (UID: \"19456502-09eb-4dc7-8877-395f1b5e7218\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" Nov 01 16:05:24.406619 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:24.406376 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19456502-09eb-4dc7-8877-395f1b5e7218-workload-socket\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz\" (UID: \"19456502-09eb-4dc7-8877-395f1b5e7218\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" Nov 01 16:05:24.406619 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:24.406521 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zc89\" (UniqueName: \"kubernetes.io/projected/19456502-09eb-4dc7-8877-395f1b5e7218-kube-api-access-5zc89\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz\" (UID: \"19456502-09eb-4dc7-8877-395f1b5e7218\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" Nov 01 16:05:24.449110 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:24.442765 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5a65503-66jkl"] Nov 01 16:05:24.494481 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:24.488255 4363 generic.go:356] "Generic (PLEG): container finished" podID="a49dcb4b-99a0-4939-9e0e-c844ce2e8d83" containerID="71d8eeaa23a90d433a76703e16ba90954a6fe8159ed71c8cb35299edd6b17cf1" exitCode=0 Nov 01 16:05:24.494481 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:24.488534 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-474cab5e-79b66887b7-brn4j" event={"ID":"a49dcb4b-99a0-4939-9e0e-c844ce2e8d83","Type":"ContainerDied","Data":"71d8eeaa23a90d433a76703e16ba90954a6fe8159ed71c8cb35299edd6b17cf1"} Nov 01 16:05:24.500291 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:24.497329 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5a65503-66jkl"] Nov 01 16:05:24.610372 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:24.594339 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:05:24.620476 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:24.615781 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 16:05:24.620476 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:24.617270 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:05:24.743174 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:24.721351 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19456502-09eb-4dc7-8877-395f1b5e7218-credential-socket\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz\" (UID: \"19456502-09eb-4dc7-8877-395f1b5e7218\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" Nov 01 16:05:24.743174 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:24.721449 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19456502-09eb-4dc7-8877-395f1b5e7218-istiod-ca-cert\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz\" (UID: \"19456502-09eb-4dc7-8877-395f1b5e7218\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" Nov 01 16:05:24.743174 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:24.721494 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19456502-09eb-4dc7-8877-395f1b5e7218-workload-certs\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz\" (UID: \"19456502-09eb-4dc7-8877-395f1b5e7218\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" Nov 01 16:05:24.743174 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:24.721538 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19456502-09eb-4dc7-8877-395f1b5e7218-istio-token\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz\" (UID: \"19456502-09eb-4dc7-8877-395f1b5e7218\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" Nov 01 16:05:24.743174 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:24.721577 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19456502-09eb-4dc7-8877-395f1b5e7218-workload-socket\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz\" (UID: \"19456502-09eb-4dc7-8877-395f1b5e7218\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" Nov 01 16:05:24.743174 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:24.721658 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zc89\" (UniqueName: \"kubernetes.io/projected/19456502-09eb-4dc7-8877-395f1b5e7218-kube-api-access-5zc89\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz\" (UID: \"19456502-09eb-4dc7-8877-395f1b5e7218\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" Nov 01 16:05:24.743174 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:24.721764 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19456502-09eb-4dc7-8877-395f1b5e7218-istio-data\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz\" (UID: \"19456502-09eb-4dc7-8877-395f1b5e7218\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" Nov 01 16:05:24.743174 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:24.721844 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19456502-09eb-4dc7-8877-395f1b5e7218-istio-podinfo\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz\" (UID: \"19456502-09eb-4dc7-8877-395f1b5e7218\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" Nov 01 16:05:24.743174 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:24.721901 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19456502-09eb-4dc7-8877-395f1b5e7218-istio-envoy\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz\" (UID: \"19456502-09eb-4dc7-8877-395f1b5e7218\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" Nov 01 16:05:24.743174 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:24.728816 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19456502-09eb-4dc7-8877-395f1b5e7218-credential-socket\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz\" (UID: \"19456502-09eb-4dc7-8877-395f1b5e7218\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" Nov 01 16:05:24.743174 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:24.729534 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19456502-09eb-4dc7-8877-395f1b5e7218-istiod-ca-cert\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz\" (UID: \"19456502-09eb-4dc7-8877-395f1b5e7218\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" Nov 01 16:05:24.743174 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:24.729803 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19456502-09eb-4dc7-8877-395f1b5e7218-workload-certs\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz\" (UID: \"19456502-09eb-4dc7-8877-395f1b5e7218\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" Nov 01 16:05:24.778089 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:24.758098 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19456502-09eb-4dc7-8877-395f1b5e7218-istio-data\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz\" (UID: \"19456502-09eb-4dc7-8877-395f1b5e7218\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" Nov 01 16:05:24.778089 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:24.758381 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19456502-09eb-4dc7-8877-395f1b5e7218-workload-socket\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz\" (UID: \"19456502-09eb-4dc7-8877-395f1b5e7218\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" Nov 01 16:05:25.000437 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:24.944131 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:05:25.000437 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:24.955346 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:05:25.000437 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:24.982388 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:05:25.325759 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:25.189064 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zc89\" (UniqueName: \"kubernetes.io/projected/19456502-09eb-4dc7-8877-395f1b5e7218-kube-api-access-5zc89\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz\" (UID: \"19456502-09eb-4dc7-8877-395f1b5e7218\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" Nov 01 16:05:25.325759 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:25.321769 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19456502-09eb-4dc7-8877-395f1b5e7218-istio-envoy\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz\" (UID: \"19456502-09eb-4dc7-8877-395f1b5e7218\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" Nov 01 16:05:25.330838 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:25.329341 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19456502-09eb-4dc7-8877-395f1b5e7218-istio-token\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz\" (UID: \"19456502-09eb-4dc7-8877-395f1b5e7218\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" Nov 01 16:05:25.330838 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:25.330134 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19456502-09eb-4dc7-8877-395f1b5e7218-istio-podinfo\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz\" (UID: \"19456502-09eb-4dc7-8877-395f1b5e7218\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" Nov 01 16:05:25.363843 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:25.363514 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:05:25.365339 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:25.364263 4363 generic.go:356] "Generic (PLEG): container finished" podID="519e328e-0e0a-4fc3-b608-b3ff05e24c88" containerID="8f4b2e60d6bd6963643aa6f757d6a338f275c163b358a254dc70db22d5e6f410" exitCode=0 Nov 01 16:05:25.424003 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:25.372115 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ed26f60-cdbf-421e-be76-dc5e4682115f" path="/var/lib/kubelet/pods/3ed26f60-cdbf-421e-be76-dc5e4682115f/volumes" Nov 01 16:05:25.424003 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:25.372713 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 16:05:25.424003 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:25.372792 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-474cab5e-85488d9c5c-btq6b" event={"ID":"519e328e-0e0a-4fc3-b608-b3ff05e24c88","Type":"ContainerDied","Data":"8f4b2e60d6bd6963643aa6f757d6a338f275c163b358a254dc70db22d5e6f410"} Nov 01 16:05:25.424003 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:25.372893 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 16:05:25.424003 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:25.372949 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 16:05:25.424003 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:25.372999 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:05:25.424003 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:25.411927 4363 generic.go:356] "Generic (PLEG): container finished" podID="a3367eda-e540-4fca-a638-9571e68141eb" containerID="312e2a494c1a3171477bfe17560ebea8af1fefae98cc0475813af27d0e92b863" exitCode=0 Nov 01 16:05:25.424003 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:25.413840 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fd01d20-6f66b5869b-t5dkr" event={"ID":"a3367eda-e540-4fca-a638-9571e68141eb","Type":"ContainerDied","Data":"312e2a494c1a3171477bfe17560ebea8af1fefae98cc0475813af27d0e92b863"} Nov 01 16:05:25.428916 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:25.428792 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9fd01d20-7b479cbc8c-mt2d2" Nov 01 16:05:25.443130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:25.443106 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9fd01d20-755c777847-lqtlx" Nov 01 16:05:25.623039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:25.616384 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" Nov 01 16:05:25.719966 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:25.638935 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8f0e6f8-af57-4ae0-9f2a-f22b87e77028-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8f0e6f8-af57-4ae0-9f2a-f22b87e77028-plugins-config-volume\") pod \"b8f0e6f8-af57-4ae0-9f2a-f22b87e77028\" (UID: \"b8f0e6f8-af57-4ae0-9f2a-f22b87e77028\") " Nov 01 16:05:25.719966 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:25.639233 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1fff68b-27c4-40e5-a6a0-61e1b8e966b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1fff68b-27c4-40e5-a6a0-61e1b8e966b6-plugins-config-volume\") pod \"e1fff68b-27c4-40e5-a6a0-61e1b8e966b6\" (UID: \"e1fff68b-27c4-40e5-a6a0-61e1b8e966b6\") " Nov 01 16:05:25.719966 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:25.639277 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1fff68b-27c4-40e5-a6a0-61e1b8e966b6-kube-api-access-fbb9q\" (UniqueName: \"kubernetes.io/projected/e1fff68b-27c4-40e5-a6a0-61e1b8e966b6-kube-api-access-fbb9q\") pod \"e1fff68b-27c4-40e5-a6a0-61e1b8e966b6\" (UID: \"e1fff68b-27c4-40e5-a6a0-61e1b8e966b6\") " Nov 01 16:05:25.719966 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:25.639360 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8f0e6f8-af57-4ae0-9f2a-f22b87e77028-kube-api-access-7z5hj\" (UniqueName: \"kubernetes.io/projected/b8f0e6f8-af57-4ae0-9f2a-f22b87e77028-kube-api-access-7z5hj\") pod \"b8f0e6f8-af57-4ae0-9f2a-f22b87e77028\" (UID: \"b8f0e6f8-af57-4ae0-9f2a-f22b87e77028\") " Nov 01 16:05:25.719966 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:25.669815 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8f0e6f8-af57-4ae0-9f2a-f22b87e77028-plugins-config-volume" pod "b8f0e6f8-af57-4ae0-9f2a-f22b87e77028" (UID: "b8f0e6f8-af57-4ae0-9f2a-f22b87e77028"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:05:25.719966 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:25.670451 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1fff68b-27c4-40e5-a6a0-61e1b8e966b6-plugins-config-volume" pod "e1fff68b-27c4-40e5-a6a0-61e1b8e966b6" (UID: "e1fff68b-27c4-40e5-a6a0-61e1b8e966b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:05:25.719966 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:25.712464 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1fff68b-27c4-40e5-a6a0-61e1b8e966b6-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:25.719966 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:25.712491 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8f0e6f8-af57-4ae0-9f2a-f22b87e77028-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:25.788533 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:25.788468 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1fff68b-27c4-40e5-a6a0-61e1b8e966b6-kube-api-access-fbb9q" pod "e1fff68b-27c4-40e5-a6a0-61e1b8e966b6" (UID: "e1fff68b-27c4-40e5-a6a0-61e1b8e966b6"). InnerVolumeSpecName "kube-api-access-fbb9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:05:25.853711 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:25.816070 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbb9q\" (UniqueName: \"kubernetes.io/projected/e1fff68b-27c4-40e5-a6a0-61e1b8e966b6-kube-api-access-fbb9q\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:26.034245 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.034192 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8f0e6f8-af57-4ae0-9f2a-f22b87e77028-kube-api-access-7z5hj" pod "b8f0e6f8-af57-4ae0-9f2a-f22b87e77028" (UID: "b8f0e6f8-af57-4ae0-9f2a-f22b87e77028"). InnerVolumeSpecName "kube-api-access-7z5hj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:05:26.041594 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.041532 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2a32fd7-qdlkv"] Nov 01 16:05:26.053087 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.052364 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2a32fd7-qdlkv"] Nov 01 16:05:26.066495 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.066461 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z5hj\" (UniqueName: \"kubernetes.io/projected/b8f0e6f8-af57-4ae0-9f2a-f22b87e77028-kube-api-access-7z5hj\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:26.146843 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.146030 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-474cab5e-85488d9c5c-btq6b" Nov 01 16:05:26.161207 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.160963 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fd01d20-6f66b5869b-t5dkr" Nov 01 16:05:26.165484 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.165461 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-474cab5e-79b66887b7-brn4j" Nov 01 16:05:26.268917 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.268881 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3367eda-e540-4fca-a638-9571e68141eb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3367eda-e540-4fca-a638-9571e68141eb-openshift-service-ca\") pod \"a3367eda-e540-4fca-a638-9571e68141eb\" (UID: \"a3367eda-e540-4fca-a638-9571e68141eb\") " Nov 01 16:05:26.269123 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.269029 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a49dcb4b-99a0-4939-9e0e-c844ce2e8d83-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a49dcb4b-99a0-4939-9e0e-c844ce2e8d83-plugins-config-volume\") pod \"a49dcb4b-99a0-4939-9e0e-c844ce2e8d83\" (UID: \"a49dcb4b-99a0-4939-9e0e-c844ce2e8d83\") " Nov 01 16:05:26.269123 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.269106 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/519e328e-0e0a-4fc3-b608-b3ff05e24c88-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/519e328e-0e0a-4fc3-b608-b3ff05e24c88-plugins-config-volume\") pod \"519e328e-0e0a-4fc3-b608-b3ff05e24c88\" (UID: \"519e328e-0e0a-4fc3-b608-b3ff05e24c88\") " Nov 01 16:05:26.269254 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.269175 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/519e328e-0e0a-4fc3-b608-b3ff05e24c88-kube-api-access-4rhpp\" (UniqueName: \"kubernetes.io/projected/519e328e-0e0a-4fc3-b608-b3ff05e24c88-kube-api-access-4rhpp\") pod \"519e328e-0e0a-4fc3-b608-b3ff05e24c88\" (UID: \"519e328e-0e0a-4fc3-b608-b3ff05e24c88\") " Nov 01 16:05:26.269254 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.269223 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3367eda-e540-4fca-a638-9571e68141eb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3367eda-e540-4fca-a638-9571e68141eb-maas-api-tls\") pod \"a3367eda-e540-4fca-a638-9571e68141eb\" (UID: \"a3367eda-e540-4fca-a638-9571e68141eb\") " Nov 01 16:05:26.269465 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.269438 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a49dcb4b-99a0-4939-9e0e-c844ce2e8d83-plugins-config-volume" pod "a49dcb4b-99a0-4939-9e0e-c844ce2e8d83" (UID: "a49dcb4b-99a0-4939-9e0e-c844ce2e8d83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:05:26.269465 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.269455 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a49dcb4b-99a0-4939-9e0e-c844ce2e8d83-kube-api-access-72cpp\" (UniqueName: \"kubernetes.io/projected/a49dcb4b-99a0-4939-9e0e-c844ce2e8d83-kube-api-access-72cpp\") pod \"a49dcb4b-99a0-4939-9e0e-c844ce2e8d83\" (UID: \"a49dcb4b-99a0-4939-9e0e-c844ce2e8d83\") " Nov 01 16:05:26.269612 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.269507 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3367eda-e540-4fca-a638-9571e68141eb-kube-api-access-fwfbx\" (UniqueName: \"kubernetes.io/projected/a3367eda-e540-4fca-a638-9571e68141eb-kube-api-access-fwfbx\") pod \"a3367eda-e540-4fca-a638-9571e68141eb\" (UID: \"a3367eda-e540-4fca-a638-9571e68141eb\") " Nov 01 16:05:26.269612 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.269534 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3367eda-e540-4fca-a638-9571e68141eb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3367eda-e540-4fca-a638-9571e68141eb-metrics-certs\") pod \"a3367eda-e540-4fca-a638-9571e68141eb\" (UID: \"a3367eda-e540-4fca-a638-9571e68141eb\") " Nov 01 16:05:26.269723 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.269658 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3367eda-e540-4fca-a638-9571e68141eb-openshift-service-ca" pod "a3367eda-e540-4fca-a638-9571e68141eb" (UID: "a3367eda-e540-4fca-a638-9571e68141eb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:05:26.269787 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.269735 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/519e328e-0e0a-4fc3-b608-b3ff05e24c88-plugins-config-volume" pod "519e328e-0e0a-4fc3-b608-b3ff05e24c88" (UID: "519e328e-0e0a-4fc3-b608-b3ff05e24c88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:05:26.270438 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.270372 4363 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3367eda-e540-4fca-a638-9571e68141eb-openshift-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:26.270438 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.270398 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a49dcb4b-99a0-4939-9e0e-c844ce2e8d83-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:26.270438 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.270414 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/519e328e-0e0a-4fc3-b608-b3ff05e24c88-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:26.272195 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.272163 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3367eda-e540-4fca-a638-9571e68141eb-kube-api-access-fwfbx" pod "a3367eda-e540-4fca-a638-9571e68141eb" (UID: "a3367eda-e540-4fca-a638-9571e68141eb"). InnerVolumeSpecName "kube-api-access-fwfbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:05:26.272303 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.272279 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3367eda-e540-4fca-a638-9571e68141eb-maas-api-tls" pod "a3367eda-e540-4fca-a638-9571e68141eb" (UID: "a3367eda-e540-4fca-a638-9571e68141eb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:05:26.272903 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.272859 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a49dcb4b-99a0-4939-9e0e-c844ce2e8d83-kube-api-access-72cpp" pod "a49dcb4b-99a0-4939-9e0e-c844ce2e8d83" (UID: "a49dcb4b-99a0-4939-9e0e-c844ce2e8d83"). InnerVolumeSpecName "kube-api-access-72cpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:05:26.273067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.273007 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3367eda-e540-4fca-a638-9571e68141eb-metrics-certs" pod "a3367eda-e540-4fca-a638-9571e68141eb" (UID: "a3367eda-e540-4fca-a638-9571e68141eb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:05:26.273138 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.273112 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/519e328e-0e0a-4fc3-b608-b3ff05e24c88-kube-api-access-4rhpp" pod "519e328e-0e0a-4fc3-b608-b3ff05e24c88" (UID: "519e328e-0e0a-4fc3-b608-b3ff05e24c88"). InnerVolumeSpecName "kube-api-access-4rhpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:05:26.382879 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.382745 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72cpp\" (UniqueName: \"kubernetes.io/projected/a49dcb4b-99a0-4939-9e0e-c844ce2e8d83-kube-api-access-72cpp\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:26.382879 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.382789 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwfbx\" (UniqueName: \"kubernetes.io/projected/a3367eda-e540-4fca-a638-9571e68141eb-kube-api-access-fwfbx\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:26.382879 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.382806 4363 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3367eda-e540-4fca-a638-9571e68141eb-metrics-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:26.382879 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.382823 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rhpp\" (UniqueName: \"kubernetes.io/projected/519e328e-0e0a-4fc3-b608-b3ff05e24c88-kube-api-access-4rhpp\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:26.382879 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.382837 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3367eda-e540-4fca-a638-9571e68141eb-maas-api-tls\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:26.482595 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:05:26.480272 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19456502_09eb_4dc7_8877_395f1b5e7218.slice/crio-9e804e12cc576640c0e8b1dc53da53a1553ae04ae34311662d6329928754f346 WatchSource:0}: Error finding container 9e804e12cc576640c0e8b1dc53da53a1553ae04ae34311662d6329928754f346: Status 404 returned error can't find the container with id 9e804e12cc576640c0e8b1dc53da53a1553ae04ae34311662d6329928754f346 Nov 01 16:05:26.482595 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.482262 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9fd01d20-755c777847-lqtlx" event={"ID":"b8f0e6f8-af57-4ae0-9f2a-f22b87e77028","Type":"ContainerDied","Data":"6238c89ed797d89de59f412b6d23575da2f2fc5bf0b47080f4a590ccf827f26d"} Nov 01 16:05:26.482595 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.482327 4363 scope.go:122] "RemoveContainer" containerID="cc48d4b914473c156ea6333a78d391d693cbefc38f97c1d52a6af47217b49e1c" Nov 01 16:05:26.482595 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.482273 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9fd01d20-755c777847-lqtlx" Nov 01 16:05:26.487977 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.487949 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz"] Nov 01 16:05:26.490104 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.490074 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9fd01d20-7b479cbc8c-mt2d2" event={"ID":"e1fff68b-27c4-40e5-a6a0-61e1b8e966b6","Type":"ContainerDied","Data":"ddd97b9667e0bd58f3a363f3ad2b34b47fbe897dc457a05521f55d3bf7bd5100"} Nov 01 16:05:26.490234 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.490185 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9fd01d20-7b479cbc8c-mt2d2" Nov 01 16:05:26.502943 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.502446 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:05:26.502943 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.502525 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:05:26.502943 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.502567 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:05:26.504129 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.503487 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-474cab5e-79b66887b7-brn4j" Nov 01 16:05:26.504129 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.503556 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-474cab5e-79b66887b7-brn4j" event={"ID":"a49dcb4b-99a0-4939-9e0e-c844ce2e8d83","Type":"ContainerDied","Data":"11dcf92525217c75424df0fde26c75b8908cfe265dc5cb6d0ed03bf84dce596d"} Nov 01 16:05:26.511069 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.505957 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-474cab5e-85488d9c5c-btq6b" Nov 01 16:05:26.511069 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.507290 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-474cab5e-85488d9c5c-btq6b" event={"ID":"519e328e-0e0a-4fc3-b608-b3ff05e24c88","Type":"ContainerDied","Data":"b01c24439ec68b708b48afaf0cd80210a8fcccebf125914015daf86b0a8f7d7b"} Nov 01 16:05:26.528073 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.528004 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fd01d20-6f66b5869b-t5dkr" Nov 01 16:05:26.529381 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.529048 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fd01d20-6f66b5869b-t5dkr" event={"ID":"a3367eda-e540-4fca-a638-9571e68141eb","Type":"ContainerDied","Data":"57234f431e78b9522d8fb157c07aedd123ae97d8ba1e287bc7903e71965b19a7"} Nov 01 16:05:26.546308 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.546274 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9fd01d20-755c777847-lqtlx"] Nov 01 16:05:26.555285 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.554607 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9fd01d20-755c777847-lqtlx"] Nov 01 16:05:26.555463 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.555403 4363 scope.go:122] "RemoveContainer" containerID="7e9c94fb78110d48c999ad05c8ab0bdc384ce442052d3f84afcb132aab32487e" Nov 01 16:05:26.573175 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.570539 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9fd01d20-7b479cbc8c-mt2d2"] Nov 01 16:05:26.576638 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.576606 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9fd01d20-7b479cbc8c-mt2d2"] Nov 01 16:05:26.583432 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.583408 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-474cab5e-85488d9c5c-btq6b"] Nov 01 16:05:26.597217 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.597197 4363 scope.go:122] "RemoveContainer" containerID="71d8eeaa23a90d433a76703e16ba90954a6fe8159ed71c8cb35299edd6b17cf1" Nov 01 16:05:26.597972 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.597950 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-474cab5e-85488d9c5c-btq6b"] Nov 01 16:05:26.612030 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.611077 4363 scope.go:122] "RemoveContainer" containerID="8f4b2e60d6bd6963643aa6f757d6a338f275c163b358a254dc70db22d5e6f410" Nov 01 16:05:26.617058 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.616912 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-383fde-6f8cc9c655-m67lx"] Nov 01 16:05:26.618385 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.618297 4363 cpu_manager.go:414] "containerMap: removing container" podUID="b8f0e6f8-af57-4ae0-9f2a-f22b87e77028" containerName="payload-pre-processing" Nov 01 16:05:26.618385 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.618341 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8f0e6f8-af57-4ae0-9f2a-f22b87e77028" containerName="payload-pre-processing" Nov 01 16:05:26.618385 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.618368 4363 cpu_manager.go:414] "containerMap: removing container" podUID="e1fff68b-27c4-40e5-a6a0-61e1b8e966b6" containerName="payload-processing" Nov 01 16:05:26.618385 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.618376 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1fff68b-27c4-40e5-a6a0-61e1b8e966b6" containerName="payload-processing" Nov 01 16:05:26.618632 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.618416 4363 cpu_manager.go:414] "containerMap: removing container" podUID="a49dcb4b-99a0-4939-9e0e-c844ce2e8d83" containerName="payload-pre-processing" Nov 01 16:05:26.618632 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.618427 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a49dcb4b-99a0-4939-9e0e-c844ce2e8d83" containerName="payload-pre-processing" Nov 01 16:05:26.618632 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.618438 4363 cpu_manager.go:414] "containerMap: removing container" podUID="a3367eda-e540-4fca-a638-9571e68141eb" containerName="maas-api" Nov 01 16:05:26.618632 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.618445 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3367eda-e540-4fca-a638-9571e68141eb" containerName="maas-api" Nov 01 16:05:26.618632 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.618461 4363 cpu_manager.go:414] "containerMap: removing container" podUID="519e328e-0e0a-4fc3-b608-b3ff05e24c88" containerName="payload-processing" Nov 01 16:05:26.618632 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.618478 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="519e328e-0e0a-4fc3-b608-b3ff05e24c88" containerName="payload-processing" Nov 01 16:05:26.618915 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.618675 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8f0e6f8-af57-4ae0-9f2a-f22b87e77028" containerName="payload-pre-processing" Nov 01 16:05:26.618915 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.618689 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1fff68b-27c4-40e5-a6a0-61e1b8e966b6" containerName="payload-processing" Nov 01 16:05:26.618915 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.618700 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="a49dcb4b-99a0-4939-9e0e-c844ce2e8d83" containerName="payload-pre-processing" Nov 01 16:05:26.618915 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.618711 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3367eda-e540-4fca-a638-9571e68141eb" containerName="maas-api" Nov 01 16:05:26.618915 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.618721 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="519e328e-0e0a-4fc3-b608-b3ff05e24c88" containerName="payload-processing" Nov 01 16:05:26.632449 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.632420 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-474cab5e-79b66887b7-brn4j"] Nov 01 16:05:26.632584 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.632538 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-383fde-6f8cc9c655-m67lx" Nov 01 16:05:26.634129 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.634111 4363 scope.go:122] "RemoveContainer" containerID="312e2a494c1a3171477bfe17560ebea8af1fefae98cc0475813af27d0e92b863" Nov 01 16:05:26.639201 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.639069 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-383fde\"" Nov 01 16:05:26.639201 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.639198 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-383fde-dockercfg-84sw5\"" Nov 01 16:05:26.640627 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.640605 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-383fde-6f8cc9c655-m67lx"] Nov 01 16:05:26.648369 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.648299 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-474cab5e-79b66887b7-brn4j"] Nov 01 16:05:26.666981 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.666947 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9fd01d20-6f66b5869b-t5dkr"] Nov 01 16:05:26.674351 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.674250 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9fd01d20-6f66b5869b-t5dkr"] Nov 01 16:05:26.686046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.683552 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-383fde-8486d66857-b9mwl"] Nov 01 16:05:26.721127 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.699528 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm5k4\" (UniqueName: \"kubernetes.io/projected/79cee252-8c24-49ac-b432-49de53bb30aa-kube-api-access-wm5k4\") pod \"payload-pre-processing-e2e-shared-a-w5-383fde-6f8cc9c655-m67lx\" (UID: \"79cee252-8c24-49ac-b432-49de53bb30aa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-383fde-6f8cc9c655-m67lx" Nov 01 16:05:26.721127 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.699762 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79cee252-8c24-49ac-b432-49de53bb30aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-383fde-6f8cc9c655-m67lx\" (UID: \"79cee252-8c24-49ac-b432-49de53bb30aa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-383fde-6f8cc9c655-m67lx" Nov 01 16:05:26.721127 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.700343 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-383fde-8486d66857-b9mwl" Nov 01 16:05:26.764247 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:26.744058 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-383fde-8486d66857-b9mwl"] Nov 01 16:05:27.165258 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:27.165223 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="519e328e-0e0a-4fc3-b608-b3ff05e24c88" path="/var/lib/kubelet/pods/519e328e-0e0a-4fc3-b608-b3ff05e24c88/volumes" Nov 01 16:05:27.167044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:27.167006 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="596d6128-0721-46bc-8267-ae85b4c9f82a" path="/var/lib/kubelet/pods/596d6128-0721-46bc-8267-ae85b4c9f82a/volumes" Nov 01 16:05:27.168778 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:27.168756 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3367eda-e540-4fca-a638-9571e68141eb" path="/var/lib/kubelet/pods/a3367eda-e540-4fca-a638-9571e68141eb/volumes" Nov 01 16:05:27.170406 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:27.170386 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a49dcb4b-99a0-4939-9e0e-c844ce2e8d83" path="/var/lib/kubelet/pods/a49dcb4b-99a0-4939-9e0e-c844ce2e8d83/volumes" Nov 01 16:05:27.171339 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:27.171321 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8f0e6f8-af57-4ae0-9f2a-f22b87e77028" path="/var/lib/kubelet/pods/b8f0e6f8-af57-4ae0-9f2a-f22b87e77028/volumes" Nov 01 16:05:27.172505 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:27.172483 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1fff68b-27c4-40e5-a6a0-61e1b8e966b6" path="/var/lib/kubelet/pods/e1fff68b-27c4-40e5-a6a0-61e1b8e966b6/volumes" Nov 01 16:05:27.211308 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:27.211273 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79cee252-8c24-49ac-b432-49de53bb30aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-383fde-6f8cc9c655-m67lx\" (UID: \"79cee252-8c24-49ac-b432-49de53bb30aa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-383fde-6f8cc9c655-m67lx" Nov 01 16:05:27.211677 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:27.211651 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs4sm\" (UniqueName: \"kubernetes.io/projected/ec15b77d-c274-4eb4-bc47-6134110a0a05-kube-api-access-vs4sm\") pod \"payload-processing-e2e-shared-a-w5-383fde-8486d66857-b9mwl\" (UID: \"ec15b77d-c274-4eb4-bc47-6134110a0a05\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-383fde-8486d66857-b9mwl" Nov 01 16:05:27.211837 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:27.211821 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec15b77d-c274-4eb4-bc47-6134110a0a05-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-383fde-8486d66857-b9mwl\" (UID: \"ec15b77d-c274-4eb4-bc47-6134110a0a05\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-383fde-8486d66857-b9mwl" Nov 01 16:05:27.212057 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:27.212041 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm5k4\" (UniqueName: \"kubernetes.io/projected/79cee252-8c24-49ac-b432-49de53bb30aa-kube-api-access-wm5k4\") pod \"payload-pre-processing-e2e-shared-a-w5-383fde-6f8cc9c655-m67lx\" (UID: \"79cee252-8c24-49ac-b432-49de53bb30aa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-383fde-6f8cc9c655-m67lx" Nov 01 16:05:27.215053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:27.214134 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79cee252-8c24-49ac-b432-49de53bb30aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-383fde-6f8cc9c655-m67lx\" (UID: \"79cee252-8c24-49ac-b432-49de53bb30aa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-383fde-6f8cc9c655-m67lx" Nov 01 16:05:27.225951 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:27.225929 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm5k4\" (UniqueName: \"kubernetes.io/projected/79cee252-8c24-49ac-b432-49de53bb30aa-kube-api-access-wm5k4\") pod \"payload-pre-processing-e2e-shared-a-w5-383fde-6f8cc9c655-m67lx\" (UID: \"79cee252-8c24-49ac-b432-49de53bb30aa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-383fde-6f8cc9c655-m67lx" Nov 01 16:05:27.314213 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:27.314168 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs4sm\" (UniqueName: \"kubernetes.io/projected/ec15b77d-c274-4eb4-bc47-6134110a0a05-kube-api-access-vs4sm\") pod \"payload-processing-e2e-shared-a-w5-383fde-8486d66857-b9mwl\" (UID: \"ec15b77d-c274-4eb4-bc47-6134110a0a05\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-383fde-8486d66857-b9mwl" Nov 01 16:05:27.314398 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:27.314313 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec15b77d-c274-4eb4-bc47-6134110a0a05-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-383fde-8486d66857-b9mwl\" (UID: \"ec15b77d-c274-4eb4-bc47-6134110a0a05\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-383fde-8486d66857-b9mwl" Nov 01 16:05:27.315281 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:27.315247 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec15b77d-c274-4eb4-bc47-6134110a0a05-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-383fde-8486d66857-b9mwl\" (UID: \"ec15b77d-c274-4eb4-bc47-6134110a0a05\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-383fde-8486d66857-b9mwl" Nov 01 16:05:27.322855 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:27.322823 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs4sm\" (UniqueName: \"kubernetes.io/projected/ec15b77d-c274-4eb4-bc47-6134110a0a05-kube-api-access-vs4sm\") pod \"payload-processing-e2e-shared-a-w5-383fde-8486d66857-b9mwl\" (UID: \"ec15b77d-c274-4eb4-bc47-6134110a0a05\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-383fde-8486d66857-b9mwl" Nov 01 16:05:27.370707 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:27.369346 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-383fde-6f8cc9c655-m67lx" Nov 01 16:05:27.543038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:27.537311 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-383fde-8486d66857-b9mwl" Nov 01 16:05:27.573166 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:27.571783 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" event={"ID":"19456502-09eb-4dc7-8877-395f1b5e7218","Type":"ContainerStarted","Data":"a5bfdb3891ac394f0b406fe74719eba6447cb1299f7d34d424160ae4e9975d32"} Nov 01 16:05:27.573166 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:27.571846 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" event={"ID":"19456502-09eb-4dc7-8877-395f1b5e7218","Type":"ContainerStarted","Data":"9e804e12cc576640c0e8b1dc53da53a1553ae04ae34311662d6329928754f346"} Nov 01 16:05:27.602318 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:27.598981 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" podStartSLOduration=4.5989591789999995 podStartE2EDuration="4.598959179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:05:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:05:27.593094121 +0000 UTC m=+1060.924831773" watchObservedRunningTime="2026-11-01 16:05:27.598959179 +0000 UTC m=+1060.930696830" Nov 01 16:05:27.650707 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:27.649379 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" Nov 01 16:05:27.662826 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:27.662066 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" Nov 01 16:05:27.877879 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:27.877850 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-383fde-6f8cc9c655-m67lx"] Nov 01 16:05:28.083204 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:05:28.083157 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec15b77d_c274_4eb4_bc47_6134110a0a05.slice/crio-c1869c041c031f18fa6bcab78beea6d07d3c77bbc7c2bd0d4ac7bec911a9908c WatchSource:0}: Error finding container c1869c041c031f18fa6bcab78beea6d07d3c77bbc7c2bd0d4ac7bec911a9908c: Status 404 returned error can't find the container with id c1869c041c031f18fa6bcab78beea6d07d3c77bbc7c2bd0d4ac7bec911a9908c Nov 01 16:05:28.084483 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:28.084442 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-383fde-8486d66857-b9mwl"] Nov 01 16:05:28.101156 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:28.101129 4363 patch_prober.go:29] interesting pod/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:05:28.101264 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:28.101179 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54" podUID="a4504fd0-ac35-4f52-8403-beb7c42daa96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:05:28.589948 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:28.589858 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-383fde-6f8cc9c655-m67lx" event={"ID":"79cee252-8c24-49ac-b432-49de53bb30aa","Type":"ContainerStarted","Data":"8654f5f4d9f3f527465e938614cabe5546bd8eef384dd6c16c9b6811215bf12d"} Nov 01 16:05:28.589948 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:28.589924 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-383fde-6f8cc9c655-m67lx" event={"ID":"79cee252-8c24-49ac-b432-49de53bb30aa","Type":"ContainerStarted","Data":"1655ee3d8e082c171c872b12f7f520e77feb87496f618f68f4ef6a31cdee76fa"} Nov 01 16:05:28.590406 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:28.590079 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-383fde-6f8cc9c655-m67lx" Nov 01 16:05:28.592899 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:28.592831 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-383fde-8486d66857-b9mwl" event={"ID":"ec15b77d-c274-4eb4-bc47-6134110a0a05","Type":"ContainerStarted","Data":"f84727879a636d69b433d1c9e69e314a942a751a03949fc139bc808a056dabd6"} Nov 01 16:05:28.592899 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:28.592891 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-383fde-8486d66857-b9mwl" event={"ID":"ec15b77d-c274-4eb4-bc47-6134110a0a05","Type":"ContainerStarted","Data":"c1869c041c031f18fa6bcab78beea6d07d3c77bbc7c2bd0d4ac7bec911a9908c"} Nov 01 16:05:28.593367 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:28.593312 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" Nov 01 16:05:28.594298 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:28.594276 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" Nov 01 16:05:28.612563 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:28.612504 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-383fde-6f8cc9c655-m67lx" podStartSLOduration=2.612487849 podStartE2EDuration="2.612487849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:05:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:05:28.606653546 +0000 UTC m=+1061.938391197" watchObservedRunningTime="2026-11-01 16:05:28.612487849 +0000 UTC m=+1061.944225499" Nov 01 16:05:29.605566 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:29.605530 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-383fde-8486d66857-b9mwl" Nov 01 16:05:31.183648 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:31.183548 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-383fde-8486d66857-b9mwl" podStartSLOduration=5.183525038 podStartE2EDuration="5.183525038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:05:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:05:28.677795163 +0000 UTC m=+1062.009532814" watchObservedRunningTime="2026-11-01 16:05:31.183525038 +0000 UTC m=+1064.515262690" Nov 01 16:05:31.185636 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:31.185594 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5e2d03-cbd9bb6f9-bbprx"] Nov 01 16:05:31.229997 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:31.229960 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5e2d03-cbd9bb6f9-bbprx"] Nov 01 16:05:31.230209 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:31.230034 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5e2d03-757556fddc-j9gph"] Nov 01 16:05:31.230344 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:31.230324 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5e2d03-cbd9bb6f9-bbprx" Nov 01 16:05:31.241089 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:31.241062 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-5e2d03\"" Nov 01 16:05:31.241249 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:31.241219 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-5e2d03-dockercfg-xwd7c\"" Nov 01 16:05:31.259987 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:31.259946 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5e2d03-757556fddc-j9gph"] Nov 01 16:05:31.260117 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:31.260082 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5e2d03-757556fddc-j9gph" Nov 01 16:05:31.342042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:31.335557 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59skl\" (UniqueName: \"kubernetes.io/projected/96c20c16-b48c-40b5-853c-edddd1225652-kube-api-access-59skl\") pod \"payload-processing-e2e-shared-a-w0-5e2d03-cbd9bb6f9-bbprx\" (UID: \"96c20c16-b48c-40b5-853c-edddd1225652\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5e2d03-cbd9bb6f9-bbprx" Nov 01 16:05:31.342042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:31.335883 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96c20c16-b48c-40b5-853c-edddd1225652-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-5e2d03-cbd9bb6f9-bbprx\" (UID: \"96c20c16-b48c-40b5-853c-edddd1225652\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5e2d03-cbd9bb6f9-bbprx" Nov 01 16:05:31.499424 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:31.450671 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96c20c16-b48c-40b5-853c-edddd1225652-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-5e2d03-cbd9bb6f9-bbprx\" (UID: \"96c20c16-b48c-40b5-853c-edddd1225652\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5e2d03-cbd9bb6f9-bbprx" Nov 01 16:05:31.499424 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:31.450762 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57dbe32d-78fb-4168-a5f8-5fa0a38f2e2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-5e2d03-757556fddc-j9gph\" (UID: \"57dbe32d-78fb-4168-a5f8-5fa0a38f2e2b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5e2d03-757556fddc-j9gph" Nov 01 16:05:31.499424 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:31.450859 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l5k8\" (UniqueName: \"kubernetes.io/projected/57dbe32d-78fb-4168-a5f8-5fa0a38f2e2b-kube-api-access-4l5k8\") pod \"payload-pre-processing-e2e-shared-a-w0-5e2d03-757556fddc-j9gph\" (UID: \"57dbe32d-78fb-4168-a5f8-5fa0a38f2e2b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5e2d03-757556fddc-j9gph" Nov 01 16:05:31.499424 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:31.450913 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59skl\" (UniqueName: \"kubernetes.io/projected/96c20c16-b48c-40b5-853c-edddd1225652-kube-api-access-59skl\") pod \"payload-processing-e2e-shared-a-w0-5e2d03-cbd9bb6f9-bbprx\" (UID: \"96c20c16-b48c-40b5-853c-edddd1225652\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5e2d03-cbd9bb6f9-bbprx" Nov 01 16:05:31.507764 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:31.507648 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59skl\" (UniqueName: \"kubernetes.io/projected/96c20c16-b48c-40b5-853c-edddd1225652-kube-api-access-59skl\") pod \"payload-processing-e2e-shared-a-w0-5e2d03-cbd9bb6f9-bbprx\" (UID: \"96c20c16-b48c-40b5-853c-edddd1225652\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5e2d03-cbd9bb6f9-bbprx" Nov 01 16:05:31.704892 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:31.561336 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57dbe32d-78fb-4168-a5f8-5fa0a38f2e2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-5e2d03-757556fddc-j9gph\" (UID: \"57dbe32d-78fb-4168-a5f8-5fa0a38f2e2b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5e2d03-757556fddc-j9gph" Nov 01 16:05:31.704892 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:31.561469 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l5k8\" (UniqueName: \"kubernetes.io/projected/57dbe32d-78fb-4168-a5f8-5fa0a38f2e2b-kube-api-access-4l5k8\") pod \"payload-pre-processing-e2e-shared-a-w0-5e2d03-757556fddc-j9gph\" (UID: \"57dbe32d-78fb-4168-a5f8-5fa0a38f2e2b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5e2d03-757556fddc-j9gph" Nov 01 16:05:31.704892 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:31.562971 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57dbe32d-78fb-4168-a5f8-5fa0a38f2e2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-5e2d03-757556fddc-j9gph\" (UID: \"57dbe32d-78fb-4168-a5f8-5fa0a38f2e2b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5e2d03-757556fddc-j9gph" Nov 01 16:05:31.704892 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:31.592294 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96c20c16-b48c-40b5-853c-edddd1225652-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-5e2d03-cbd9bb6f9-bbprx\" (UID: \"96c20c16-b48c-40b5-853c-edddd1225652\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5e2d03-cbd9bb6f9-bbprx" Nov 01 16:05:31.747042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:31.737926 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l5k8\" (UniqueName: \"kubernetes.io/projected/57dbe32d-78fb-4168-a5f8-5fa0a38f2e2b-kube-api-access-4l5k8\") pod \"payload-pre-processing-e2e-shared-a-w0-5e2d03-757556fddc-j9gph\" (UID: \"57dbe32d-78fb-4168-a5f8-5fa0a38f2e2b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5e2d03-757556fddc-j9gph" Nov 01 16:05:31.877332 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:31.877291 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5e2d03-cbd9bb6f9-bbprx" Nov 01 16:05:31.907231 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:31.907047 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5e2d03-757556fddc-j9gph" Nov 01 16:05:32.254556 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:32.254185 4363 patch_prober.go:29] interesting pod/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:05:32.254556 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:32.254251 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9" podUID="86b37143-20ce-45d3-90d7-45b451012c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:05:32.380038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:32.379990 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5e2d03-cbd9bb6f9-bbprx"] Nov 01 16:05:32.391285 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:05:32.391250 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96c20c16_b48c_40b5_853c_edddd1225652.slice/crio-7db87934577925cfa89c63f8165023624730a14d82f86d43aa15e62dc0b26d29 WatchSource:0}: Error finding container 7db87934577925cfa89c63f8165023624730a14d82f86d43aa15e62dc0b26d29: Status 404 returned error can't find the container with id 7db87934577925cfa89c63f8165023624730a14d82f86d43aa15e62dc0b26d29 Nov 01 16:05:32.448928 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:32.448570 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5e2d03-757556fddc-j9gph"] Nov 01 16:05:32.449641 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:05:32.449609 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57dbe32d_78fb_4168_a5f8_5fa0a38f2e2b.slice/crio-7860d62f3148186e78814fd0206af3ecdda28b241dfd2e9f930fe83660c88f3e WatchSource:0}: Error finding container 7860d62f3148186e78814fd0206af3ecdda28b241dfd2e9f930fe83660c88f3e: Status 404 returned error can't find the container with id 7860d62f3148186e78814fd0206af3ecdda28b241dfd2e9f930fe83660c88f3e Nov 01 16:05:32.544173 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:32.543878 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv"] Nov 01 16:05:32.552927 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:32.552704 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv" Nov 01 16:05:32.558005 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:32.555694 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv"] Nov 01 16:05:32.564641 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:32.564186 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-383fde-serving-cert\"" Nov 01 16:05:32.631796 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:32.631756 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9235a8f-5d47-4464-a45f-6344c833c25a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv\" (UID: \"e9235a8f-5d47-4464-a45f-6344c833c25a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv" Nov 01 16:05:32.632142 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:32.632118 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r759j\" (UniqueName: \"kubernetes.io/projected/e9235a8f-5d47-4464-a45f-6344c833c25a-kube-api-access-r759j\") pod \"maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv\" (UID: \"e9235a8f-5d47-4464-a45f-6344c833c25a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv" Nov 01 16:05:32.632361 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:32.632344 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9235a8f-5d47-4464-a45f-6344c833c25a-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv\" (UID: \"e9235a8f-5d47-4464-a45f-6344c833c25a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv" Nov 01 16:05:32.632618 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:32.632600 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9235a8f-5d47-4464-a45f-6344c833c25a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv\" (UID: \"e9235a8f-5d47-4464-a45f-6344c833c25a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv" Nov 01 16:05:32.736132 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:32.734889 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r759j\" (UniqueName: \"kubernetes.io/projected/e9235a8f-5d47-4464-a45f-6344c833c25a-kube-api-access-r759j\") pod \"maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv\" (UID: \"e9235a8f-5d47-4464-a45f-6344c833c25a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv" Nov 01 16:05:32.736132 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:32.735002 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9235a8f-5d47-4464-a45f-6344c833c25a-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv\" (UID: \"e9235a8f-5d47-4464-a45f-6344c833c25a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv" Nov 01 16:05:32.736132 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:32.735163 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9235a8f-5d47-4464-a45f-6344c833c25a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv\" (UID: \"e9235a8f-5d47-4464-a45f-6344c833c25a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv" Nov 01 16:05:32.736132 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:32.735326 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9235a8f-5d47-4464-a45f-6344c833c25a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv\" (UID: \"e9235a8f-5d47-4464-a45f-6344c833c25a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv" Nov 01 16:05:32.736132 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:05:32.735469 4363 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-383fde-serving-cert: secret "maas-api-e2e-shared-a-w5-383fde-serving-cert" not found Nov 01 16:05:32.736132 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:05:32.735526 4363 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9235a8f-5d47-4464-a45f-6344c833c25a-maas-api-tls podName:e9235a8f-5d47-4464-a45f-6344c833c25a nodeName:}" failed. No retries permitted until 2026-11-01 16:05:33.23550662 +0000 UTC m=+1066.567244254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e9235a8f-5d47-4464-a45f-6344c833c25a-maas-api-tls") pod "maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv" (UID: "e9235a8f-5d47-4464-a45f-6344c833c25a") : secret "maas-api-e2e-shared-a-w5-383fde-serving-cert" not found Nov 01 16:05:32.737253 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:32.736678 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9235a8f-5d47-4464-a45f-6344c833c25a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv\" (UID: \"e9235a8f-5d47-4464-a45f-6344c833c25a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv" Nov 01 16:05:32.739237 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:32.738308 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5e2d03-cbd9bb6f9-bbprx" event={"ID":"96c20c16-b48c-40b5-853c-edddd1225652","Type":"ContainerStarted","Data":"5657a369b4632c5bc020f3a2169db572af33e2980861b959df88957d23a9f99d"} Nov 01 16:05:32.739237 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:32.738400 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5e2d03-cbd9bb6f9-bbprx" event={"ID":"96c20c16-b48c-40b5-853c-edddd1225652","Type":"ContainerStarted","Data":"7db87934577925cfa89c63f8165023624730a14d82f86d43aa15e62dc0b26d29"} Nov 01 16:05:32.739237 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:32.739186 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5e2d03-cbd9bb6f9-bbprx" Nov 01 16:05:32.742699 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:32.741581 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5e2d03-757556fddc-j9gph" event={"ID":"57dbe32d-78fb-4168-a5f8-5fa0a38f2e2b","Type":"ContainerStarted","Data":"44fdf0f6ebd17a0dd4f8dbc5a4b72bb1afe67bb9be6ce8ecd1f6f08822eaccf9"} Nov 01 16:05:32.742913 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:32.742889 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5e2d03-757556fddc-j9gph" event={"ID":"57dbe32d-78fb-4168-a5f8-5fa0a38f2e2b","Type":"ContainerStarted","Data":"7860d62f3148186e78814fd0206af3ecdda28b241dfd2e9f930fe83660c88f3e"} Nov 01 16:05:32.743680 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:32.743645 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5e2d03-757556fddc-j9gph" Nov 01 16:05:32.747445 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:32.747421 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9235a8f-5d47-4464-a45f-6344c833c25a-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv\" (UID: \"e9235a8f-5d47-4464-a45f-6344c833c25a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv" Nov 01 16:05:32.780503 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:32.780437 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5e2d03-cbd9bb6f9-bbprx" podStartSLOduration=1.780414897 podStartE2EDuration="1.780414897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:05:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:05:32.763484339 +0000 UTC m=+1066.095222017" watchObservedRunningTime="2026-11-01 16:05:32.780414897 +0000 UTC m=+1066.112152548" Nov 01 16:05:32.802441 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:32.802405 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r759j\" (UniqueName: \"kubernetes.io/projected/e9235a8f-5d47-4464-a45f-6344c833c25a-kube-api-access-r759j\") pod \"maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv\" (UID: \"e9235a8f-5d47-4464-a45f-6344c833c25a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv" Nov 01 16:05:33.254900 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:33.254849 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9235a8f-5d47-4464-a45f-6344c833c25a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv\" (UID: \"e9235a8f-5d47-4464-a45f-6344c833c25a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv" Nov 01 16:05:33.271817 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:33.271772 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9235a8f-5d47-4464-a45f-6344c833c25a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv\" (UID: \"e9235a8f-5d47-4464-a45f-6344c833c25a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv" Nov 01 16:05:33.489432 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:33.489375 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv" Nov 01 16:05:34.034839 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.034735 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5e2d03-757556fddc-j9gph" podStartSLOduration=3.034719366 podStartE2EDuration="3.034719366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:05:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:05:32.801371761 +0000 UTC m=+1066.133109413" watchObservedRunningTime="2026-11-01 16:05:34.034719366 +0000 UTC m=+1067.366457056" Nov 01 16:05:34.039424 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.037549 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv"] Nov 01 16:05:34.071950 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.071924 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54" Nov 01 16:05:34.116435 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.116403 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4504fd0-ac35-4f52-8403-beb7c42daa96-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4504fd0-ac35-4f52-8403-beb7c42daa96-workload-socket\") pod \"a4504fd0-ac35-4f52-8403-beb7c42daa96\" (UID: \"a4504fd0-ac35-4f52-8403-beb7c42daa96\") " Nov 01 16:05:34.116612 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.116472 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4504fd0-ac35-4f52-8403-beb7c42daa96-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4504fd0-ac35-4f52-8403-beb7c42daa96-istio-envoy\") pod \"a4504fd0-ac35-4f52-8403-beb7c42daa96\" (UID: \"a4504fd0-ac35-4f52-8403-beb7c42daa96\") " Nov 01 16:05:34.116612 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.116529 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4504fd0-ac35-4f52-8403-beb7c42daa96-kube-api-access-chrdj\" (UniqueName: \"kubernetes.io/projected/a4504fd0-ac35-4f52-8403-beb7c42daa96-kube-api-access-chrdj\") pod \"a4504fd0-ac35-4f52-8403-beb7c42daa96\" (UID: \"a4504fd0-ac35-4f52-8403-beb7c42daa96\") " Nov 01 16:05:34.116612 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.116576 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4504fd0-ac35-4f52-8403-beb7c42daa96-istio-token\" (UniqueName: \"kubernetes.io/projected/a4504fd0-ac35-4f52-8403-beb7c42daa96-istio-token\") pod \"a4504fd0-ac35-4f52-8403-beb7c42daa96\" (UID: \"a4504fd0-ac35-4f52-8403-beb7c42daa96\") " Nov 01 16:05:34.116783 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.116683 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4504fd0-ac35-4f52-8403-beb7c42daa96-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4504fd0-ac35-4f52-8403-beb7c42daa96-istio-data\") pod \"a4504fd0-ac35-4f52-8403-beb7c42daa96\" (UID: \"a4504fd0-ac35-4f52-8403-beb7c42daa96\") " Nov 01 16:05:34.116783 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.116747 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4504fd0-ac35-4f52-8403-beb7c42daa96-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4504fd0-ac35-4f52-8403-beb7c42daa96-istiod-ca-cert\") pod \"a4504fd0-ac35-4f52-8403-beb7c42daa96\" (UID: \"a4504fd0-ac35-4f52-8403-beb7c42daa96\") " Nov 01 16:05:34.116885 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.116843 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4504fd0-ac35-4f52-8403-beb7c42daa96-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4504fd0-ac35-4f52-8403-beb7c42daa96-istio-podinfo\") pod \"a4504fd0-ac35-4f52-8403-beb7c42daa96\" (UID: \"a4504fd0-ac35-4f52-8403-beb7c42daa96\") " Nov 01 16:05:34.116984 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.116969 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4504fd0-ac35-4f52-8403-beb7c42daa96-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4504fd0-ac35-4f52-8403-beb7c42daa96-credential-socket\") pod \"a4504fd0-ac35-4f52-8403-beb7c42daa96\" (UID: \"a4504fd0-ac35-4f52-8403-beb7c42daa96\") " Nov 01 16:05:34.117063 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.116997 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4504fd0-ac35-4f52-8403-beb7c42daa96-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4504fd0-ac35-4f52-8403-beb7c42daa96-workload-certs\") pod \"a4504fd0-ac35-4f52-8403-beb7c42daa96\" (UID: \"a4504fd0-ac35-4f52-8403-beb7c42daa96\") " Nov 01 16:05:34.118268 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.118238 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4504fd0-ac35-4f52-8403-beb7c42daa96-workload-socket" pod "a4504fd0-ac35-4f52-8403-beb7c42daa96" (UID: "a4504fd0-ac35-4f52-8403-beb7c42daa96"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:05:34.119235 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.118752 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4504fd0-ac35-4f52-8403-beb7c42daa96-istio-data" pod "a4504fd0-ac35-4f52-8403-beb7c42daa96" (UID: "a4504fd0-ac35-4f52-8403-beb7c42daa96"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:05:34.119858 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.119797 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4504fd0-ac35-4f52-8403-beb7c42daa96-istiod-ca-cert" pod "a4504fd0-ac35-4f52-8403-beb7c42daa96" (UID: "a4504fd0-ac35-4f52-8403-beb7c42daa96"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:05:34.119858 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.119797 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4504fd0-ac35-4f52-8403-beb7c42daa96-workload-certs" pod "a4504fd0-ac35-4f52-8403-beb7c42daa96" (UID: "a4504fd0-ac35-4f52-8403-beb7c42daa96"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:05:34.120259 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.120236 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4504fd0-ac35-4f52-8403-beb7c42daa96-credential-socket" pod "a4504fd0-ac35-4f52-8403-beb7c42daa96" (UID: "a4504fd0-ac35-4f52-8403-beb7c42daa96"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:05:34.131897 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.131856 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4504fd0-ac35-4f52-8403-beb7c42daa96-istio-envoy" pod "a4504fd0-ac35-4f52-8403-beb7c42daa96" (UID: "a4504fd0-ac35-4f52-8403-beb7c42daa96"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:05:34.131998 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.131899 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4504fd0-ac35-4f52-8403-beb7c42daa96-kube-api-access-chrdj" pod "a4504fd0-ac35-4f52-8403-beb7c42daa96" (UID: "a4504fd0-ac35-4f52-8403-beb7c42daa96"). InnerVolumeSpecName "kube-api-access-chrdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:05:34.132220 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.132194 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4504fd0-ac35-4f52-8403-beb7c42daa96-istio-token" pod "a4504fd0-ac35-4f52-8403-beb7c42daa96" (UID: "a4504fd0-ac35-4f52-8403-beb7c42daa96"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:05:34.133571 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.133538 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4504fd0-ac35-4f52-8403-beb7c42daa96-istio-podinfo" pod "a4504fd0-ac35-4f52-8403-beb7c42daa96" (UID: "a4504fd0-ac35-4f52-8403-beb7c42daa96"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:05:34.263101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.243797 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4504fd0-ac35-4f52-8403-beb7c42daa96-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:34.263101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.243837 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4504fd0-ac35-4f52-8403-beb7c42daa96-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:34.263101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.243853 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4504fd0-ac35-4f52-8403-beb7c42daa96-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:34.263101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.243869 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4504fd0-ac35-4f52-8403-beb7c42daa96-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:34.263101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.243884 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4504fd0-ac35-4f52-8403-beb7c42daa96-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:34.263101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.243897 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chrdj\" (UniqueName: \"kubernetes.io/projected/a4504fd0-ac35-4f52-8403-beb7c42daa96-kube-api-access-chrdj\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:34.263101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.243911 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4504fd0-ac35-4f52-8403-beb7c42daa96-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:34.263101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.243925 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4504fd0-ac35-4f52-8403-beb7c42daa96-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:34.263101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.243936 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4504fd0-ac35-4f52-8403-beb7c42daa96-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:34.439483 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.439456 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9" Nov 01 16:05:34.554989 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.554886 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86b37143-20ce-45d3-90d7-45b451012c14-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86b37143-20ce-45d3-90d7-45b451012c14-istio-envoy\") pod \"86b37143-20ce-45d3-90d7-45b451012c14\" (UID: \"86b37143-20ce-45d3-90d7-45b451012c14\") " Nov 01 16:05:34.555130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.555001 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86b37143-20ce-45d3-90d7-45b451012c14-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b37143-20ce-45d3-90d7-45b451012c14-workload-socket\") pod \"86b37143-20ce-45d3-90d7-45b451012c14\" (UID: \"86b37143-20ce-45d3-90d7-45b451012c14\") " Nov 01 16:05:34.555130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.555073 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86b37143-20ce-45d3-90d7-45b451012c14-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86b37143-20ce-45d3-90d7-45b451012c14-istio-data\") pod \"86b37143-20ce-45d3-90d7-45b451012c14\" (UID: \"86b37143-20ce-45d3-90d7-45b451012c14\") " Nov 01 16:05:34.555130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.555122 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86b37143-20ce-45d3-90d7-45b451012c14-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86b37143-20ce-45d3-90d7-45b451012c14-istiod-ca-cert\") pod \"86b37143-20ce-45d3-90d7-45b451012c14\" (UID: \"86b37143-20ce-45d3-90d7-45b451012c14\") " Nov 01 16:05:34.555395 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.555146 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86b37143-20ce-45d3-90d7-45b451012c14-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86b37143-20ce-45d3-90d7-45b451012c14-istio-podinfo\") pod \"86b37143-20ce-45d3-90d7-45b451012c14\" (UID: \"86b37143-20ce-45d3-90d7-45b451012c14\") " Nov 01 16:05:34.555395 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.555182 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86b37143-20ce-45d3-90d7-45b451012c14-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86b37143-20ce-45d3-90d7-45b451012c14-workload-certs\") pod \"86b37143-20ce-45d3-90d7-45b451012c14\" (UID: \"86b37143-20ce-45d3-90d7-45b451012c14\") " Nov 01 16:05:34.555395 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.555293 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b37143-20ce-45d3-90d7-45b451012c14-workload-socket" pod "86b37143-20ce-45d3-90d7-45b451012c14" (UID: "86b37143-20ce-45d3-90d7-45b451012c14"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:05:34.555395 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.555348 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86b37143-20ce-45d3-90d7-45b451012c14-istio-token\" (UniqueName: \"kubernetes.io/projected/86b37143-20ce-45d3-90d7-45b451012c14-istio-token\") pod \"86b37143-20ce-45d3-90d7-45b451012c14\" (UID: \"86b37143-20ce-45d3-90d7-45b451012c14\") " Nov 01 16:05:34.555695 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.555465 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86b37143-20ce-45d3-90d7-45b451012c14-kube-api-access-5n6x4\" (UniqueName: \"kubernetes.io/projected/86b37143-20ce-45d3-90d7-45b451012c14-kube-api-access-5n6x4\") pod \"86b37143-20ce-45d3-90d7-45b451012c14\" (UID: \"86b37143-20ce-45d3-90d7-45b451012c14\") " Nov 01 16:05:34.555695 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.555492 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b37143-20ce-45d3-90d7-45b451012c14-istio-data" pod "86b37143-20ce-45d3-90d7-45b451012c14" (UID: "86b37143-20ce-45d3-90d7-45b451012c14"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:05:34.555695 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.555519 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86b37143-20ce-45d3-90d7-45b451012c14-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b37143-20ce-45d3-90d7-45b451012c14-credential-socket\") pod \"86b37143-20ce-45d3-90d7-45b451012c14\" (UID: \"86b37143-20ce-45d3-90d7-45b451012c14\") " Nov 01 16:05:34.555695 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.555640 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b37143-20ce-45d3-90d7-45b451012c14-workload-certs" pod "86b37143-20ce-45d3-90d7-45b451012c14" (UID: "86b37143-20ce-45d3-90d7-45b451012c14"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:05:34.555695 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.555667 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86b37143-20ce-45d3-90d7-45b451012c14-istiod-ca-cert" pod "86b37143-20ce-45d3-90d7-45b451012c14" (UID: "86b37143-20ce-45d3-90d7-45b451012c14"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:05:34.556002 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.555887 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b37143-20ce-45d3-90d7-45b451012c14-credential-socket" pod "86b37143-20ce-45d3-90d7-45b451012c14" (UID: "86b37143-20ce-45d3-90d7-45b451012c14"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:05:34.556418 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.556386 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b37143-20ce-45d3-90d7-45b451012c14-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:34.556418 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.556410 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b37143-20ce-45d3-90d7-45b451012c14-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:34.556543 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.556427 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86b37143-20ce-45d3-90d7-45b451012c14-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:34.556543 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.556442 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86b37143-20ce-45d3-90d7-45b451012c14-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:34.556543 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.556456 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86b37143-20ce-45d3-90d7-45b451012c14-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:34.557702 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.557666 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b37143-20ce-45d3-90d7-45b451012c14-istio-envoy" pod "86b37143-20ce-45d3-90d7-45b451012c14" (UID: "86b37143-20ce-45d3-90d7-45b451012c14"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:05:34.558159 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.558122 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86b37143-20ce-45d3-90d7-45b451012c14-kube-api-access-5n6x4" pod "86b37143-20ce-45d3-90d7-45b451012c14" (UID: "86b37143-20ce-45d3-90d7-45b451012c14"). InnerVolumeSpecName "kube-api-access-5n6x4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:05:34.558505 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.558485 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86b37143-20ce-45d3-90d7-45b451012c14-istio-token" pod "86b37143-20ce-45d3-90d7-45b451012c14" (UID: "86b37143-20ce-45d3-90d7-45b451012c14"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:05:34.558767 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.558748 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86b37143-20ce-45d3-90d7-45b451012c14-istio-podinfo" pod "86b37143-20ce-45d3-90d7-45b451012c14" (UID: "86b37143-20ce-45d3-90d7-45b451012c14"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:05:34.657999 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.657971 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86b37143-20ce-45d3-90d7-45b451012c14-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:34.657999 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.658000 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n6x4\" (UniqueName: \"kubernetes.io/projected/86b37143-20ce-45d3-90d7-45b451012c14-kube-api-access-5n6x4\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:34.658166 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.658012 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86b37143-20ce-45d3-90d7-45b451012c14-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:34.658166 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.658060 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86b37143-20ce-45d3-90d7-45b451012c14-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:34.799809 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.799716 4363 generic.go:356] "Generic (PLEG): container finished" podID="86b37143-20ce-45d3-90d7-45b451012c14" containerID="c34de563140d0a4a76025d99491d588ee2ae1f61c1a504e992e9a67b10e71baf" exitCode=0 Nov 01 16:05:34.799987 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.799827 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9" Nov 01 16:05:34.809902 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.809412 4363 generic.go:356] "Generic (PLEG): container finished" podID="a4504fd0-ac35-4f52-8403-beb7c42daa96" containerID="9a427625ec052056690996064900fc5e0eff4838367e779abb16c2fd23883b40" exitCode=0 Nov 01 16:05:34.810271 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.810223 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54" Nov 01 16:05:34.810444 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.810393 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv" Nov 01 16:05:34.810444 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.810435 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9" event={"ID":"86b37143-20ce-45d3-90d7-45b451012c14","Type":"ContainerDied","Data":"c34de563140d0a4a76025d99491d588ee2ae1f61c1a504e992e9a67b10e71baf"} Nov 01 16:05:34.810569 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.810472 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9" event={"ID":"86b37143-20ce-45d3-90d7-45b451012c14","Type":"ContainerDied","Data":"a911f0e4ff312b96c99ad3246f33d8082a5738e22eec9068aeb36282a835bc5b"} Nov 01 16:05:34.810569 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.810497 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv" event={"ID":"e9235a8f-5d47-4464-a45f-6344c833c25a","Type":"ContainerStarted","Data":"b32697768fa5b9f216d1ad2a33127efb0fd6ff1dce0870459cb6135e7284c70c"} Nov 01 16:05:34.810569 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.810517 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv" event={"ID":"e9235a8f-5d47-4464-a45f-6344c833c25a","Type":"ContainerStarted","Data":"1e2f9e2c0d21ecfa3f8e31c6088ca6b3b4e77db63695badfb20b95d3f68c8bc1"} Nov 01 16:05:34.810569 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.810536 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54" event={"ID":"a4504fd0-ac35-4f52-8403-beb7c42daa96","Type":"ContainerDied","Data":"9a427625ec052056690996064900fc5e0eff4838367e779abb16c2fd23883b40"} Nov 01 16:05:34.810569 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.810562 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54" event={"ID":"a4504fd0-ac35-4f52-8403-beb7c42daa96","Type":"ContainerDied","Data":"6a768033148c0b8460589a7b3000ca9f7d8e4109f7d53e8e25a7fbfbe76396fb"} Nov 01 16:05:34.810780 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.810592 4363 scope.go:122] "RemoveContainer" containerID="c34de563140d0a4a76025d99491d588ee2ae1f61c1a504e992e9a67b10e71baf" Nov 01 16:05:34.831414 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.831343 4363 scope.go:122] "RemoveContainer" containerID="c34de563140d0a4a76025d99491d588ee2ae1f61c1a504e992e9a67b10e71baf" Nov 01 16:05:34.833331 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:05:34.833309 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c34de563140d0a4a76025d99491d588ee2ae1f61c1a504e992e9a67b10e71baf\": container with ID starting with c34de563140d0a4a76025d99491d588ee2ae1f61c1a504e992e9a67b10e71baf not found: ID does not exist" containerID="c34de563140d0a4a76025d99491d588ee2ae1f61c1a504e992e9a67b10e71baf" Nov 01 16:05:34.833454 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.833339 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c34de563140d0a4a76025d99491d588ee2ae1f61c1a504e992e9a67b10e71baf"} err="failed to get container status \"c34de563140d0a4a76025d99491d588ee2ae1f61c1a504e992e9a67b10e71baf\": rpc error: code = NotFound desc = could not find container \"c34de563140d0a4a76025d99491d588ee2ae1f61c1a504e992e9a67b10e71baf\": container with ID starting with c34de563140d0a4a76025d99491d588ee2ae1f61c1a504e992e9a67b10e71baf not found: ID does not exist" Nov 01 16:05:34.833454 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.833357 4363 scope.go:122] "RemoveContainer" containerID="9a427625ec052056690996064900fc5e0eff4838367e779abb16c2fd23883b40" Nov 01 16:05:34.834090 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.834054 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv" podStartSLOduration=2.358770553 podStartE2EDuration="2.834037856s" totalImagesPullingTime="475.267303ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:05:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:05:34.830187389 +0000 UTC m=+1068.161925043" watchObservedRunningTime="2026-11-01 16:05:34.834037856 +0000 UTC m=+1068.165775508" Nov 01 16:05:34.845903 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.845878 4363 scope.go:122] "RemoveContainer" containerID="9a427625ec052056690996064900fc5e0eff4838367e779abb16c2fd23883b40" Nov 01 16:05:34.846261 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:05:34.846240 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a427625ec052056690996064900fc5e0eff4838367e779abb16c2fd23883b40\": container with ID starting with 9a427625ec052056690996064900fc5e0eff4838367e779abb16c2fd23883b40 not found: ID does not exist" containerID="9a427625ec052056690996064900fc5e0eff4838367e779abb16c2fd23883b40" Nov 01 16:05:34.846339 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.846268 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a427625ec052056690996064900fc5e0eff4838367e779abb16c2fd23883b40"} err="failed to get container status \"9a427625ec052056690996064900fc5e0eff4838367e779abb16c2fd23883b40\": rpc error: code = NotFound desc = could not find container \"9a427625ec052056690996064900fc5e0eff4838367e779abb16c2fd23883b40\": container with ID starting with 9a427625ec052056690996064900fc5e0eff4838367e779abb16c2fd23883b40 not found: ID does not exist" Nov 01 16:05:34.853262 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.853240 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54"] Nov 01 16:05:34.882345 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.882314 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-474cab5e-openshift-default-5c5fff6655-dpt54"] Nov 01 16:05:34.885547 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.885516 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9"] Nov 01 16:05:34.896084 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:34.896047 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-9fd01d20-openshift-default-744446d849-cstp9"] Nov 01 16:05:36.803832 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:36.803787 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86b37143-20ce-45d3-90d7-45b451012c14" path="/var/lib/kubelet/pods/86b37143-20ce-45d3-90d7-45b451012c14/volumes" Nov 01 16:05:36.804687 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:36.804664 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4504fd0-ac35-4f52-8403-beb7c42daa96" path="/var/lib/kubelet/pods/a4504fd0-ac35-4f52-8403-beb7c42daa96/volumes" Nov 01 16:05:37.040189 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:37.040159 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5e2d03-86f8cf979c-xd2rs"] Nov 01 16:05:37.040897 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:37.040883 4363 cpu_manager.go:414] "containerMap: removing container" podUID="86b37143-20ce-45d3-90d7-45b451012c14" containerName="istio-proxy" Nov 01 16:05:37.040946 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:37.040899 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86b37143-20ce-45d3-90d7-45b451012c14" containerName="istio-proxy" Nov 01 16:05:37.040946 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:37.040941 4363 cpu_manager.go:414] "containerMap: removing container" podUID="a4504fd0-ac35-4f52-8403-beb7c42daa96" containerName="istio-proxy" Nov 01 16:05:37.041010 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:37.040947 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4504fd0-ac35-4f52-8403-beb7c42daa96" containerName="istio-proxy" Nov 01 16:05:37.041068 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:37.041053 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4504fd0-ac35-4f52-8403-beb7c42daa96" containerName="istio-proxy" Nov 01 16:05:37.041068 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:37.041068 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="86b37143-20ce-45d3-90d7-45b451012c14" containerName="istio-proxy" Nov 01 16:05:37.044070 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:37.044050 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5e2d03-86f8cf979c-xd2rs" Nov 01 16:05:37.053164 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:37.053140 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-5e2d03-serving-cert\"" Nov 01 16:05:37.056712 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:37.056652 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5e2d03-86f8cf979c-xd2rs"] Nov 01 16:05:37.135890 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:37.135852 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-5e2d03-86f8cf979c-xd2rs\" (UID: \"0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5e2d03-86f8cf979c-xd2rs" Nov 01 16:05:37.135890 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:37.135890 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-5e2d03-86f8cf979c-xd2rs\" (UID: \"0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5e2d03-86f8cf979c-xd2rs" Nov 01 16:05:37.136133 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:37.136099 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-5e2d03-86f8cf979c-xd2rs\" (UID: \"0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5e2d03-86f8cf979c-xd2rs" Nov 01 16:05:37.136182 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:37.136164 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmfr6\" (UniqueName: \"kubernetes.io/projected/0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18-kube-api-access-xmfr6\") pod \"maas-api-e2e-shared-a-w0-5e2d03-86f8cf979c-xd2rs\" (UID: \"0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5e2d03-86f8cf979c-xd2rs" Nov 01 16:05:37.303048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:37.279611 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-5e2d03-86f8cf979c-xd2rs\" (UID: \"0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5e2d03-86f8cf979c-xd2rs" Nov 01 16:05:37.303048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:37.302317 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-5e2d03-86f8cf979c-xd2rs\" (UID: \"0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5e2d03-86f8cf979c-xd2rs" Nov 01 16:05:37.303048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:37.302624 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-5e2d03-86f8cf979c-xd2rs\" (UID: \"0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5e2d03-86f8cf979c-xd2rs" Nov 01 16:05:37.303048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:37.302709 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmfr6\" (UniqueName: \"kubernetes.io/projected/0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18-kube-api-access-xmfr6\") pod \"maas-api-e2e-shared-a-w0-5e2d03-86f8cf979c-xd2rs\" (UID: \"0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5e2d03-86f8cf979c-xd2rs" Nov 01 16:05:37.303397 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:37.299055 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-5e2d03-86f8cf979c-xd2rs\" (UID: \"0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5e2d03-86f8cf979c-xd2rs" Nov 01 16:05:37.355095 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:37.335844 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmfr6\" (UniqueName: \"kubernetes.io/projected/0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18-kube-api-access-xmfr6\") pod \"maas-api-e2e-shared-a-w0-5e2d03-86f8cf979c-xd2rs\" (UID: \"0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5e2d03-86f8cf979c-xd2rs" Nov 01 16:05:37.356043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:37.355999 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-5e2d03-86f8cf979c-xd2rs\" (UID: \"0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5e2d03-86f8cf979c-xd2rs" Nov 01 16:05:37.370126 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:37.360287 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-5e2d03-86f8cf979c-xd2rs\" (UID: \"0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5e2d03-86f8cf979c-xd2rs" Nov 01 16:05:37.377694 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:37.377443 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5e2d03-86f8cf979c-xd2rs" Nov 01 16:05:37.835041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:37.808387 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5e2d03-86f8cf979c-xd2rs"] Nov 01 16:05:37.990041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:37.989836 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5e2d03-86f8cf979c-xd2rs" event={"ID":"0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18","Type":"ContainerStarted","Data":"19acf69af6482201b16ac4de8a634f70646839fa91db5790849b58dad484682b"} Nov 01 16:05:39.000743 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:39.000703 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5e2d03-86f8cf979c-xd2rs" event={"ID":"0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18","Type":"ContainerStarted","Data":"b23b7d28d16d49e60b83cb8498611689b32f6c1190f9fbe0a85ae63c67d15945"} Nov 01 16:05:39.001283 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:39.000936 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5e2d03-86f8cf979c-xd2rs" Nov 01 16:05:39.021267 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:39.021218 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5e2d03-86f8cf979c-xd2rs" podStartSLOduration=1.526822454 podStartE2EDuration="2.021203691s" totalImagesPullingTime="494.381237ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:05:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:05:39.016414025 +0000 UTC m=+1072.348151681" watchObservedRunningTime="2026-11-01 16:05:39.021203691 +0000 UTC m=+1072.352941340" Nov 01 16:05:39.609215 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:39.609186 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-383fde-6f8cc9c655-m67lx" Nov 01 16:05:40.680054 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:40.677296 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-383fde-8486d66857-b9mwl" Nov 01 16:05:40.860687 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:40.860661 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv" Nov 01 16:05:44.813724 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:44.813696 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5e2d03-cbd9bb6f9-bbprx" Nov 01 16:05:44.819321 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:44.819296 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5e2d03-757556fddc-j9gph" Nov 01 16:05:45.031762 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:45.031736 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5e2d03-86f8cf979c-xd2rs" Nov 01 16:05:46.239676 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:46.236297 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m"] Nov 01 16:05:46.244824 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:46.244797 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" Nov 01 16:05:46.252541 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:46.249941 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m"] Nov 01 16:05:46.252541 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:46.252461 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-a9a967-openshift-default-dockercfg-8bwnt\"" Nov 01 16:05:46.414042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:46.402214 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/420774ff-0dfc-44ef-a9c3-19a41f76a985-istio-envoy\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m\" (UID: \"420774ff-0dfc-44ef-a9c3-19a41f76a985\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" Nov 01 16:05:46.428044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:46.414311 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/420774ff-0dfc-44ef-a9c3-19a41f76a985-istiod-ca-cert\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m\" (UID: \"420774ff-0dfc-44ef-a9c3-19a41f76a985\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" Nov 01 16:05:46.428044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:46.424235 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/420774ff-0dfc-44ef-a9c3-19a41f76a985-workload-certs\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m\" (UID: \"420774ff-0dfc-44ef-a9c3-19a41f76a985\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" Nov 01 16:05:46.428044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:46.424287 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/420774ff-0dfc-44ef-a9c3-19a41f76a985-istio-podinfo\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m\" (UID: \"420774ff-0dfc-44ef-a9c3-19a41f76a985\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" Nov 01 16:05:46.428044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:46.424359 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/420774ff-0dfc-44ef-a9c3-19a41f76a985-istio-data\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m\" (UID: \"420774ff-0dfc-44ef-a9c3-19a41f76a985\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" Nov 01 16:05:46.479038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:46.473513 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/420774ff-0dfc-44ef-a9c3-19a41f76a985-istio-token\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m\" (UID: \"420774ff-0dfc-44ef-a9c3-19a41f76a985\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" Nov 01 16:05:46.479038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:46.473611 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjqgv\" (UniqueName: \"kubernetes.io/projected/420774ff-0dfc-44ef-a9c3-19a41f76a985-kube-api-access-fjqgv\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m\" (UID: \"420774ff-0dfc-44ef-a9c3-19a41f76a985\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" Nov 01 16:05:46.479038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:46.473717 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/420774ff-0dfc-44ef-a9c3-19a41f76a985-credential-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m\" (UID: \"420774ff-0dfc-44ef-a9c3-19a41f76a985\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" Nov 01 16:05:46.479038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:46.473860 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/420774ff-0dfc-44ef-a9c3-19a41f76a985-workload-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m\" (UID: \"420774ff-0dfc-44ef-a9c3-19a41f76a985\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" Nov 01 16:05:46.597092 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:46.595371 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/420774ff-0dfc-44ef-a9c3-19a41f76a985-credential-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m\" (UID: \"420774ff-0dfc-44ef-a9c3-19a41f76a985\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" Nov 01 16:05:46.597092 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:46.595497 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/420774ff-0dfc-44ef-a9c3-19a41f76a985-workload-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m\" (UID: \"420774ff-0dfc-44ef-a9c3-19a41f76a985\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" Nov 01 16:05:46.597092 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:46.595657 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/420774ff-0dfc-44ef-a9c3-19a41f76a985-istio-envoy\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m\" (UID: \"420774ff-0dfc-44ef-a9c3-19a41f76a985\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" Nov 01 16:05:46.597092 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:46.595686 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/420774ff-0dfc-44ef-a9c3-19a41f76a985-istiod-ca-cert\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m\" (UID: \"420774ff-0dfc-44ef-a9c3-19a41f76a985\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" Nov 01 16:05:46.597092 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:46.595898 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/420774ff-0dfc-44ef-a9c3-19a41f76a985-workload-certs\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m\" (UID: \"420774ff-0dfc-44ef-a9c3-19a41f76a985\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" Nov 01 16:05:46.597092 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:46.595929 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/420774ff-0dfc-44ef-a9c3-19a41f76a985-istio-podinfo\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m\" (UID: \"420774ff-0dfc-44ef-a9c3-19a41f76a985\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" Nov 01 16:05:46.597092 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:46.595995 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/420774ff-0dfc-44ef-a9c3-19a41f76a985-istio-data\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m\" (UID: \"420774ff-0dfc-44ef-a9c3-19a41f76a985\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" Nov 01 16:05:46.597092 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:46.596213 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/420774ff-0dfc-44ef-a9c3-19a41f76a985-istio-token\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m\" (UID: \"420774ff-0dfc-44ef-a9c3-19a41f76a985\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" Nov 01 16:05:46.597092 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:46.596372 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjqgv\" (UniqueName: \"kubernetes.io/projected/420774ff-0dfc-44ef-a9c3-19a41f76a985-kube-api-access-fjqgv\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m\" (UID: \"420774ff-0dfc-44ef-a9c3-19a41f76a985\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" Nov 01 16:05:46.600165 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:46.600137 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/420774ff-0dfc-44ef-a9c3-19a41f76a985-istiod-ca-cert\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m\" (UID: \"420774ff-0dfc-44ef-a9c3-19a41f76a985\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" Nov 01 16:05:46.603039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:46.602673 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/420774ff-0dfc-44ef-a9c3-19a41f76a985-istio-data\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m\" (UID: \"420774ff-0dfc-44ef-a9c3-19a41f76a985\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" Nov 01 16:05:46.603182 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:46.603109 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/420774ff-0dfc-44ef-a9c3-19a41f76a985-credential-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m\" (UID: \"420774ff-0dfc-44ef-a9c3-19a41f76a985\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" Nov 01 16:05:46.606530 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:46.604315 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/420774ff-0dfc-44ef-a9c3-19a41f76a985-workload-certs\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m\" (UID: \"420774ff-0dfc-44ef-a9c3-19a41f76a985\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" Nov 01 16:05:46.607328 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:46.607302 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/420774ff-0dfc-44ef-a9c3-19a41f76a985-workload-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m\" (UID: \"420774ff-0dfc-44ef-a9c3-19a41f76a985\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" Nov 01 16:05:46.655457 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:46.655419 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/420774ff-0dfc-44ef-a9c3-19a41f76a985-istio-envoy\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m\" (UID: \"420774ff-0dfc-44ef-a9c3-19a41f76a985\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" Nov 01 16:05:46.662790 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:46.657140 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/420774ff-0dfc-44ef-a9c3-19a41f76a985-istio-podinfo\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m\" (UID: \"420774ff-0dfc-44ef-a9c3-19a41f76a985\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" Nov 01 16:05:46.681708 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:46.678691 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/420774ff-0dfc-44ef-a9c3-19a41f76a985-istio-token\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m\" (UID: \"420774ff-0dfc-44ef-a9c3-19a41f76a985\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" Nov 01 16:05:46.681708 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:46.679378 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjqgv\" (UniqueName: \"kubernetes.io/projected/420774ff-0dfc-44ef-a9c3-19a41f76a985-kube-api-access-fjqgv\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m\" (UID: \"420774ff-0dfc-44ef-a9c3-19a41f76a985\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" Nov 01 16:05:46.867457 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:46.867330 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" Nov 01 16:05:47.683886 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:47.683856 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m"] Nov 01 16:05:47.686231 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:05:47.686191 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420774ff_0dfc_44ef_a9c3_19a41f76a985.slice/crio-afb3d68a77c7d155a871f29ac9a0d4bbc60b3ac92f18e103f033145c2364d02a WatchSource:0}: Error finding container afb3d68a77c7d155a871f29ac9a0d4bbc60b3ac92f18e103f033145c2364d02a: Status 404 returned error can't find the container with id afb3d68a77c7d155a871f29ac9a0d4bbc60b3ac92f18e103f033145c2364d02a Nov 01 16:05:47.689930 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:47.689890 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:05:47.690043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:47.689981 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:05:47.690090 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:47.690045 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:05:48.300690 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:48.300654 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" event={"ID":"420774ff-0dfc-44ef-a9c3-19a41f76a985","Type":"ContainerStarted","Data":"071c82dff01760262ca73b5e4ec7256daf0a0d77811695e658ab57db7052b1da"} Nov 01 16:05:48.300799 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:48.300723 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" event={"ID":"420774ff-0dfc-44ef-a9c3-19a41f76a985","Type":"ContainerStarted","Data":"afb3d68a77c7d155a871f29ac9a0d4bbc60b3ac92f18e103f033145c2364d02a"} Nov 01 16:05:48.326411 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:05:48.326382 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:05:48.355875 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:48.355829 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" podStartSLOduration=2.355815853 podStartE2EDuration="2.355815853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:05:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:05:48.350716586 +0000 UTC m=+1081.682454237" watchObservedRunningTime="2026-11-01 16:05:48.355815853 +0000 UTC m=+1081.687553503" Nov 01 16:05:48.869458 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:48.869429 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" Nov 01 16:05:48.876081 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:48.876054 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" Nov 01 16:05:49.309766 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:49.309679 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" Nov 01 16:05:49.310974 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:49.310952 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" Nov 01 16:05:52.022975 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:52.022940 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a9a967-6948b69c67-tgrtj"] Nov 01 16:05:52.027605 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:52.027583 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a9a967-6948b69c67-tgrtj" Nov 01 16:05:52.034855 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:52.034830 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-a9a967\"" Nov 01 16:05:52.036815 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:52.036789 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a9a967-6948b69c67-tgrtj"] Nov 01 16:05:52.166061 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:52.162958 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwdhg\" (UniqueName: \"kubernetes.io/projected/de4bccaf-0cc6-4d9a-8b5c-800217cff553-kube-api-access-mwdhg\") pod \"payload-pre-processing-e2e-shared-b-w5-a9a967-6948b69c67-tgrtj\" (UID: \"de4bccaf-0cc6-4d9a-8b5c-800217cff553\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a9a967-6948b69c67-tgrtj" Nov 01 16:05:52.166061 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:52.163344 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de4bccaf-0cc6-4d9a-8b5c-800217cff553-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-a9a967-6948b69c67-tgrtj\" (UID: \"de4bccaf-0cc6-4d9a-8b5c-800217cff553\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a9a967-6948b69c67-tgrtj" Nov 01 16:05:52.276792 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:52.274818 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de4bccaf-0cc6-4d9a-8b5c-800217cff553-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-a9a967-6948b69c67-tgrtj\" (UID: \"de4bccaf-0cc6-4d9a-8b5c-800217cff553\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a9a967-6948b69c67-tgrtj" Nov 01 16:05:52.276792 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:52.274995 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwdhg\" (UniqueName: \"kubernetes.io/projected/de4bccaf-0cc6-4d9a-8b5c-800217cff553-kube-api-access-mwdhg\") pod \"payload-pre-processing-e2e-shared-b-w5-a9a967-6948b69c67-tgrtj\" (UID: \"de4bccaf-0cc6-4d9a-8b5c-800217cff553\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a9a967-6948b69c67-tgrtj" Nov 01 16:05:52.276792 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:52.276673 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de4bccaf-0cc6-4d9a-8b5c-800217cff553-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-a9a967-6948b69c67-tgrtj\" (UID: \"de4bccaf-0cc6-4d9a-8b5c-800217cff553\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a9a967-6948b69c67-tgrtj" Nov 01 16:05:52.328048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:52.326732 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwdhg\" (UniqueName: \"kubernetes.io/projected/de4bccaf-0cc6-4d9a-8b5c-800217cff553-kube-api-access-mwdhg\") pod \"payload-pre-processing-e2e-shared-b-w5-a9a967-6948b69c67-tgrtj\" (UID: \"de4bccaf-0cc6-4d9a-8b5c-800217cff553\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a9a967-6948b69c67-tgrtj" Nov 01 16:05:52.361893 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:52.361862 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a9a967-6948b69c67-tgrtj" Nov 01 16:05:52.962083 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:52.959905 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a9a967-6948b69c67-tgrtj"] Nov 01 16:05:53.011043 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:05:53.002361 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde4bccaf_0cc6_4d9a_8b5c_800217cff553.slice/crio-e768adca3aeb7806969a46b8c5d26cd275ef5116406a68900bd5b1ec034df435 WatchSource:0}: Error finding container e768adca3aeb7806969a46b8c5d26cd275ef5116406a68900bd5b1ec034df435: Status 404 returned error can't find the container with id e768adca3aeb7806969a46b8c5d26cd275ef5116406a68900bd5b1ec034df435 Nov 01 16:05:53.486732 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:53.486698 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a9a967-6948b69c67-tgrtj" event={"ID":"de4bccaf-0cc6-4d9a-8b5c-800217cff553","Type":"ContainerStarted","Data":"646cf3813c38dae5d27f6ea50c5de77e494d6e91c9e92e741f0d679dd66a43fb"} Nov 01 16:05:53.487169 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:53.486754 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a9a967-6948b69c67-tgrtj" event={"ID":"de4bccaf-0cc6-4d9a-8b5c-800217cff553","Type":"ContainerStarted","Data":"e768adca3aeb7806969a46b8c5d26cd275ef5116406a68900bd5b1ec034df435"} Nov 01 16:05:53.487169 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:53.486776 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a9a967-6948b69c67-tgrtj" Nov 01 16:05:53.507332 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:53.507272 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a9a967-6948b69c67-tgrtj" podStartSLOduration=2.50725378 podStartE2EDuration="2.50725378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:05:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:05:53.500550759 +0000 UTC m=+1086.832288409" watchObservedRunningTime="2026-11-01 16:05:53.50725378 +0000 UTC m=+1086.838991430" Nov 01 16:05:56.229309 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:56.229273 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d121-5595856559-g7tk9"] Nov 01 16:05:56.235492 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:56.235464 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d121-5595856559-g7tk9" Nov 01 16:05:56.242782 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:56.242741 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-c8d121-dockercfg-kl924\"" Nov 01 16:05:56.248875 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:56.248847 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d121-5595856559-g7tk9"] Nov 01 16:05:56.262173 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:56.262151 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-c8d121\"" Nov 01 16:05:56.266370 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:56.266344 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k46kd\" (UniqueName: \"kubernetes.io/projected/55c3d81b-79de-44cb-9838-8b07da7c911b-kube-api-access-k46kd\") pod \"payload-pre-processing-e2e-shared-b-w0-c8d121-5595856559-g7tk9\" (UID: \"55c3d81b-79de-44cb-9838-8b07da7c911b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d121-5595856559-g7tk9" Nov 01 16:05:56.266613 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:56.266585 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55c3d81b-79de-44cb-9838-8b07da7c911b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-c8d121-5595856559-g7tk9\" (UID: \"55c3d81b-79de-44cb-9838-8b07da7c911b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d121-5595856559-g7tk9" Nov 01 16:05:56.425038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:56.423757 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55c3d81b-79de-44cb-9838-8b07da7c911b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-c8d121-5595856559-g7tk9\" (UID: \"55c3d81b-79de-44cb-9838-8b07da7c911b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d121-5595856559-g7tk9" Nov 01 16:05:56.425038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:56.424128 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k46kd\" (UniqueName: \"kubernetes.io/projected/55c3d81b-79de-44cb-9838-8b07da7c911b-kube-api-access-k46kd\") pod \"payload-pre-processing-e2e-shared-b-w0-c8d121-5595856559-g7tk9\" (UID: \"55c3d81b-79de-44cb-9838-8b07da7c911b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d121-5595856559-g7tk9" Nov 01 16:05:56.460183 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:56.425444 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55c3d81b-79de-44cb-9838-8b07da7c911b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-c8d121-5595856559-g7tk9\" (UID: \"55c3d81b-79de-44cb-9838-8b07da7c911b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d121-5595856559-g7tk9" Nov 01 16:05:56.484100 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:56.468372 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k46kd\" (UniqueName: \"kubernetes.io/projected/55c3d81b-79de-44cb-9838-8b07da7c911b-kube-api-access-k46kd\") pod \"payload-pre-processing-e2e-shared-b-w0-c8d121-5595856559-g7tk9\" (UID: \"55c3d81b-79de-44cb-9838-8b07da7c911b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d121-5595856559-g7tk9" Nov 01 16:05:56.588399 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:56.588363 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d121-5595856559-g7tk9" Nov 01 16:05:57.074534 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:05:57.074495 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c3d81b_79de_44cb_9838_8b07da7c911b.slice/crio-43b8978f4bc7a41ff6ea4452e8ccde65472fedc9a3ed3ae26bccb6a70e6ca595 WatchSource:0}: Error finding container 43b8978f4bc7a41ff6ea4452e8ccde65472fedc9a3ed3ae26bccb6a70e6ca595: Status 404 returned error can't find the container with id 43b8978f4bc7a41ff6ea4452e8ccde65472fedc9a3ed3ae26bccb6a70e6ca595 Nov 01 16:05:57.074672 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:57.074537 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d121-5595856559-g7tk9"] Nov 01 16:05:57.578696 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:57.578609 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d121-5595856559-g7tk9" event={"ID":"55c3d81b-79de-44cb-9838-8b07da7c911b","Type":"ContainerStarted","Data":"176a0024fc3496d320e703914d7fca1fb729234e11b273152fa97d284fe177f9"} Nov 01 16:05:57.578696 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:57.578692 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d121-5595856559-g7tk9" event={"ID":"55c3d81b-79de-44cb-9838-8b07da7c911b","Type":"ContainerStarted","Data":"43b8978f4bc7a41ff6ea4452e8ccde65472fedc9a3ed3ae26bccb6a70e6ca595"} Nov 01 16:05:57.579175 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:57.578719 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d121-5595856559-g7tk9" Nov 01 16:05:57.595145 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:05:57.595104 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d121-5595856559-g7tk9" podStartSLOduration=1.5950904320000001 podStartE2EDuration="1.595090432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:05:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:05:57.593370101 +0000 UTC m=+1090.925107751" watchObservedRunningTime="2026-11-01 16:05:57.595090432 +0000 UTC m=+1090.926828082" Nov 01 16:06:04.528482 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:04.528443 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a9a967-6948b69c67-tgrtj" Nov 01 16:06:08.590032 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:08.589984 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d121-5595856559-g7tk9" Nov 01 16:06:18.364180 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:06:18.364135 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:06:18.669224 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:18.664491 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-rsvc9"] Nov 01 16:06:18.671997 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:18.671583 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-rsvc9"] Nov 01 16:06:18.671997 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:18.671691 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-rsvc9" Nov 01 16:06:18.737499 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:18.728390 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-76063d10-dockercfg-vtdwb\"" Nov 01 16:06:18.737499 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:18.728543 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-76063d10\"" Nov 01 16:06:18.988614 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:18.988555 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-9p2wc"] Nov 01 16:06:19.186049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:19.162229 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be288817-6d19-47a1-a31c-7ee1c4946815-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-rsvc9\" (UID: \"be288817-6d19-47a1-a31c-7ee1c4946815\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-rsvc9" Nov 01 16:06:19.186049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:19.172222 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtg6p\" (UniqueName: \"kubernetes.io/projected/be288817-6d19-47a1-a31c-7ee1c4946815-kube-api-access-dtg6p\") pod \"payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-rsvc9\" (UID: \"be288817-6d19-47a1-a31c-7ee1c4946815\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-rsvc9" Nov 01 16:06:19.368008 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:19.367968 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be288817-6d19-47a1-a31c-7ee1c4946815-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-rsvc9\" (UID: \"be288817-6d19-47a1-a31c-7ee1c4946815\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-rsvc9" Nov 01 16:06:19.368671 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:19.368642 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtg6p\" (UniqueName: \"kubernetes.io/projected/be288817-6d19-47a1-a31c-7ee1c4946815-kube-api-access-dtg6p\") pod \"payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-rsvc9\" (UID: \"be288817-6d19-47a1-a31c-7ee1c4946815\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-rsvc9" Nov 01 16:06:19.369746 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:19.369716 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be288817-6d19-47a1-a31c-7ee1c4946815-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-rsvc9\" (UID: \"be288817-6d19-47a1-a31c-7ee1c4946815\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-rsvc9" Nov 01 16:06:19.374768 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:19.374742 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-9p2wc"] Nov 01 16:06:19.410048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:19.407489 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-9p2wc" Nov 01 16:06:19.453661 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:19.453623 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtg6p\" (UniqueName: \"kubernetes.io/projected/be288817-6d19-47a1-a31c-7ee1c4946815-kube-api-access-dtg6p\") pod \"payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-rsvc9\" (UID: \"be288817-6d19-47a1-a31c-7ee1c4946815\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-rsvc9" Nov 01 16:06:19.469851 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:19.469823 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2vkk\" (UniqueName: \"kubernetes.io/projected/2143f802-6cb6-409d-9015-84056431984b-kube-api-access-x2vkk\") pod \"payload-processing-e2e-mt-76063d10-f66f6f4bf-9p2wc\" (UID: \"2143f802-6cb6-409d-9015-84056431984b\") " pod="openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-9p2wc" Nov 01 16:06:19.470813 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:19.470786 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2143f802-6cb6-409d-9015-84056431984b-plugins-config-volume\") pod \"payload-processing-e2e-mt-76063d10-f66f6f4bf-9p2wc\" (UID: \"2143f802-6cb6-409d-9015-84056431984b\") " pod="openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-9p2wc" Nov 01 16:06:19.575002 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:19.574958 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2143f802-6cb6-409d-9015-84056431984b-plugins-config-volume\") pod \"payload-processing-e2e-mt-76063d10-f66f6f4bf-9p2wc\" (UID: \"2143f802-6cb6-409d-9015-84056431984b\") " pod="openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-9p2wc" Nov 01 16:06:19.576489 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:19.576459 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2vkk\" (UniqueName: \"kubernetes.io/projected/2143f802-6cb6-409d-9015-84056431984b-kube-api-access-x2vkk\") pod \"payload-processing-e2e-mt-76063d10-f66f6f4bf-9p2wc\" (UID: \"2143f802-6cb6-409d-9015-84056431984b\") " pod="openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-9p2wc" Nov 01 16:06:19.592755 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:19.577285 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2143f802-6cb6-409d-9015-84056431984b-plugins-config-volume\") pod \"payload-processing-e2e-mt-76063d10-f66f6f4bf-9p2wc\" (UID: \"2143f802-6cb6-409d-9015-84056431984b\") " pod="openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-9p2wc" Nov 01 16:06:19.603782 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:19.597320 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-rsvc9" Nov 01 16:06:19.699916 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:19.699872 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2vkk\" (UniqueName: \"kubernetes.io/projected/2143f802-6cb6-409d-9015-84056431984b-kube-api-access-x2vkk\") pod \"payload-processing-e2e-mt-76063d10-f66f6f4bf-9p2wc\" (UID: \"2143f802-6cb6-409d-9015-84056431984b\") " pod="openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-9p2wc" Nov 01 16:06:19.951088 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:19.945259 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-9p2wc" Nov 01 16:06:20.869817 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:20.869570 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-rsvc9"] Nov 01 16:06:20.869817 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:20.869621 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6948-3e1fb2"] Nov 01 16:06:20.935041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:20.934278 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6948-3e1fb2" Nov 01 16:06:20.940289 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:20.940190 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hqwjk\"" Nov 01 16:06:20.963204 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:20.963154 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6948-3e1fb2"] Nov 01 16:06:20.972448 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:20.970826 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-9p2wc"] Nov 01 16:06:20.976678 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:06:20.976653 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2143f802_6cb6_409d_9015_84056431984b.slice/crio-de379ebc012df6ab08094249daefce0118f68b53ee338e0aea4f2adcf0775919 WatchSource:0}: Error finding container de379ebc012df6ab08094249daefce0118f68b53ee338e0aea4f2adcf0775919: Status 404 returned error can't find the container with id de379ebc012df6ab08094249daefce0118f68b53ee338e0aea4f2adcf0775919 Nov 01 16:06:21.076090 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:21.076058 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjtj9\" (UniqueName: \"kubernetes.io/projected/0a3ea081-93aa-48ce-aeea-b6ea749b48a2-kube-api-access-kjtj9\") pod \"test-curl-6948-3e1fb2\" (UID: \"0a3ea081-93aa-48ce-aeea-b6ea749b48a2\") " pod="opendatahub/test-curl-6948-3e1fb2" Nov 01 16:06:21.214399 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:21.191155 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjtj9\" (UniqueName: \"kubernetes.io/projected/0a3ea081-93aa-48ce-aeea-b6ea749b48a2-kube-api-access-kjtj9\") pod \"test-curl-6948-3e1fb2\" (UID: \"0a3ea081-93aa-48ce-aeea-b6ea749b48a2\") " pod="opendatahub/test-curl-6948-3e1fb2" Nov 01 16:06:21.233051 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:21.224652 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjtj9\" (UniqueName: \"kubernetes.io/projected/0a3ea081-93aa-48ce-aeea-b6ea749b48a2-kube-api-access-kjtj9\") pod \"test-curl-6948-3e1fb2\" (UID: \"0a3ea081-93aa-48ce-aeea-b6ea749b48a2\") " pod="opendatahub/test-curl-6948-3e1fb2" Nov 01 16:06:21.461805 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:21.461360 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6948-3e1fb2" Nov 01 16:06:21.761852 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:21.761620 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-9p2wc" event={"ID":"2143f802-6cb6-409d-9015-84056431984b","Type":"ContainerStarted","Data":"f2270ff47454d6163baf0602be60a8443767475b60f4b1c6e26d4fdc63836527"} Nov 01 16:06:21.761852 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:21.761692 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-9p2wc" event={"ID":"2143f802-6cb6-409d-9015-84056431984b","Type":"ContainerStarted","Data":"de379ebc012df6ab08094249daefce0118f68b53ee338e0aea4f2adcf0775919"} Nov 01 16:06:21.762509 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:21.762484 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-9p2wc" Nov 01 16:06:21.766336 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:21.766313 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-rsvc9" event={"ID":"be288817-6d19-47a1-a31c-7ee1c4946815","Type":"ContainerStarted","Data":"b0fe2ea799cbccef4323ad290ca30f27445c90c73f6230964ae650cf498081ce"} Nov 01 16:06:21.766446 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:21.766353 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-rsvc9" event={"ID":"be288817-6d19-47a1-a31c-7ee1c4946815","Type":"ContainerStarted","Data":"f1cddc1770522cf18134a31b522b78c7a09bd142a14e441c24c2d3c53f53a29d"} Nov 01 16:06:21.766622 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:21.766606 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-rsvc9" Nov 01 16:06:21.850283 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:21.849085 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-9p2wc" podStartSLOduration=3.849066031 podStartE2EDuration="3.849066031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:06:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:06:21.847526181 +0000 UTC m=+1115.179263832" watchObservedRunningTime="2026-11-01 16:06:21.849066031 +0000 UTC m=+1115.180803680" Nov 01 16:06:21.868314 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:21.867528 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-rsvc9" podStartSLOduration=3.867489919 podStartE2EDuration="3.867489919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:06:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:06:21.864850943 +0000 UTC m=+1115.196588618" watchObservedRunningTime="2026-11-01 16:06:21.867489919 +0000 UTC m=+1115.199227570" Nov 01 16:06:22.382892 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:22.382852 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6948-3e1fb2"] Nov 01 16:06:22.774324 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:22.774233 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6948-3e1fb2" event={"ID":"0a3ea081-93aa-48ce-aeea-b6ea749b48a2","Type":"ContainerStarted","Data":"653d13d01923030495586d8042d11bdecc435f33308243d2e399db280a2ad288"} Nov 01 16:06:26.937181 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:26.936300 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6948-3e1fb2" event={"ID":"0a3ea081-93aa-48ce-aeea-b6ea749b48a2","Type":"ContainerStarted","Data":"fef888cbb0454778f82d61ea64863822513ac66a7b9fe49360e292be6d1d7e70"} Nov 01 16:06:26.954331 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:26.954274 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6948-3e1fb2" podStartSLOduration=2.970263107 podStartE2EDuration="6.954258972s" totalImagesPullingTime="3.983995865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:06:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:06:26.950808811 +0000 UTC m=+1120.282546464" watchObservedRunningTime="2026-11-01 16:06:26.954258972 +0000 UTC m=+1120.285996621" Nov 01 16:06:27.929270 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:27.929232 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6948-3e1fb2"] Nov 01 16:06:27.934147 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:27.934118 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6948-3e1fb2"] Nov 01 16:06:28.082054 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:28.081533 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6948-9d7fa9"] Nov 01 16:06:28.082922 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:28.082893 4363 cpu_manager.go:414] "containerMap: removing container" podUID="0a3ea081-93aa-48ce-aeea-b6ea749b48a2" containerName="test-curl-6948-3e1fb2" Nov 01 16:06:28.083125 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:28.083104 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a3ea081-93aa-48ce-aeea-b6ea749b48a2" containerName="test-curl-6948-3e1fb2" Nov 01 16:06:28.083501 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:28.083481 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a3ea081-93aa-48ce-aeea-b6ea749b48a2" containerName="test-curl-6948-3e1fb2" Nov 01 16:06:28.119292 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:28.118735 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6948-9d7fa9"] Nov 01 16:06:28.119292 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:28.118869 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6948-9d7fa9" Nov 01 16:06:28.200092 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:28.189417 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkgs5\" (UniqueName: \"kubernetes.io/projected/f4849583-a9bc-4f1d-93e8-43c273c75ce5-kube-api-access-nkgs5\") pod \"test-curl-6948-9d7fa9\" (UID: \"f4849583-a9bc-4f1d-93e8-43c273c75ce5\") " pod="opendatahub/test-curl-6948-9d7fa9" Nov 01 16:06:28.292078 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:28.291337 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkgs5\" (UniqueName: \"kubernetes.io/projected/f4849583-a9bc-4f1d-93e8-43c273c75ce5-kube-api-access-nkgs5\") pod \"test-curl-6948-9d7fa9\" (UID: \"f4849583-a9bc-4f1d-93e8-43c273c75ce5\") " pod="opendatahub/test-curl-6948-9d7fa9" Nov 01 16:06:28.552533 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:28.552442 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkgs5\" (UniqueName: \"kubernetes.io/projected/f4849583-a9bc-4f1d-93e8-43c273c75ce5-kube-api-access-nkgs5\") pod \"test-curl-6948-9d7fa9\" (UID: \"f4849583-a9bc-4f1d-93e8-43c273c75ce5\") " pod="opendatahub/test-curl-6948-9d7fa9" Nov 01 16:06:28.605541 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:28.605058 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6948-9d7fa9" Nov 01 16:06:29.038160 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:29.025426 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6948-3e1fb2" podUID="0a3ea081-93aa-48ce-aeea-b6ea749b48a2" containerName="test-curl-6948-3e1fb2" containerID="cri-o://fef888cbb0454778f82d61ea64863822513ac66a7b9fe49360e292be6d1d7e70" gracePeriod=2 Nov 01 16:06:29.216293 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:29.216262 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6948-9d7fa9"] Nov 01 16:06:29.217110 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:06:29.217080 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4849583_a9bc_4f1d_93e8_43c273c75ce5.slice/crio-a7e753d4e4ff9e82bed78614dc3c16f1f2f405da4d09da158413f6cdd5c6271e WatchSource:0}: Error finding container a7e753d4e4ff9e82bed78614dc3c16f1f2f405da4d09da158413f6cdd5c6271e: Status 404 returned error can't find the container with id a7e753d4e4ff9e82bed78614dc3c16f1f2f405da4d09da158413f6cdd5c6271e Nov 01 16:06:29.723600 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:29.723160 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:06:29.723600 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:29.723242 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" podUID="19456502-09eb-4dc7-8877-395f1b5e7218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:06:30.039002 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:30.038916 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6948-9d7fa9" event={"ID":"f4849583-a9bc-4f1d-93e8-43c273c75ce5","Type":"ContainerStarted","Data":"867659458f8de7410d24a64ec6422242a0084b03e5c2435dab7308c8aa08dbf5"} Nov 01 16:06:30.039002 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:30.038969 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6948-9d7fa9" event={"ID":"f4849583-a9bc-4f1d-93e8-43c273c75ce5","Type":"ContainerStarted","Data":"a7e753d4e4ff9e82bed78614dc3c16f1f2f405da4d09da158413f6cdd5c6271e"} Nov 01 16:06:30.056656 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:30.056593 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6948-9d7fa9" podStartSLOduration=2.056574046 podStartE2EDuration="2.056574046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:06:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:06:30.052101401 +0000 UTC m=+1123.383839051" watchObservedRunningTime="2026-11-01 16:06:30.056574046 +0000 UTC m=+1123.388311698" Nov 01 16:06:30.777008 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:30.776908 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6948-9d7fa9"] Nov 01 16:06:30.790162 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:30.788919 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6948-9d7fa9"] Nov 01 16:06:31.151191 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.063695 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6948-e44af9"] Nov 01 16:06:31.151191 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.065108 4363 cpu_manager.go:414] "containerMap: removing container" podUID="f4849583-a9bc-4f1d-93e8-43c273c75ce5" containerName="test-curl-6948-9d7fa9" Nov 01 16:06:31.151191 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.065126 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4849583-a9bc-4f1d-93e8-43c273c75ce5" containerName="test-curl-6948-9d7fa9" Nov 01 16:06:31.151191 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.065251 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4849583-a9bc-4f1d-93e8-43c273c75ce5" containerName="test-curl-6948-9d7fa9" Nov 01 16:06:31.151191 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.090685 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6948-e44af9" Nov 01 16:06:31.152226 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.151335 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6948-e44af9"] Nov 01 16:06:31.167630 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.167592 4363 generic.go:356] "Generic (PLEG): container finished" podID="0a3ea081-93aa-48ce-aeea-b6ea749b48a2" containerID="fef888cbb0454778f82d61ea64863822513ac66a7b9fe49360e292be6d1d7e70" exitCode=137 Nov 01 16:06:31.257483 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.257447 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbqh7\" (UniqueName: \"kubernetes.io/projected/3acc0ff2-ddc0-4b94-8d50-445d2c4d6d47-kube-api-access-fbqh7\") pod \"test-curl-6948-e44af9\" (UID: \"3acc0ff2-ddc0-4b94-8d50-445d2c4d6d47\") " pod="opendatahub/test-curl-6948-e44af9" Nov 01 16:06:31.309033 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.308987 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72"] Nov 01 16:06:31.320368 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.320323 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" Nov 01 16:06:31.327546 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.327517 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72"] Nov 01 16:06:31.359073 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.359001 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27bb1f1-cb87-4003-862c-4839331f5c9e-workload-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72\" (UID: \"a27bb1f1-cb87-4003-862c-4839331f5c9e\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" Nov 01 16:06:31.359647 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.359243 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a27bb1f1-cb87-4003-862c-4839331f5c9e-istio-token\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72\" (UID: \"a27bb1f1-cb87-4003-862c-4839331f5c9e\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" Nov 01 16:06:31.359647 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.359292 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27bb1f1-cb87-4003-862c-4839331f5c9e-credential-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72\" (UID: \"a27bb1f1-cb87-4003-862c-4839331f5c9e\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" Nov 01 16:06:31.359647 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.359312 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a27bb1f1-cb87-4003-862c-4839331f5c9e-workload-certs\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72\" (UID: \"a27bb1f1-cb87-4003-862c-4839331f5c9e\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" Nov 01 16:06:31.359647 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.359336 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a27bb1f1-cb87-4003-862c-4839331f5c9e-istio-envoy\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72\" (UID: \"a27bb1f1-cb87-4003-862c-4839331f5c9e\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" Nov 01 16:06:31.359647 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.359465 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a27bb1f1-cb87-4003-862c-4839331f5c9e-istiod-ca-cert\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72\" (UID: \"a27bb1f1-cb87-4003-862c-4839331f5c9e\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" Nov 01 16:06:31.359647 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.359536 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbqh7\" (UniqueName: \"kubernetes.io/projected/3acc0ff2-ddc0-4b94-8d50-445d2c4d6d47-kube-api-access-fbqh7\") pod \"test-curl-6948-e44af9\" (UID: \"3acc0ff2-ddc0-4b94-8d50-445d2c4d6d47\") " pod="opendatahub/test-curl-6948-e44af9" Nov 01 16:06:31.359647 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.359592 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a27bb1f1-cb87-4003-862c-4839331f5c9e-istio-data\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72\" (UID: \"a27bb1f1-cb87-4003-862c-4839331f5c9e\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" Nov 01 16:06:31.359647 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.359608 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94cgp\" (UniqueName: \"kubernetes.io/projected/a27bb1f1-cb87-4003-862c-4839331f5c9e-kube-api-access-94cgp\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72\" (UID: \"a27bb1f1-cb87-4003-862c-4839331f5c9e\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" Nov 01 16:06:31.360087 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.359961 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a27bb1f1-cb87-4003-862c-4839331f5c9e-istio-podinfo\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72\" (UID: \"a27bb1f1-cb87-4003-862c-4839331f5c9e\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" Nov 01 16:06:31.370130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.370102 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbqh7\" (UniqueName: \"kubernetes.io/projected/3acc0ff2-ddc0-4b94-8d50-445d2c4d6d47-kube-api-access-fbqh7\") pod \"test-curl-6948-e44af9\" (UID: \"3acc0ff2-ddc0-4b94-8d50-445d2c4d6d47\") " pod="opendatahub/test-curl-6948-e44af9" Nov 01 16:06:31.393540 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.393511 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6948-3e1fb2" Nov 01 16:06:31.461140 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.461114 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a3ea081-93aa-48ce-aeea-b6ea749b48a2-kube-api-access-kjtj9\" (UniqueName: \"kubernetes.io/projected/0a3ea081-93aa-48ce-aeea-b6ea749b48a2-kube-api-access-kjtj9\") pod \"0a3ea081-93aa-48ce-aeea-b6ea749b48a2\" (UID: \"0a3ea081-93aa-48ce-aeea-b6ea749b48a2\") " Nov 01 16:06:31.461438 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.461416 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a27bb1f1-cb87-4003-862c-4839331f5c9e-istio-data\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72\" (UID: \"a27bb1f1-cb87-4003-862c-4839331f5c9e\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" Nov 01 16:06:31.461547 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.461442 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94cgp\" (UniqueName: \"kubernetes.io/projected/a27bb1f1-cb87-4003-862c-4839331f5c9e-kube-api-access-94cgp\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72\" (UID: \"a27bb1f1-cb87-4003-862c-4839331f5c9e\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" Nov 01 16:06:31.461547 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.461504 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a27bb1f1-cb87-4003-862c-4839331f5c9e-istio-podinfo\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72\" (UID: \"a27bb1f1-cb87-4003-862c-4839331f5c9e\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" Nov 01 16:06:31.461645 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.461552 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27bb1f1-cb87-4003-862c-4839331f5c9e-workload-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72\" (UID: \"a27bb1f1-cb87-4003-862c-4839331f5c9e\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" Nov 01 16:06:31.461645 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.461574 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a27bb1f1-cb87-4003-862c-4839331f5c9e-istio-token\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72\" (UID: \"a27bb1f1-cb87-4003-862c-4839331f5c9e\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" Nov 01 16:06:31.461645 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.461618 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27bb1f1-cb87-4003-862c-4839331f5c9e-credential-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72\" (UID: \"a27bb1f1-cb87-4003-862c-4839331f5c9e\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" Nov 01 16:06:31.461800 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.461647 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a27bb1f1-cb87-4003-862c-4839331f5c9e-workload-certs\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72\" (UID: \"a27bb1f1-cb87-4003-862c-4839331f5c9e\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" Nov 01 16:06:31.461800 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.461679 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a27bb1f1-cb87-4003-862c-4839331f5c9e-istio-envoy\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72\" (UID: \"a27bb1f1-cb87-4003-862c-4839331f5c9e\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" Nov 01 16:06:31.461953 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.461827 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a27bb1f1-cb87-4003-862c-4839331f5c9e-istio-data\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72\" (UID: \"a27bb1f1-cb87-4003-862c-4839331f5c9e\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" Nov 01 16:06:31.461953 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.461847 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a27bb1f1-cb87-4003-862c-4839331f5c9e-istiod-ca-cert\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72\" (UID: \"a27bb1f1-cb87-4003-862c-4839331f5c9e\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" Nov 01 16:06:31.461953 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.461916 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27bb1f1-cb87-4003-862c-4839331f5c9e-workload-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72\" (UID: \"a27bb1f1-cb87-4003-862c-4839331f5c9e\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" Nov 01 16:06:31.462169 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.462009 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27bb1f1-cb87-4003-862c-4839331f5c9e-credential-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72\" (UID: \"a27bb1f1-cb87-4003-862c-4839331f5c9e\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" Nov 01 16:06:31.462169 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.462122 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a27bb1f1-cb87-4003-862c-4839331f5c9e-workload-certs\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72\" (UID: \"a27bb1f1-cb87-4003-862c-4839331f5c9e\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" Nov 01 16:06:31.462676 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.462635 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a27bb1f1-cb87-4003-862c-4839331f5c9e-istiod-ca-cert\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72\" (UID: \"a27bb1f1-cb87-4003-862c-4839331f5c9e\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" Nov 01 16:06:31.463680 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.463660 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a3ea081-93aa-48ce-aeea-b6ea749b48a2-kube-api-access-kjtj9" pod "0a3ea081-93aa-48ce-aeea-b6ea749b48a2" (UID: "0a3ea081-93aa-48ce-aeea-b6ea749b48a2"). InnerVolumeSpecName "kube-api-access-kjtj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:06:31.463984 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.463966 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a27bb1f1-cb87-4003-862c-4839331f5c9e-istio-envoy\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72\" (UID: \"a27bb1f1-cb87-4003-862c-4839331f5c9e\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" Nov 01 16:06:31.464057 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.463998 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a27bb1f1-cb87-4003-862c-4839331f5c9e-istio-podinfo\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72\" (UID: \"a27bb1f1-cb87-4003-862c-4839331f5c9e\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" Nov 01 16:06:31.469752 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.469729 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a27bb1f1-cb87-4003-862c-4839331f5c9e-istio-token\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72\" (UID: \"a27bb1f1-cb87-4003-862c-4839331f5c9e\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" Nov 01 16:06:31.469837 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.469763 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94cgp\" (UniqueName: \"kubernetes.io/projected/a27bb1f1-cb87-4003-862c-4839331f5c9e-kube-api-access-94cgp\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72\" (UID: \"a27bb1f1-cb87-4003-862c-4839331f5c9e\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" Nov 01 16:06:31.544963 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.544936 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6948-e44af9" Nov 01 16:06:31.563455 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.563431 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjtj9\" (UniqueName: \"kubernetes.io/projected/0a3ea081-93aa-48ce-aeea-b6ea749b48a2-kube-api-access-kjtj9\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:06:31.712041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:31.704783 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" Nov 01 16:06:32.034103 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:32.034069 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6948-e44af9"] Nov 01 16:06:32.034444 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:06:32.034267 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3acc0ff2_ddc0_4b94_8d50_445d2c4d6d47.slice/crio-0083934ceebcc84a0c942ca4a9a1070533c303a573bb5fc85c818410c1a2741b WatchSource:0}: Error finding container 0083934ceebcc84a0c942ca4a9a1070533c303a573bb5fc85c818410c1a2741b: Status 404 returned error can't find the container with id 0083934ceebcc84a0c942ca4a9a1070533c303a573bb5fc85c818410c1a2741b Nov 01 16:06:32.132999 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:32.132970 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72"] Nov 01 16:06:32.133170 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:06:32.133149 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda27bb1f1_cb87_4003_862c_4839331f5c9e.slice/crio-e341d8071458ade3c1fd32c03ba1121b15a1de65849fa47f7bf5a06385777a06 WatchSource:0}: Error finding container e341d8071458ade3c1fd32c03ba1121b15a1de65849fa47f7bf5a06385777a06: Status 404 returned error can't find the container with id e341d8071458ade3c1fd32c03ba1121b15a1de65849fa47f7bf5a06385777a06 Nov 01 16:06:32.136344 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:32.136313 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:06:32.136428 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:32.136376 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:06:32.136428 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:32.136406 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:06:32.176386 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:32.176361 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" event={"ID":"a27bb1f1-cb87-4003-862c-4839331f5c9e","Type":"ContainerStarted","Data":"e341d8071458ade3c1fd32c03ba1121b15a1de65849fa47f7bf5a06385777a06"} Nov 01 16:06:32.177996 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:32.177973 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6948-3e1fb2" Nov 01 16:06:32.178126 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:32.178052 4363 scope.go:122] "RemoveContainer" containerID="fef888cbb0454778f82d61ea64863822513ac66a7b9fe49360e292be6d1d7e70" Nov 01 16:06:32.181649 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:32.181610 4363 generic.go:356] "Generic (PLEG): container finished" podID="420774ff-0dfc-44ef-a9c3-19a41f76a985" containerID="071c82dff01760262ca73b5e4ec7256daf0a0d77811695e658ab57db7052b1da" exitCode=0 Nov 01 16:06:32.181748 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:32.181730 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" event={"ID":"420774ff-0dfc-44ef-a9c3-19a41f76a985","Type":"ContainerDied","Data":"071c82dff01760262ca73b5e4ec7256daf0a0d77811695e658ab57db7052b1da"} Nov 01 16:06:32.182726 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:32.182228 4363 scope.go:122] "RemoveContainer" containerID="071c82dff01760262ca73b5e4ec7256daf0a0d77811695e658ab57db7052b1da" Nov 01 16:06:32.184497 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:32.184462 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:06:32.184707 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:32.184563 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:06:32.184707 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:32.184611 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:06:32.184707 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:32.184652 4363 generic.go:356] "Generic (PLEG): container finished" podID="19456502-09eb-4dc7-8877-395f1b5e7218" containerID="a5bfdb3891ac394f0b406fe74719eba6447cb1299f7d34d424160ae4e9975d32" exitCode=0 Nov 01 16:06:32.184943 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:32.184739 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" event={"ID":"19456502-09eb-4dc7-8877-395f1b5e7218","Type":"ContainerDied","Data":"a5bfdb3891ac394f0b406fe74719eba6447cb1299f7d34d424160ae4e9975d32"} Nov 01 16:06:32.185239 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:32.185188 4363 scope.go:122] "RemoveContainer" containerID="a5bfdb3891ac394f0b406fe74719eba6447cb1299f7d34d424160ae4e9975d32" Nov 01 16:06:32.186960 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:32.186929 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:06:32.187092 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:32.187009 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:06:32.187180 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:32.187123 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:06:32.187946 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:32.187449 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6948-9d7fa9" podUID="f4849583-a9bc-4f1d-93e8-43c273c75ce5" containerName="test-curl-6948-9d7fa9" containerID="cri-o://867659458f8de7410d24a64ec6422242a0084b03e5c2435dab7308c8aa08dbf5" gracePeriod=2 Nov 01 16:06:32.187946 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:32.187557 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6948-e44af9" event={"ID":"3acc0ff2-ddc0-4b94-8d50-445d2c4d6d47","Type":"ContainerStarted","Data":"2145d19cfaf2008d4e5eacfd3846e5de93c21ad297507f2d94539b3c1c4fd581"} Nov 01 16:06:32.187946 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:32.187587 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6948-e44af9" event={"ID":"3acc0ff2-ddc0-4b94-8d50-445d2c4d6d47","Type":"ContainerStarted","Data":"0083934ceebcc84a0c942ca4a9a1070533c303a573bb5fc85c818410c1a2741b"} Nov 01 16:06:32.245228 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:06:32.243128 4363 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6948-3e1fb2\" is forbidden: User \"system:node:ip-10-0-128-238.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-238.us-east-2.compute.internal' and this object" podUID="0a3ea081-93aa-48ce-aeea-b6ea749b48a2" pod="opendatahub/test-curl-6948-3e1fb2" Nov 01 16:06:32.248076 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:32.246922 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6948-e44af9" podStartSLOduration=2.24690571 podStartE2EDuration="2.24690571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:06:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:06:32.241181891 +0000 UTC m=+1125.572919542" watchObservedRunningTime="2026-11-01 16:06:32.24690571 +0000 UTC m=+1125.578643361" Nov 01 16:06:32.777778 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:32.777689 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-rsvc9" Nov 01 16:06:32.779009 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:32.778981 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-9p2wc" Nov 01 16:06:32.804081 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:06:32.804040 4363 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6948-3e1fb2\" is forbidden: User \"system:node:ip-10-0-128-238.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-238.us-east-2.compute.internal' and this object" podUID="0a3ea081-93aa-48ce-aeea-b6ea749b48a2" pod="opendatahub/test-curl-6948-3e1fb2" Nov 01 16:06:32.815368 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:32.815336 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a3ea081-93aa-48ce-aeea-b6ea749b48a2" path="/var/lib/kubelet/pods/0a3ea081-93aa-48ce-aeea-b6ea749b48a2/volumes" Nov 01 16:06:32.829409 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:06:32.829374 4363 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6948-3e1fb2\" is forbidden: User \"system:node:ip-10-0-128-238.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-238.us-east-2.compute.internal' and this object" podUID="0a3ea081-93aa-48ce-aeea-b6ea749b48a2" pod="opendatahub/test-curl-6948-3e1fb2" Nov 01 16:06:33.302330 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:33.302186 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6948-e44af9"] Nov 01 16:06:33.562101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:33.546259 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6948-e44af9"] Nov 01 16:06:33.758753 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:33.758708 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" event={"ID":"420774ff-0dfc-44ef-a9c3-19a41f76a985","Type":"ContainerStarted","Data":"681ff7838753f6e911ecadf9f3b2eb5b04772a76a9cfbff05dece650caa4a517"} Nov 01 16:06:33.767138 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:33.767088 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" event={"ID":"19456502-09eb-4dc7-8877-395f1b5e7218","Type":"ContainerStarted","Data":"78c2dd5df3b02f6d6dd5d479cbb2392b759b43038a569cc0550ee78de1e5ee13"} Nov 01 16:06:33.769867 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:33.769838 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" event={"ID":"a27bb1f1-cb87-4003-862c-4839331f5c9e","Type":"ContainerStarted","Data":"052cf4da2e63f6ac8ee0ff54803cec24498a47a85e98dbb4c86e6417ac97a930"} Nov 01 16:06:33.915005 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:33.914964 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" Nov 01 16:06:33.990042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:33.967115 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 01 16:06:33.990042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:33.967202 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" podUID="420774ff-0dfc-44ef-a9c3-19a41f76a985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 01 16:06:34.100041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:34.093100 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6948-49e0a8"] Nov 01 16:06:34.113298 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:34.113262 4363 cpu_manager.go:414] "containerMap: removing container" podUID="3acc0ff2-ddc0-4b94-8d50-445d2c4d6d47" containerName="test-curl-6948-e44af9" Nov 01 16:06:34.113548 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:34.113532 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3acc0ff2-ddc0-4b94-8d50-445d2c4d6d47" containerName="test-curl-6948-e44af9" Nov 01 16:06:34.114161 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:34.114142 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="3acc0ff2-ddc0-4b94-8d50-445d2c4d6d47" containerName="test-curl-6948-e44af9" Nov 01 16:06:34.184137 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:34.173757 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6948-49e0a8" Nov 01 16:06:34.311948 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:34.290999 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:06:34.338180 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:34.338138 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 01 16:06:34.390459 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:34.339392 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" Nov 01 16:06:34.429047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:34.396916 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4lgr\" (UniqueName: \"kubernetes.io/projected/9f3a2621-6cbb-4ae1-88bc-61873eb0d0a2-kube-api-access-g4lgr\") pod \"test-curl-6948-49e0a8\" (UID: \"9f3a2621-6cbb-4ae1-88bc-61873eb0d0a2\") " pod="opendatahub/test-curl-6948-49e0a8" Nov 01 16:06:34.551097 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:34.541230 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6948-49e0a8"] Nov 01 16:06:34.648235 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:34.647870 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4lgr\" (UniqueName: \"kubernetes.io/projected/9f3a2621-6cbb-4ae1-88bc-61873eb0d0a2-kube-api-access-g4lgr\") pod \"test-curl-6948-49e0a8\" (UID: \"9f3a2621-6cbb-4ae1-88bc-61873eb0d0a2\") " pod="opendatahub/test-curl-6948-49e0a8" Nov 01 16:06:34.648235 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:34.648226 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" Nov 01 16:06:34.665045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:34.653291 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 01 16:06:34.665045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:34.653350 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" podUID="19456502-09eb-4dc7-8877-395f1b5e7218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 01 16:06:34.813099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:34.758455 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" Nov 01 16:06:35.079113 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:35.015614 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:06:35.079113 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:35.025054 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:06:35.079113 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:35.025498 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:06:35.079113 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:34.783290 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 01 16:06:35.079113 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:35.026696 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" podUID="a27bb1f1-cb87-4003-862c-4839331f5c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 01 16:06:35.274169 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:35.274131 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 01 16:06:35.327117 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:35.320745 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" podUID="420774ff-0dfc-44ef-a9c3-19a41f76a985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 01 16:06:35.327117 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:35.321669 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4lgr\" (UniqueName: \"kubernetes.io/projected/9f3a2621-6cbb-4ae1-88bc-61873eb0d0a2-kube-api-access-g4lgr\") pod \"test-curl-6948-49e0a8\" (UID: \"9f3a2621-6cbb-4ae1-88bc-61873eb0d0a2\") " pod="opendatahub/test-curl-6948-49e0a8" Nov 01 16:06:35.327117 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:35.325779 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" podStartSLOduration=4.325761835 podStartE2EDuration="4.325761835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:06:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:06:33.968032676 +0000 UTC m=+1127.299770322" watchObservedRunningTime="2026-11-01 16:06:35.325761835 +0000 UTC m=+1128.657499489" Nov 01 16:06:35.341082 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:35.340992 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6948-49e0a8" Nov 01 16:06:35.389923 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:35.389873 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:06:35.487777 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:35.486458 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 16:06:35.541050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:35.540984 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:06:35.737174 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:35.692769 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:06:36.045244 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:36.045149 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 01 16:06:36.093080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:36.092922 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" podUID="a27bb1f1-cb87-4003-862c-4839331f5c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 01 16:06:36.093352 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:36.066281 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:06:36.101071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:36.066224 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:06:36.118156 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:36.115098 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:06:36.118156 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:36.115232 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:06:36.118156 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:36.051169 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 01 16:06:36.118156 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:36.116118 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" podUID="19456502-09eb-4dc7-8877-395f1b5e7218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 01 16:06:36.147834 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:36.147790 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 01 16:06:36.148163 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:36.148106 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" podUID="420774ff-0dfc-44ef-a9c3-19a41f76a985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 01 16:06:36.212001 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:36.211972 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 16:06:36.219092 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:36.218499 4363 generic.go:356] "Generic (PLEG): container finished" podID="f4849583-a9bc-4f1d-93e8-43c273c75ce5" containerID="867659458f8de7410d24a64ec6422242a0084b03e5c2435dab7308c8aa08dbf5" exitCode=137 Nov 01 16:06:36.234385 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:36.234351 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6948-e44af9" podUID="3acc0ff2-ddc0-4b94-8d50-445d2c4d6d47" containerName="test-curl-6948-e44af9" containerID="cri-o://2145d19cfaf2008d4e5eacfd3846e5de93c21ad297507f2d94539b3c1c4fd581" gracePeriod=2 Nov 01 16:06:36.600077 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:36.557362 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:06:36.600077 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:36.557990 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:06:36.675101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:36.657203 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 01 16:06:36.675101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:36.657265 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" podUID="19456502-09eb-4dc7-8877-395f1b5e7218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 01 16:06:36.675101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:36.660783 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:06:36.705744 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:36.705689 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:06:36.707032 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:36.706994 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:06:36.884096 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:36.877769 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 01 16:06:36.884096 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:36.877846 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" podUID="a27bb1f1-cb87-4003-862c-4839331f5c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 01 16:06:37.019041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:37.015244 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:06:37.069041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:37.063690 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 01 16:06:37.069041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:37.063756 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" podUID="420774ff-0dfc-44ef-a9c3-19a41f76a985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 01 16:06:37.120416 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:37.115141 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:06:37.246090 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:37.215487 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:06:37.734377 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:37.734308 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:06:37.788591 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:37.788555 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 01 16:06:37.788871 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:37.788843 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" podUID="a27bb1f1-cb87-4003-862c-4839331f5c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 01 16:06:37.812044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:37.789524 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 01 16:06:38.523042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:38.472886 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" podUID="19456502-09eb-4dc7-8877-395f1b5e7218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 01 16:06:38.523358 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:38.208743 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:06:38.588082 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:06:38.345833 4363 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.537s" Nov 01 16:06:38.588082 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:38.584033 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:06:38.588082 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:38.415153 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 01 16:06:38.622080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:37.924686 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr" podUID="305de5fb-52ed-401a-89fe-b7d802472f5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:06:38.623344 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:38.472807 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 01 16:06:38.623567 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:38.623539 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" podUID="420774ff-0dfc-44ef-a9c3-19a41f76a985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 01 16:06:38.623686 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:38.528703 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:06:38.733042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:38.692943 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 16:06:38.733042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:38.693054 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 16:06:38.733042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:38.511138 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6948-9d7fa9" Nov 01 16:06:38.733042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:38.528529 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:06:38.733042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:38.558462 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:06:38.800009 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:38.799977 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 01 16:06:38.804557 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:38.804516 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" podUID="a27bb1f1-cb87-4003-862c-4839331f5c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 01 16:06:38.963100 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:38.852399 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:06:38.963100 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:38.854613 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4849583-a9bc-4f1d-93e8-43c273c75ce5-kube-api-access-nkgs5\" (UniqueName: \"kubernetes.io/projected/f4849583-a9bc-4f1d-93e8-43c273c75ce5-kube-api-access-nkgs5\") pod \"f4849583-a9bc-4f1d-93e8-43c273c75ce5\" (UID: \"f4849583-a9bc-4f1d-93e8-43c273c75ce5\") " Nov 01 16:06:38.963100 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:38.869546 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4849583-a9bc-4f1d-93e8-43c273c75ce5-kube-api-access-nkgs5" pod "f4849583-a9bc-4f1d-93e8-43c273c75ce5" (UID: "f4849583-a9bc-4f1d-93e8-43c273c75ce5"). InnerVolumeSpecName "kube-api-access-nkgs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:06:38.963100 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:38.937205 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6948-9d7fa9" Nov 01 16:06:38.963100 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:38.939683 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8h5j2" podUID="14cafac8-b130-4780-aa9e-e60215541376" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded" Nov 01 16:06:38.996039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:38.995169 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 01 16:06:38.996039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:38.995235 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" podUID="19456502-09eb-4dc7-8877-395f1b5e7218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 01 16:06:39.250619 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:39.250484 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 01 16:06:39.250885 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:39.250852 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" podUID="420774ff-0dfc-44ef-a9c3-19a41f76a985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 01 16:06:39.505287 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:06:39.505091 4363 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6948-9d7fa9\" is forbidden: User \"system:node:ip-10-0-128-238.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-238.us-east-2.compute.internal' and this object" podUID="f4849583-a9bc-4f1d-93e8-43c273c75ce5" pod="opendatahub/test-curl-6948-9d7fa9" Nov 01 16:06:39.531605 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:39.531570 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkgs5\" (UniqueName: \"kubernetes.io/projected/f4849583-a9bc-4f1d-93e8-43c273c75ce5-kube-api-access-nkgs5\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:06:39.581241 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:39.576538 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4849583-a9bc-4f1d-93e8-43c273c75ce5" path="/var/lib/kubelet/pods/f4849583-a9bc-4f1d-93e8-43c273c75ce5/volumes" Nov 01 16:06:39.581241 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:39.577402 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6948-49e0a8"] Nov 01 16:06:39.581241 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:39.577451 4363 scope.go:122] "RemoveContainer" containerID="867659458f8de7410d24a64ec6422242a0084b03e5c2435dab7308c8aa08dbf5" Nov 01 16:06:39.599050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:39.583406 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:06:39.599050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:39.593586 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 01 16:06:39.645744 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:39.645693 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:06:39.728044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:39.701710 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:06:39.755227 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:39.728693 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:06:39.756507 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:39.756428 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:06:40.166434 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:39.757237 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 01 16:06:40.166983 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:40.166952 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" podUID="19456502-09eb-4dc7-8877-395f1b5e7218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 01 16:06:40.167482 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:39.896637 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:06:40.180605 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:39.959877 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 01 16:06:40.180829 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:40.180800 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" podUID="420774ff-0dfc-44ef-a9c3-19a41f76a985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 01 16:06:40.222615 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:40.222582 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 01 16:06:40.226044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:40.225990 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" podUID="a27bb1f1-cb87-4003-862c-4839331f5c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 01 16:06:40.465119 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:40.439240 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:06:40.465119 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:40.461663 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6948-49e0a8" event={"ID":"9f3a2621-6cbb-4ae1-88bc-61873eb0d0a2","Type":"ContainerStarted","Data":"c92de3428e634d42a8377234c72e58f6650f079aab5b784c3aeb7ada1a08258b"} Nov 01 16:06:40.465119 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:40.461751 4363 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 16:06:40.753094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:40.686624 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:06:40.754532 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:40.715103 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:06:40.808913 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:40.715163 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 01 16:06:40.953047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:40.716527 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f47890-84fdfb9c59-fs5nn"] Nov 01 16:06:40.974056 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:40.771229 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 01 16:06:40.974056 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:40.970311 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" podUID="a27bb1f1-cb87-4003-862c-4839331f5c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 01 16:06:40.974056 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:40.776743 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:06:41.001054 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:40.772742 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:06:41.001054 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:40.776697 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 01 16:06:41.001054 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:40.999957 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" podUID="19456502-09eb-4dc7-8877-395f1b5e7218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 01 16:06:41.001054 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:40.946227 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 01 16:06:41.001054 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:41.000130 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" podUID="420774ff-0dfc-44ef-a9c3-19a41f76a985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 01 16:06:41.184040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:41.168127 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:06:41.246041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:41.239546 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f47890-84fdfb9c59-fs5nn" Nov 01 16:06:41.272049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:41.266309 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-f47890-dockercfg-9k9tc\"" Nov 01 16:06:41.311348 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:41.292286 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbvbt\" (UniqueName: \"kubernetes.io/projected/18faa082-6418-4665-b862-09efa76dfcc0-kube-api-access-mbvbt\") pod \"payload-pre-processing-e2e-shared-a-w3-f47890-84fdfb9c59-fs5nn\" (UID: \"18faa082-6418-4665-b862-09efa76dfcc0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f47890-84fdfb9c59-fs5nn" Nov 01 16:06:41.311348 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:41.292415 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18faa082-6418-4665-b862-09efa76dfcc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-f47890-84fdfb9c59-fs5nn\" (UID: \"18faa082-6418-4665-b862-09efa76dfcc0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f47890-84fdfb9c59-fs5nn" Nov 01 16:06:41.388668 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:41.344991 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-f47890\"" Nov 01 16:06:41.489206 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:41.452947 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbvbt\" (UniqueName: \"kubernetes.io/projected/18faa082-6418-4665-b862-09efa76dfcc0-kube-api-access-mbvbt\") pod \"payload-pre-processing-e2e-shared-a-w3-f47890-84fdfb9c59-fs5nn\" (UID: \"18faa082-6418-4665-b862-09efa76dfcc0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f47890-84fdfb9c59-fs5nn" Nov 01 16:06:41.543666 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:41.537322 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 16:06:41.543821 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:41.543700 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f47890-84fdfb9c59-fs5nn"] Nov 01 16:06:41.543821 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:41.543638 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18faa082-6418-4665-b862-09efa76dfcc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-f47890-84fdfb9c59-fs5nn\" (UID: \"18faa082-6418-4665-b862-09efa76dfcc0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f47890-84fdfb9c59-fs5nn" Nov 01 16:06:41.544110 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:41.543998 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:06:41.545185 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:41.545150 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18faa082-6418-4665-b862-09efa76dfcc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-f47890-84fdfb9c59-fs5nn\" (UID: \"18faa082-6418-4665-b862-09efa76dfcc0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f47890-84fdfb9c59-fs5nn" Nov 01 16:06:41.571569 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:41.571469 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbvbt\" (UniqueName: \"kubernetes.io/projected/18faa082-6418-4665-b862-09efa76dfcc0-kube-api-access-mbvbt\") pod \"payload-pre-processing-e2e-shared-a-w3-f47890-84fdfb9c59-fs5nn\" (UID: \"18faa082-6418-4665-b862-09efa76dfcc0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f47890-84fdfb9c59-fs5nn" Nov 01 16:06:41.642333 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:41.638267 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 01 16:06:41.649472 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:41.649432 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" podUID="19456502-09eb-4dc7-8877-395f1b5e7218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 01 16:06:41.657300 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:41.657273 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:06:41.750159 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:41.720433 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f47890-84fdfb9c59-fs5nn" Nov 01 16:06:41.792046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:41.762891 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:06:41.792046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:41.762927 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:06:41.810042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:41.793380 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 01 16:06:41.810377 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:41.810336 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" podUID="a27bb1f1-cb87-4003-862c-4839331f5c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 01 16:06:42.072698 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:41.765957 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:06:42.073997 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:42.068665 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 01 16:06:42.079353 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:42.079319 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" podUID="420774ff-0dfc-44ef-a9c3-19a41f76a985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 01 16:06:42.079548 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:42.069722 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:06:42.273962 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:42.273599 4363 generic.go:356] "Generic (PLEG): container finished" podID="3acc0ff2-ddc0-4b94-8d50-445d2c4d6d47" containerID="2145d19cfaf2008d4e5eacfd3846e5de93c21ad297507f2d94539b3c1c4fd581" exitCode=137 Nov 01 16:06:42.800125 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:42.797793 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:06:42.800125 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:42.798154 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:06:42.860864 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:42.830864 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:06:43.081094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:43.064199 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 01 16:06:43.081094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:43.064263 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" podUID="a27bb1f1-cb87-4003-862c-4839331f5c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 01 16:06:43.081094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:43.064417 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 01 16:06:43.081094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:43.064465 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" podUID="420774ff-0dfc-44ef-a9c3-19a41f76a985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 01 16:06:43.081094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:43.070217 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 01 16:06:43.081094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:43.070264 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" podUID="19456502-09eb-4dc7-8877-395f1b5e7218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 01 16:06:43.165837 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:43.149003 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 01 16:06:43.165837 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:43.150688 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:06:43.478121 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:43.478089 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:06:43.750096 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:43.739352 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:06:43.819524 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:43.761396 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 01 16:06:43.819524 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:43.761469 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" podUID="19456502-09eb-4dc7-8877-395f1b5e7218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 01 16:06:43.819524 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:43.768536 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" Nov 01 16:06:43.819524 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:43.807036 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:06:43.834160 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:43.834115 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:06:43.835263 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:43.834871 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:06:43.954608 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:43.852579 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f47890-84fdfb9c59-fs5nn"] Nov 01 16:06:43.994654 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:43.994626 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 16:06:43.994945 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:43.994926 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" Nov 01 16:06:44.132966 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:43.878788 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6948-e44af9" Nov 01 16:06:44.134519 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:44.134488 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3acc0ff2-ddc0-4b94-8d50-445d2c4d6d47-kube-api-access-fbqh7\" (UniqueName: \"kubernetes.io/projected/3acc0ff2-ddc0-4b94-8d50-445d2c4d6d47-kube-api-access-fbqh7\") pod \"3acc0ff2-ddc0-4b94-8d50-445d2c4d6d47\" (UID: \"3acc0ff2-ddc0-4b94-8d50-445d2c4d6d47\") " Nov 01 16:06:44.232368 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:44.219121 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" Nov 01 16:06:44.232368 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:44.219211 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" Nov 01 16:06:44.232368 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:44.219431 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3acc0ff2-ddc0-4b94-8d50-445d2c4d6d47-kube-api-access-fbqh7" pod "3acc0ff2-ddc0-4b94-8d50-445d2c4d6d47" (UID: "3acc0ff2-ddc0-4b94-8d50-445d2c4d6d47"). InnerVolumeSpecName "kube-api-access-fbqh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:06:44.240053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:44.233617 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:06:44.240053 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:06:44.233801 4363 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6948-e44af9\" is forbidden: User \"system:node:ip-10-0-128-238.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-238.us-east-2.compute.internal' and this object" podUID="3acc0ff2-ddc0-4b94-8d50-445d2c4d6d47" pod="opendatahub/test-curl-6948-e44af9" Nov 01 16:06:44.244002 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:44.242970 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" Nov 01 16:06:44.265669 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:44.265641 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" Nov 01 16:06:44.300241 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:44.300216 4363 scope.go:122] "RemoveContainer" containerID="2145d19cfaf2008d4e5eacfd3846e5de93c21ad297507f2d94539b3c1c4fd581" Nov 01 16:06:44.301960 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:44.301932 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbqh7\" (UniqueName: \"kubernetes.io/projected/3acc0ff2-ddc0-4b94-8d50-445d2c4d6d47-kube-api-access-fbqh7\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:06:44.315547 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:44.315512 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f47890-84fdfb9c59-fs5nn" event={"ID":"18faa082-6418-4665-b862-09efa76dfcc0","Type":"ContainerStarted","Data":"3f6db0e6f8d11490c6aa2b6e355e1b714069c1304719171183cc69e3324b4d5f"} Nov 01 16:06:44.341447 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:06:44.341345 4363 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6948-e44af9\" is forbidden: User \"system:node:ip-10-0-128-238.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-238.us-east-2.compute.internal' and this object" podUID="3acc0ff2-ddc0-4b94-8d50-445d2c4d6d47" pod="opendatahub/test-curl-6948-e44af9" Nov 01 16:06:44.344725 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:06:44.344689 4363 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6948-e44af9\" is forbidden: User \"system:node:ip-10-0-128-238.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-238.us-east-2.compute.internal' and this object" podUID="3acc0ff2-ddc0-4b94-8d50-445d2c4d6d47" pod="opendatahub/test-curl-6948-e44af9" Nov 01 16:06:44.645181 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:44.645149 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" Nov 01 16:06:44.648542 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:06:44.648492 4363 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6948-e44af9\" is forbidden: User \"system:node:ip-10-0-128-238.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-238.us-east-2.compute.internal' and this object" podUID="3acc0ff2-ddc0-4b94-8d50-445d2c4d6d47" pod="opendatahub/test-curl-6948-e44af9" Nov 01 16:06:44.799511 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:44.799478 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3acc0ff2-ddc0-4b94-8d50-445d2c4d6d47" path="/var/lib/kubelet/pods/3acc0ff2-ddc0-4b94-8d50-445d2c4d6d47/volumes" Nov 01 16:06:45.323498 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:45.323465 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6948-e44af9" Nov 01 16:06:45.325215 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:45.325185 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f47890-84fdfb9c59-fs5nn" event={"ID":"18faa082-6418-4665-b862-09efa76dfcc0","Type":"ContainerStarted","Data":"29c3809b77e4154932cca1a2bde9b52af7ad2e0b5f4d661ed64bb44b9d9c7e85"} Nov 01 16:06:45.325387 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:45.325278 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f47890-84fdfb9c59-fs5nn" Nov 01 16:06:45.325684 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:06:45.325653 4363 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6948-e44af9\" is forbidden: User \"system:node:ip-10-0-128-238.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-238.us-east-2.compute.internal' and this object" podUID="3acc0ff2-ddc0-4b94-8d50-445d2c4d6d47" pod="opendatahub/test-curl-6948-e44af9" Nov 01 16:06:45.326990 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:45.326960 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6948-49e0a8" event={"ID":"9f3a2621-6cbb-4ae1-88bc-61873eb0d0a2","Type":"ContainerStarted","Data":"e451703741aebcba77e97d040caada46265b6d56b84da9aa9553e3c1713f32e9"} Nov 01 16:06:45.328529 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:45.328509 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" Nov 01 16:06:45.342624 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:06:45.342596 4363 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6948-e44af9\" is forbidden: User \"system:node:ip-10-0-128-238.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-238.us-east-2.compute.internal' and this object" podUID="3acc0ff2-ddc0-4b94-8d50-445d2c4d6d47" pod="opendatahub/test-curl-6948-e44af9" Nov 01 16:06:45.343604 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:45.343563 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f47890-84fdfb9c59-fs5nn" podStartSLOduration=5.343550049 podStartE2EDuration="5.343550049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:06:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:06:45.340647087 +0000 UTC m=+1138.672384738" watchObservedRunningTime="2026-11-01 16:06:45.343550049 +0000 UTC m=+1138.675287699" Nov 01 16:06:45.366915 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:45.366868 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6948-49e0a8" podStartSLOduration=12.366854949 podStartE2EDuration="12.366854949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:06:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:06:45.361688745 +0000 UTC m=+1138.693426393" watchObservedRunningTime="2026-11-01 16:06:45.366854949 +0000 UTC m=+1138.698592605" Nov 01 16:06:45.382589 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:06:45.382561 4363 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6948-e44af9\" is forbidden: User \"system:node:ip-10-0-128-238.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-238.us-east-2.compute.internal' and this object" podUID="3acc0ff2-ddc0-4b94-8d50-445d2c4d6d47" pod="opendatahub/test-curl-6948-e44af9" Nov 01 16:06:46.112637 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:46.112598 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6948-49e0a8"] Nov 01 16:06:46.120027 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:46.119178 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6948-49e0a8"] Nov 01 16:06:46.478270 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:46.478183 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm"] Nov 01 16:06:46.479119 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:46.479098 4363 cpu_manager.go:414] "containerMap: removing container" podUID="9f3a2621-6cbb-4ae1-88bc-61873eb0d0a2" containerName="test-curl-6948-49e0a8" Nov 01 16:06:46.479119 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:46.479119 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f3a2621-6cbb-4ae1-88bc-61873eb0d0a2" containerName="test-curl-6948-49e0a8" Nov 01 16:06:46.479305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:46.479270 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f3a2621-6cbb-4ae1-88bc-61873eb0d0a2" containerName="test-curl-6948-49e0a8" Nov 01 16:06:46.485671 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:46.485646 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" Nov 01 16:06:46.493648 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:46.492986 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-76063d10\"/\"kube-root-ca.crt\"" Nov 01 16:06:46.493648 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:46.493102 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-76063d10\"/\"openshift-service-ca.crt\"" Nov 01 16:06:46.495958 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:46.493799 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-76063d10\"/\"default-dockercfg-pk6bw\"" Nov 01 16:06:46.496115 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:46.493947 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-76063d10\"/\"e2e-lifecycle-model-76063d10-kserve-self-signed-certs\"" Nov 01 16:06:46.505672 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:46.505637 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1e164ab-e940-44e0-8829-8626e4162a9f-dshm\") pod \"e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm\" (UID: \"f1e164ab-e940-44e0-8829-8626e4162a9f\") " pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" Nov 01 16:06:46.506122 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:46.505781 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1e164ab-e940-44e0-8829-8626e4162a9f-model-cache\") pod \"e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm\" (UID: \"f1e164ab-e940-44e0-8829-8626e4162a9f\") " pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" Nov 01 16:06:46.506122 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:46.505850 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqxgk\" (UniqueName: \"kubernetes.io/projected/f1e164ab-e940-44e0-8829-8626e4162a9f-kube-api-access-wqxgk\") pod \"e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm\" (UID: \"f1e164ab-e940-44e0-8829-8626e4162a9f\") " pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" Nov 01 16:06:46.506122 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:46.506078 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1e164ab-e940-44e0-8829-8626e4162a9f-tls-certs\") pod \"e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm\" (UID: \"f1e164ab-e940-44e0-8829-8626e4162a9f\") " pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" Nov 01 16:06:46.506122 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:46.506107 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1e164ab-e940-44e0-8829-8626e4162a9f-tmp-dir\") pod \"e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm\" (UID: \"f1e164ab-e940-44e0-8829-8626e4162a9f\") " pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" Nov 01 16:06:46.506352 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:46.506174 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1e164ab-e940-44e0-8829-8626e4162a9f-home\") pod \"e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm\" (UID: \"f1e164ab-e940-44e0-8829-8626e4162a9f\") " pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" Nov 01 16:06:46.506936 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:06:46.506893 4363 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6948-e44af9\" is forbidden: User \"system:node:ip-10-0-128-238.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-238.us-east-2.compute.internal' and this object" podUID="3acc0ff2-ddc0-4b94-8d50-445d2c4d6d47" pod="opendatahub/test-curl-6948-e44af9" Nov 01 16:06:46.507074 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:46.506937 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm"] Nov 01 16:06:46.608094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:46.608058 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1e164ab-e940-44e0-8829-8626e4162a9f-home\") pod \"e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm\" (UID: \"f1e164ab-e940-44e0-8829-8626e4162a9f\") " pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" Nov 01 16:06:46.608273 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:46.608130 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1e164ab-e940-44e0-8829-8626e4162a9f-dshm\") pod \"e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm\" (UID: \"f1e164ab-e940-44e0-8829-8626e4162a9f\") " pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" Nov 01 16:06:46.608351 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:46.608271 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1e164ab-e940-44e0-8829-8626e4162a9f-model-cache\") pod \"e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm\" (UID: \"f1e164ab-e940-44e0-8829-8626e4162a9f\") " pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" Nov 01 16:06:46.608412 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:46.608347 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqxgk\" (UniqueName: \"kubernetes.io/projected/f1e164ab-e940-44e0-8829-8626e4162a9f-kube-api-access-wqxgk\") pod \"e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm\" (UID: \"f1e164ab-e940-44e0-8829-8626e4162a9f\") " pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" Nov 01 16:06:46.608535 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:46.608512 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1e164ab-e940-44e0-8829-8626e4162a9f-home\") pod \"e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm\" (UID: \"f1e164ab-e940-44e0-8829-8626e4162a9f\") " pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" Nov 01 16:06:46.608622 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:46.608606 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1e164ab-e940-44e0-8829-8626e4162a9f-tls-certs\") pod \"e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm\" (UID: \"f1e164ab-e940-44e0-8829-8626e4162a9f\") " pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" Nov 01 16:06:46.608682 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:46.608639 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1e164ab-e940-44e0-8829-8626e4162a9f-tmp-dir\") pod \"e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm\" (UID: \"f1e164ab-e940-44e0-8829-8626e4162a9f\") " pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" Nov 01 16:06:46.608682 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:46.608607 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1e164ab-e940-44e0-8829-8626e4162a9f-model-cache\") pod \"e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm\" (UID: \"f1e164ab-e940-44e0-8829-8626e4162a9f\") " pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" Nov 01 16:06:46.609010 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:46.608990 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1e164ab-e940-44e0-8829-8626e4162a9f-tmp-dir\") pod \"e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm\" (UID: \"f1e164ab-e940-44e0-8829-8626e4162a9f\") " pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" Nov 01 16:06:46.610395 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:46.610372 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1e164ab-e940-44e0-8829-8626e4162a9f-dshm\") pod \"e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm\" (UID: \"f1e164ab-e940-44e0-8829-8626e4162a9f\") " pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" Nov 01 16:06:46.610832 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:46.610812 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1e164ab-e940-44e0-8829-8626e4162a9f-tls-certs\") pod \"e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm\" (UID: \"f1e164ab-e940-44e0-8829-8626e4162a9f\") " pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" Nov 01 16:06:46.618410 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:46.618386 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqxgk\" (UniqueName: \"kubernetes.io/projected/f1e164ab-e940-44e0-8829-8626e4162a9f-kube-api-access-wqxgk\") pod \"e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm\" (UID: \"f1e164ab-e940-44e0-8829-8626e4162a9f\") " pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" Nov 01 16:06:46.816493 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:46.816403 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" Nov 01 16:06:46.825043 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:06:46.823222 4363 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6948-e44af9\" is forbidden: User \"system:node:ip-10-0-128-238.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-238.us-east-2.compute.internal' and this object" podUID="3acc0ff2-ddc0-4b94-8d50-445d2c4d6d47" pod="opendatahub/test-curl-6948-e44af9" Nov 01 16:06:47.360976 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:47.360937 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm"] Nov 01 16:06:47.375395 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:47.374149 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6948-49e0a8" podUID="9f3a2621-6cbb-4ae1-88bc-61873eb0d0a2" containerName="test-curl-6948-49e0a8" containerID="cri-o://e451703741aebcba77e97d040caada46265b6d56b84da9aa9553e3c1713f32e9" gracePeriod=2 Nov 01 16:06:48.322619 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:06:48.322582 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:06:48.329682 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:48.329650 4363 scope.go:122] "RemoveContainer" containerID="b62b87e7fcb3db436f1f4cbc2f2f4e0111d76446032e68d0a55af9436b3e684f" Nov 01 16:06:48.344480 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:48.344450 4363 scope.go:122] "RemoveContainer" containerID="88837088c4265258d66c1f2f7290b09f331038498e321d7d5338ad944c842feb" Nov 01 16:06:48.354640 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:48.354616 4363 scope.go:122] "RemoveContainer" containerID="5c3dbb67b4561eb4c4cab0ae26bea63f3337df73d3f31fbed4da387cf75b7ba1" Nov 01 16:06:48.386355 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:48.386315 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" event={"ID":"f1e164ab-e940-44e0-8829-8626e4162a9f","Type":"ContainerStarted","Data":"339bacc523e4532427358aaa627c777e82f8d61627e8b3613a97ccbf1fd9ab02"} Nov 01 16:06:48.386743 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:48.386382 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" event={"ID":"f1e164ab-e940-44e0-8829-8626e4162a9f","Type":"ContainerStarted","Data":"c9bf3fc3b589a648148d34b922917bfd80ec2b50bfbe5082985d8a4bcf403b7f"} Nov 01 16:06:48.408897 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:48.408840 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" podStartSLOduration=2.408821668 podStartE2EDuration="2.408821668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:06:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:06:48.403210689 +0000 UTC m=+1141.734948353" watchObservedRunningTime="2026-11-01 16:06:48.408821668 +0000 UTC m=+1141.740559319" Nov 01 16:06:48.821697 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:48.821665 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" Nov 01 16:06:49.646488 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:49.646388 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6948-49e0a8" Nov 01 16:06:49.767042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:49.755438 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f3a2621-6cbb-4ae1-88bc-61873eb0d0a2-kube-api-access-g4lgr\" (UniqueName: \"kubernetes.io/projected/9f3a2621-6cbb-4ae1-88bc-61873eb0d0a2-kube-api-access-g4lgr\") pod \"9f3a2621-6cbb-4ae1-88bc-61873eb0d0a2\" (UID: \"9f3a2621-6cbb-4ae1-88bc-61873eb0d0a2\") " Nov 01 16:06:49.817248 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:49.813513 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f3a2621-6cbb-4ae1-88bc-61873eb0d0a2-kube-api-access-g4lgr" pod "9f3a2621-6cbb-4ae1-88bc-61873eb0d0a2" (UID: "9f3a2621-6cbb-4ae1-88bc-61873eb0d0a2"). InnerVolumeSpecName "kube-api-access-g4lgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:06:49.862041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:49.861060 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4lgr\" (UniqueName: \"kubernetes.io/projected/9f3a2621-6cbb-4ae1-88bc-61873eb0d0a2-kube-api-access-g4lgr\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:06:50.480413 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:50.480373 4363 generic.go:356] "Generic (PLEG): container finished" podID="9f3a2621-6cbb-4ae1-88bc-61873eb0d0a2" containerID="e451703741aebcba77e97d040caada46265b6d56b84da9aa9553e3c1713f32e9" exitCode=137 Nov 01 16:06:50.480595 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:50.480446 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6948-49e0a8" Nov 01 16:06:50.480595 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:50.480457 4363 scope.go:122] "RemoveContainer" containerID="e451703741aebcba77e97d040caada46265b6d56b84da9aa9553e3c1713f32e9" Nov 01 16:06:50.493083 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:50.493010 4363 scope.go:122] "RemoveContainer" containerID="e451703741aebcba77e97d040caada46265b6d56b84da9aa9553e3c1713f32e9" Nov 01 16:06:50.493459 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:06:50.493430 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e451703741aebcba77e97d040caada46265b6d56b84da9aa9553e3c1713f32e9\": container with ID starting with e451703741aebcba77e97d040caada46265b6d56b84da9aa9553e3c1713f32e9 not found: ID does not exist" containerID="e451703741aebcba77e97d040caada46265b6d56b84da9aa9553e3c1713f32e9" Nov 01 16:06:50.493551 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:50.493463 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e451703741aebcba77e97d040caada46265b6d56b84da9aa9553e3c1713f32e9"} err="failed to get container status \"e451703741aebcba77e97d040caada46265b6d56b84da9aa9553e3c1713f32e9\": rpc error: code = NotFound desc = could not find container \"e451703741aebcba77e97d040caada46265b6d56b84da9aa9553e3c1713f32e9\": container with ID starting with e451703741aebcba77e97d040caada46265b6d56b84da9aa9553e3c1713f32e9 not found: ID does not exist" Nov 01 16:06:50.493861 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:06:50.493838 4363 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6948-49e0a8\" is forbidden: User \"system:node:ip-10-0-128-238.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-238.us-east-2.compute.internal' and this object" podUID="9f3a2621-6cbb-4ae1-88bc-61873eb0d0a2" pod="opendatahub/test-curl-6948-49e0a8" Nov 01 16:06:50.867731 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:50.867691 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f3a2621-6cbb-4ae1-88bc-61873eb0d0a2" path="/var/lib/kubelet/pods/9f3a2621-6cbb-4ae1-88bc-61873eb0d0a2/volumes" Nov 01 16:06:55.918405 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:55.918253 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-61bb54-84479f79f5-rc5cd"] Nov 01 16:06:55.928476 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:55.927404 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-61bb54-84479f79f5-rc5cd" Nov 01 16:06:55.933095 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:55.931794 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-61bb54-84479f79f5-rc5cd"] Nov 01 16:06:56.020121 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:56.010556 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4hxp\" (UniqueName: \"kubernetes.io/projected/c7434b5f-5698-447d-a93a-41ffcbe77f9b-kube-api-access-n4hxp\") pod \"payload-pre-processing-e2e-api-a-61bb54-84479f79f5-rc5cd\" (UID: \"c7434b5f-5698-447d-a93a-41ffcbe77f9b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-61bb54-84479f79f5-rc5cd" Nov 01 16:06:56.020121 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:56.010739 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7434b5f-5698-447d-a93a-41ffcbe77f9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-61bb54-84479f79f5-rc5cd\" (UID: \"c7434b5f-5698-447d-a93a-41ffcbe77f9b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-61bb54-84479f79f5-rc5cd" Nov 01 16:06:56.020121 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:56.011119 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-61bb54\"" Nov 01 16:06:56.020121 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:56.011260 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-61bb54-dockercfg-stwmb\"" Nov 01 16:06:56.123043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:56.117581 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7434b5f-5698-447d-a93a-41ffcbe77f9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-61bb54-84479f79f5-rc5cd\" (UID: \"c7434b5f-5698-447d-a93a-41ffcbe77f9b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-61bb54-84479f79f5-rc5cd" Nov 01 16:06:56.123043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:56.118110 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4hxp\" (UniqueName: \"kubernetes.io/projected/c7434b5f-5698-447d-a93a-41ffcbe77f9b-kube-api-access-n4hxp\") pod \"payload-pre-processing-e2e-api-a-61bb54-84479f79f5-rc5cd\" (UID: \"c7434b5f-5698-447d-a93a-41ffcbe77f9b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-61bb54-84479f79f5-rc5cd" Nov 01 16:06:56.123043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:56.119492 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7434b5f-5698-447d-a93a-41ffcbe77f9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-61bb54-84479f79f5-rc5cd\" (UID: \"c7434b5f-5698-447d-a93a-41ffcbe77f9b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-61bb54-84479f79f5-rc5cd" Nov 01 16:06:56.154053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:56.149739 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4hxp\" (UniqueName: \"kubernetes.io/projected/c7434b5f-5698-447d-a93a-41ffcbe77f9b-kube-api-access-n4hxp\") pod \"payload-pre-processing-e2e-api-a-61bb54-84479f79f5-rc5cd\" (UID: \"c7434b5f-5698-447d-a93a-41ffcbe77f9b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-61bb54-84479f79f5-rc5cd" Nov 01 16:06:56.256172 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:56.256064 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-61bb54-84479f79f5-rc5cd" Nov 01 16:06:56.376098 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:56.369227 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f47890-84fdfb9c59-fs5nn" Nov 01 16:06:56.821571 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:56.821431 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" Nov 01 16:06:56.838242 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:56.838057 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" Nov 01 16:06:56.926149 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:06:56.925538 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7434b5f_5698_447d_a93a_41ffcbe77f9b.slice/crio-504b4f8729c9dbae255fc33e7a0b8eb240dcbdf586d6388020120c2da48f69e4 WatchSource:0}: Error finding container 504b4f8729c9dbae255fc33e7a0b8eb240dcbdf586d6388020120c2da48f69e4: Status 404 returned error can't find the container with id 504b4f8729c9dbae255fc33e7a0b8eb240dcbdf586d6388020120c2da48f69e4 Nov 01 16:06:56.926149 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:56.925987 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-61bb54-84479f79f5-rc5cd"] Nov 01 16:06:57.769560 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:57.769524 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-61bb54-84479f79f5-rc5cd" event={"ID":"c7434b5f-5698-447d-a93a-41ffcbe77f9b","Type":"ContainerStarted","Data":"bebd0e2654355afd01582c29307e860c3912fa71aa9a8491ad376d1781ef8e98"} Nov 01 16:06:57.769759 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:57.769622 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-61bb54-84479f79f5-rc5cd" Nov 01 16:06:57.769759 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:57.769646 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-61bb54-84479f79f5-rc5cd" event={"ID":"c7434b5f-5698-447d-a93a-41ffcbe77f9b","Type":"ContainerStarted","Data":"504b4f8729c9dbae255fc33e7a0b8eb240dcbdf586d6388020120c2da48f69e4"} Nov 01 16:06:57.783401 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:57.783371 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" Nov 01 16:06:57.784392 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:06:57.784346 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-61bb54-84479f79f5-rc5cd" podStartSLOduration=2.784331927 podStartE2EDuration="2.784331927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:06:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:06:57.783351219 +0000 UTC m=+1151.115088871" watchObservedRunningTime="2026-11-01 16:06:57.784331927 +0000 UTC m=+1151.116069620" Nov 01 16:07:05.813111 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:05.803149 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" podUID="f1e164ab-e940-44e0-8829-8626e4162a9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.115:8000/ready\": context deadline exceeded" Nov 01 16:07:05.978042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:05.975122 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:06.004247 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:06.001393 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:06.151463 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:06.151418 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:07:06.586101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:06.585843 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": context deadline exceeded" Nov 01 16:07:06.587276 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:06.586300 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:06.734965 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:06.732368 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:07:06.769042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:06.760163 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:06.843225 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:06.843186 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" podUID="f1e164ab-e940-44e0-8829-8626e4162a9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.115:8000/ready\": context deadline exceeded" Nov 01 16:07:06.879259 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:06.871242 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" Nov 01 16:07:07.116116 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:07.115986 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 01 16:07:07.116405 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:07.116371 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:07:07.196730 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:07.196680 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:07:07.358080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:07.358050 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 16:07:07.394105 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:07.388030 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:07:07.394105 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:07.388102 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:07:07.591948 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:07.591902 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:07.592855 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:07.592752 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:07:07.626354 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:07.626306 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:07:07.671980 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:07.671900 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:07:07.736324 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:07.736285 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:07.738200 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:07.738174 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 16:07:07.862037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:07.851640 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" podUID="f1e164ab-e940-44e0-8829-8626e4162a9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:07.901996 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:07.901958 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" podUID="f1e164ab-e940-44e0-8829-8626e4162a9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.115:8000/health\": context deadline exceeded" Nov 01 16:07:07.985127 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:07.985010 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 16:07:08.146235 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:08.145904 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:08.152762 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:08.152713 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:08.210654 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:08.210609 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:07:08.210848 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:08.210694 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" Nov 01 16:07:08.232154 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:08.232084 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 16:07:08.830461 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:08.830426 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-61bb54-84479f79f5-rc5cd" Nov 01 16:07:10.246660 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:10.238616 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-61bb54-7b8559df57-hkdqs"] Nov 01 16:07:10.369039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:10.359376 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61bb54-7b8559df57-hkdqs" Nov 01 16:07:10.405386 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:10.405347 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-61bb54-serving-cert\"" Nov 01 16:07:10.425227 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:10.411652 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-61bb54-7b8559df57-hkdqs"] Nov 01 16:07:10.519347 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:10.519037 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eac30ab3-746f-430c-864b-6b469263e9e1-metrics-certs\") pod \"maas-api-e2e-api-a-61bb54-7b8559df57-hkdqs\" (UID: \"eac30ab3-746f-430c-864b-6b469263e9e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61bb54-7b8559df57-hkdqs" Nov 01 16:07:10.519704 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:10.519679 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eac30ab3-746f-430c-864b-6b469263e9e1-maas-api-tls\") pod \"maas-api-e2e-api-a-61bb54-7b8559df57-hkdqs\" (UID: \"eac30ab3-746f-430c-864b-6b469263e9e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61bb54-7b8559df57-hkdqs" Nov 01 16:07:10.520059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:10.520038 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eac30ab3-746f-430c-864b-6b469263e9e1-openshift-service-ca\") pod \"maas-api-e2e-api-a-61bb54-7b8559df57-hkdqs\" (UID: \"eac30ab3-746f-430c-864b-6b469263e9e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61bb54-7b8559df57-hkdqs" Nov 01 16:07:10.520257 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:10.520241 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9x4n\" (UniqueName: \"kubernetes.io/projected/eac30ab3-746f-430c-864b-6b469263e9e1-kube-api-access-q9x4n\") pod \"maas-api-e2e-api-a-61bb54-7b8559df57-hkdqs\" (UID: \"eac30ab3-746f-430c-864b-6b469263e9e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61bb54-7b8559df57-hkdqs" Nov 01 16:07:10.622526 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:10.622305 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eac30ab3-746f-430c-864b-6b469263e9e1-openshift-service-ca\") pod \"maas-api-e2e-api-a-61bb54-7b8559df57-hkdqs\" (UID: \"eac30ab3-746f-430c-864b-6b469263e9e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61bb54-7b8559df57-hkdqs" Nov 01 16:07:10.622751 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:10.622696 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9x4n\" (UniqueName: \"kubernetes.io/projected/eac30ab3-746f-430c-864b-6b469263e9e1-kube-api-access-q9x4n\") pod \"maas-api-e2e-api-a-61bb54-7b8559df57-hkdqs\" (UID: \"eac30ab3-746f-430c-864b-6b469263e9e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61bb54-7b8559df57-hkdqs" Nov 01 16:07:10.622847 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:10.622823 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eac30ab3-746f-430c-864b-6b469263e9e1-metrics-certs\") pod \"maas-api-e2e-api-a-61bb54-7b8559df57-hkdqs\" (UID: \"eac30ab3-746f-430c-864b-6b469263e9e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61bb54-7b8559df57-hkdqs" Nov 01 16:07:10.622954 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:10.622937 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eac30ab3-746f-430c-864b-6b469263e9e1-maas-api-tls\") pod \"maas-api-e2e-api-a-61bb54-7b8559df57-hkdqs\" (UID: \"eac30ab3-746f-430c-864b-6b469263e9e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61bb54-7b8559df57-hkdqs" Nov 01 16:07:10.623773 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:10.623730 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eac30ab3-746f-430c-864b-6b469263e9e1-openshift-service-ca\") pod \"maas-api-e2e-api-a-61bb54-7b8559df57-hkdqs\" (UID: \"eac30ab3-746f-430c-864b-6b469263e9e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61bb54-7b8559df57-hkdqs" Nov 01 16:07:10.643623 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:10.643585 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eac30ab3-746f-430c-864b-6b469263e9e1-metrics-certs\") pod \"maas-api-e2e-api-a-61bb54-7b8559df57-hkdqs\" (UID: \"eac30ab3-746f-430c-864b-6b469263e9e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61bb54-7b8559df57-hkdqs" Nov 01 16:07:10.643968 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:10.643949 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eac30ab3-746f-430c-864b-6b469263e9e1-maas-api-tls\") pod \"maas-api-e2e-api-a-61bb54-7b8559df57-hkdqs\" (UID: \"eac30ab3-746f-430c-864b-6b469263e9e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61bb54-7b8559df57-hkdqs" Nov 01 16:07:10.644329 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:10.644308 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9x4n\" (UniqueName: \"kubernetes.io/projected/eac30ab3-746f-430c-864b-6b469263e9e1-kube-api-access-q9x4n\") pod \"maas-api-e2e-api-a-61bb54-7b8559df57-hkdqs\" (UID: \"eac30ab3-746f-430c-864b-6b469263e9e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61bb54-7b8559df57-hkdqs" Nov 01 16:07:10.733426 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:10.733388 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61bb54-7b8559df57-hkdqs" Nov 01 16:07:11.390184 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:11.390139 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-61bb54-7b8559df57-hkdqs"] Nov 01 16:07:11.480360 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:11.480320 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61bb54-7b8559df57-hkdqs" event={"ID":"eac30ab3-746f-430c-864b-6b469263e9e1","Type":"ContainerStarted","Data":"a9d42f6cfdce7aef8c70595a200f85a1611e8d58be23a10bf6ac0af7cec9679f"} Nov 01 16:07:11.964535 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:11.964507 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-734859-5fc5fb544f-n2j8r"] Nov 01 16:07:11.970757 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:11.970732 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-734859-5fc5fb544f-n2j8r" Nov 01 16:07:11.976536 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:11.976509 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-734859-dockercfg-7vgv6\"" Nov 01 16:07:11.976690 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:11.976572 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-734859\"" Nov 01 16:07:11.979343 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:11.979315 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-734859-5fc5fb544f-n2j8r"] Nov 01 16:07:12.006286 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:12.006256 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-734859-589cbc787-xqhjc"] Nov 01 16:07:12.014734 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:12.014705 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-734859-589cbc787-xqhjc" Nov 01 16:07:12.020603 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:12.020562 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-734859-589cbc787-xqhjc"] Nov 01 16:07:12.072552 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:12.071628 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2970538a-8b72-47f6-a76b-455f828c8d83-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-734859-5fc5fb544f-n2j8r\" (UID: \"2970538a-8b72-47f6-a76b-455f828c8d83\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-734859-5fc5fb544f-n2j8r" Nov 01 16:07:12.072552 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:12.071834 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcbmq\" (UniqueName: \"kubernetes.io/projected/2970538a-8b72-47f6-a76b-455f828c8d83-kube-api-access-bcbmq\") pod \"payload-processing-e2e-shared-b-w3-734859-5fc5fb544f-n2j8r\" (UID: \"2970538a-8b72-47f6-a76b-455f828c8d83\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-734859-5fc5fb544f-n2j8r" Nov 01 16:07:12.072552 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:12.071991 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c56804c0-0810-4106-b393-d292cb816780-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-734859-589cbc787-xqhjc\" (UID: \"c56804c0-0810-4106-b393-d292cb816780\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-734859-589cbc787-xqhjc" Nov 01 16:07:12.072552 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:12.072284 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vwds\" (UniqueName: \"kubernetes.io/projected/c56804c0-0810-4106-b393-d292cb816780-kube-api-access-5vwds\") pod \"payload-pre-processing-e2e-shared-b-w3-734859-589cbc787-xqhjc\" (UID: \"c56804c0-0810-4106-b393-d292cb816780\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-734859-589cbc787-xqhjc" Nov 01 16:07:12.175555 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:12.175508 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vwds\" (UniqueName: \"kubernetes.io/projected/c56804c0-0810-4106-b393-d292cb816780-kube-api-access-5vwds\") pod \"payload-pre-processing-e2e-shared-b-w3-734859-589cbc787-xqhjc\" (UID: \"c56804c0-0810-4106-b393-d292cb816780\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-734859-589cbc787-xqhjc" Nov 01 16:07:12.176341 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:12.176307 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2970538a-8b72-47f6-a76b-455f828c8d83-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-734859-5fc5fb544f-n2j8r\" (UID: \"2970538a-8b72-47f6-a76b-455f828c8d83\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-734859-5fc5fb544f-n2j8r" Nov 01 16:07:12.176639 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:12.176619 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcbmq\" (UniqueName: \"kubernetes.io/projected/2970538a-8b72-47f6-a76b-455f828c8d83-kube-api-access-bcbmq\") pod \"payload-processing-e2e-shared-b-w3-734859-5fc5fb544f-n2j8r\" (UID: \"2970538a-8b72-47f6-a76b-455f828c8d83\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-734859-5fc5fb544f-n2j8r" Nov 01 16:07:12.203239 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:12.203145 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c56804c0-0810-4106-b393-d292cb816780-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-734859-589cbc787-xqhjc\" (UID: \"c56804c0-0810-4106-b393-d292cb816780\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-734859-589cbc787-xqhjc" Nov 01 16:07:12.203630 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:12.180253 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2970538a-8b72-47f6-a76b-455f828c8d83-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-734859-5fc5fb544f-n2j8r\" (UID: \"2970538a-8b72-47f6-a76b-455f828c8d83\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-734859-5fc5fb544f-n2j8r" Nov 01 16:07:12.204110 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:12.204052 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c56804c0-0810-4106-b393-d292cb816780-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-734859-589cbc787-xqhjc\" (UID: \"c56804c0-0810-4106-b393-d292cb816780\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-734859-589cbc787-xqhjc" Nov 01 16:07:12.223206 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:12.223176 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcbmq\" (UniqueName: \"kubernetes.io/projected/2970538a-8b72-47f6-a76b-455f828c8d83-kube-api-access-bcbmq\") pod \"payload-processing-e2e-shared-b-w3-734859-5fc5fb544f-n2j8r\" (UID: \"2970538a-8b72-47f6-a76b-455f828c8d83\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-734859-5fc5fb544f-n2j8r" Nov 01 16:07:12.224762 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:12.224706 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vwds\" (UniqueName: \"kubernetes.io/projected/c56804c0-0810-4106-b393-d292cb816780-kube-api-access-5vwds\") pod \"payload-pre-processing-e2e-shared-b-w3-734859-589cbc787-xqhjc\" (UID: \"c56804c0-0810-4106-b393-d292cb816780\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-734859-589cbc787-xqhjc" Nov 01 16:07:12.341743 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:12.341702 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-734859-5fc5fb544f-n2j8r" Nov 01 16:07:12.345921 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:12.345795 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-734859-589cbc787-xqhjc" Nov 01 16:07:12.675592 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:12.675559 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61bb54-7b8559df57-hkdqs" event={"ID":"eac30ab3-746f-430c-864b-6b469263e9e1","Type":"ContainerStarted","Data":"cf1d129fd09a543b233ed3ec2be6a52341ab9ead21272a6d884ca4904d869c32"} Nov 01 16:07:12.677455 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:12.677434 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61bb54-7b8559df57-hkdqs" Nov 01 16:07:12.706105 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:12.705916 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61bb54-7b8559df57-hkdqs" podStartSLOduration=2.166296547 podStartE2EDuration="2.70589856s" totalImagesPullingTime="539.602013ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:07:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:07:12.701137263 +0000 UTC m=+1166.032874912" watchObservedRunningTime="2026-11-01 16:07:12.70589856 +0000 UTC m=+1166.037636216" Nov 01 16:07:13.373040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:13.368308 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-734859-5fc5fb544f-n2j8r"] Nov 01 16:07:13.528072 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:13.527517 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-734859-589cbc787-xqhjc"] Nov 01 16:07:13.689741 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:13.689714 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-734859-589cbc787-xqhjc" event={"ID":"c56804c0-0810-4106-b393-d292cb816780","Type":"ContainerStarted","Data":"efd65cc2489c9b60beb7c024858ceb8e39193fd2dca46820a60aab3f668f91e7"} Nov 01 16:07:13.691886 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:13.691857 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-734859-5fc5fb544f-n2j8r" event={"ID":"2970538a-8b72-47f6-a76b-455f828c8d83","Type":"ContainerStarted","Data":"9a2cfd0a9e4a231492630b9e50963f0372fd1dd18284b3f34d3af9e267f00136"} Nov 01 16:07:13.691999 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:13.691935 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-734859-5fc5fb544f-n2j8r" Nov 01 16:07:13.691999 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:13.691967 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-734859-5fc5fb544f-n2j8r" event={"ID":"2970538a-8b72-47f6-a76b-455f828c8d83","Type":"ContainerStarted","Data":"caa805cc66c2359cb820fe84b4b8b735b78a94a5f87ced028190cc21740bbc69"} Nov 01 16:07:13.719244 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:13.719172 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-734859-5fc5fb544f-n2j8r" podStartSLOduration=2.719156903 podStartE2EDuration="2.719156903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:07:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:07:13.713353208 +0000 UTC m=+1167.045090870" watchObservedRunningTime="2026-11-01 16:07:13.719156903 +0000 UTC m=+1167.050894583" Nov 01 16:07:14.703177 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:14.703137 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-734859-589cbc787-xqhjc" event={"ID":"c56804c0-0810-4106-b393-d292cb816780","Type":"ContainerStarted","Data":"26a7ab71a9f9ca0a81874a278ac144ca1c17b3cb9a8250783630073d54b2e1cd"} Nov 01 16:07:14.703635 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:14.703235 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-734859-589cbc787-xqhjc" Nov 01 16:07:14.721523 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:14.721472 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-734859-589cbc787-xqhjc" podStartSLOduration=3.721457059 podStartE2EDuration="3.721457059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:07:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:07:14.718094214 +0000 UTC m=+1168.049831864" watchObservedRunningTime="2026-11-01 16:07:14.721457059 +0000 UTC m=+1168.053194710" Nov 01 16:07:14.842812 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:14.842778 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-9a482b-86b559b58f-frnp7"] Nov 01 16:07:14.847999 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:14.847977 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9a482b-86b559b58f-frnp7" Nov 01 16:07:14.852999 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:14.851814 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-9a482b\"" Nov 01 16:07:14.852999 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:14.852947 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-9a482b-dockercfg-gdxr8\"" Nov 01 16:07:14.863432 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:14.858687 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-9a482b-86b559b58f-frnp7"] Nov 01 16:07:14.864773 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:14.864751 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a482b-79b5ff7995-5zhdr"] Nov 01 16:07:14.875708 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:14.875011 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a482b-79b5ff7995-5zhdr" Nov 01 16:07:14.883173 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:14.883141 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a482b-79b5ff7995-5zhdr"] Nov 01 16:07:14.994214 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:14.981867 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdxcw\" (UniqueName: \"kubernetes.io/projected/b8b1f822-bf88-4f11-b621-da764f27c492-kube-api-access-fdxcw\") pod \"payload-pre-processing-e2e-shared-a-w4-9a482b-79b5ff7995-5zhdr\" (UID: \"b8b1f822-bf88-4f11-b621-da764f27c492\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a482b-79b5ff7995-5zhdr" Nov 01 16:07:14.994214 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:14.982011 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e533e983-dbc0-4ce2-8d58-c74c0c0d857b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-9a482b-86b559b58f-frnp7\" (UID: \"e533e983-dbc0-4ce2-8d58-c74c0c0d857b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9a482b-86b559b58f-frnp7" Nov 01 16:07:14.994214 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:14.982367 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz9qc\" (UniqueName: \"kubernetes.io/projected/e533e983-dbc0-4ce2-8d58-c74c0c0d857b-kube-api-access-kz9qc\") pod \"payload-processing-e2e-shared-a-w4-9a482b-86b559b58f-frnp7\" (UID: \"e533e983-dbc0-4ce2-8d58-c74c0c0d857b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9a482b-86b559b58f-frnp7" Nov 01 16:07:14.994214 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:14.982419 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8b1f822-bf88-4f11-b621-da764f27c492-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-9a482b-79b5ff7995-5zhdr\" (UID: \"b8b1f822-bf88-4f11-b621-da764f27c492\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a482b-79b5ff7995-5zhdr" Nov 01 16:07:15.199421 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:15.129345 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz9qc\" (UniqueName: \"kubernetes.io/projected/e533e983-dbc0-4ce2-8d58-c74c0c0d857b-kube-api-access-kz9qc\") pod \"payload-processing-e2e-shared-a-w4-9a482b-86b559b58f-frnp7\" (UID: \"e533e983-dbc0-4ce2-8d58-c74c0c0d857b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9a482b-86b559b58f-frnp7" Nov 01 16:07:15.199740 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:15.199706 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8b1f822-bf88-4f11-b621-da764f27c492-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-9a482b-79b5ff7995-5zhdr\" (UID: \"b8b1f822-bf88-4f11-b621-da764f27c492\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a482b-79b5ff7995-5zhdr" Nov 01 16:07:15.200149 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:15.200123 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdxcw\" (UniqueName: \"kubernetes.io/projected/b8b1f822-bf88-4f11-b621-da764f27c492-kube-api-access-fdxcw\") pod \"payload-pre-processing-e2e-shared-a-w4-9a482b-79b5ff7995-5zhdr\" (UID: \"b8b1f822-bf88-4f11-b621-da764f27c492\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a482b-79b5ff7995-5zhdr" Nov 01 16:07:15.200422 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:15.200397 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e533e983-dbc0-4ce2-8d58-c74c0c0d857b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-9a482b-86b559b58f-frnp7\" (UID: \"e533e983-dbc0-4ce2-8d58-c74c0c0d857b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9a482b-86b559b58f-frnp7" Nov 01 16:07:15.201633 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:15.201603 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8b1f822-bf88-4f11-b621-da764f27c492-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-9a482b-79b5ff7995-5zhdr\" (UID: \"b8b1f822-bf88-4f11-b621-da764f27c492\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a482b-79b5ff7995-5zhdr" Nov 01 16:07:15.202036 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:15.202000 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e533e983-dbc0-4ce2-8d58-c74c0c0d857b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-9a482b-86b559b58f-frnp7\" (UID: \"e533e983-dbc0-4ce2-8d58-c74c0c0d857b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9a482b-86b559b58f-frnp7" Nov 01 16:07:15.266170 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:15.254643 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz9qc\" (UniqueName: \"kubernetes.io/projected/e533e983-dbc0-4ce2-8d58-c74c0c0d857b-kube-api-access-kz9qc\") pod \"payload-processing-e2e-shared-a-w4-9a482b-86b559b58f-frnp7\" (UID: \"e533e983-dbc0-4ce2-8d58-c74c0c0d857b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9a482b-86b559b58f-frnp7" Nov 01 16:07:15.276749 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:15.267084 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdxcw\" (UniqueName: \"kubernetes.io/projected/b8b1f822-bf88-4f11-b621-da764f27c492-kube-api-access-fdxcw\") pod \"payload-pre-processing-e2e-shared-a-w4-9a482b-79b5ff7995-5zhdr\" (UID: \"b8b1f822-bf88-4f11-b621-da764f27c492\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a482b-79b5ff7995-5zhdr" Nov 01 16:07:15.480037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:15.476425 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9a482b-86b559b58f-frnp7" Nov 01 16:07:15.534218 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:15.524340 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a482b-79b5ff7995-5zhdr" Nov 01 16:07:15.851649 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:15.851605 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-9a482b-86b559b58f-frnp7"] Nov 01 16:07:15.858682 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:07:15.858654 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode533e983_dbc0_4ce2_8d58_c74c0c0d857b.slice/crio-73c29e7379d48e3c35757d692221cdf9b5b6c1e001886745379723ec8bbf399b WatchSource:0}: Error finding container 73c29e7379d48e3c35757d692221cdf9b5b6c1e001886745379723ec8bbf399b: Status 404 returned error can't find the container with id 73c29e7379d48e3c35757d692221cdf9b5b6c1e001886745379723ec8bbf399b Nov 01 16:07:15.874173 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:15.874055 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a482b-79b5ff7995-5zhdr"] Nov 01 16:07:15.874308 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:07:15.874228 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8b1f822_bf88_4f11_b621_da764f27c492.slice/crio-8f12c47aeb5420c6e680e46f1c54a53e246982b87a43d51fae1652e7ba568081 WatchSource:0}: Error finding container 8f12c47aeb5420c6e680e46f1c54a53e246982b87a43d51fae1652e7ba568081: Status 404 returned error can't find the container with id 8f12c47aeb5420c6e680e46f1c54a53e246982b87a43d51fae1652e7ba568081 Nov 01 16:07:16.771232 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:16.771191 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9a482b-86b559b58f-frnp7" event={"ID":"e533e983-dbc0-4ce2-8d58-c74c0c0d857b","Type":"ContainerStarted","Data":"7b9c924a8440d30cff88c1b74062462a018d785e9e5fa51b35dc3e3c2b251e2a"} Nov 01 16:07:16.771509 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:16.771490 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9a482b-86b559b58f-frnp7" event={"ID":"e533e983-dbc0-4ce2-8d58-c74c0c0d857b","Type":"ContainerStarted","Data":"73c29e7379d48e3c35757d692221cdf9b5b6c1e001886745379723ec8bbf399b"} Nov 01 16:07:16.772371 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:16.772292 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9a482b-86b559b58f-frnp7" Nov 01 16:07:16.776565 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:16.776521 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a482b-79b5ff7995-5zhdr" event={"ID":"b8b1f822-bf88-4f11-b621-da764f27c492","Type":"ContainerStarted","Data":"30756e3465ba1fd2331edb2ab06ca3d66dd6dbe07de218a977971bdf4fd5be48"} Nov 01 16:07:16.776751 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:16.776730 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a482b-79b5ff7995-5zhdr" event={"ID":"b8b1f822-bf88-4f11-b621-da764f27c492","Type":"ContainerStarted","Data":"8f12c47aeb5420c6e680e46f1c54a53e246982b87a43d51fae1652e7ba568081"} Nov 01 16:07:16.777639 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:16.777619 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a482b-79b5ff7995-5zhdr" Nov 01 16:07:16.820415 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:16.820352 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a482b-79b5ff7995-5zhdr" podStartSLOduration=2.82033719 podStartE2EDuration="2.82033719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:07:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:07:16.8165731 +0000 UTC m=+1170.148310751" watchObservedRunningTime="2026-11-01 16:07:16.82033719 +0000 UTC m=+1170.152074839" Nov 01 16:07:16.821059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:16.821025 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9a482b-86b559b58f-frnp7" podStartSLOduration=2.821004972 podStartE2EDuration="2.821004972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:07:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:07:16.800238823 +0000 UTC m=+1170.131976477" watchObservedRunningTime="2026-11-01 16:07:16.821004972 +0000 UTC m=+1170.152742625" Nov 01 16:07:18.311482 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:07:18.311440 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:07:18.697945 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:18.697918 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61bb54-7b8559df57-hkdqs" Nov 01 16:07:24.724065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:24.718205 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-734859-5fc5fb544f-n2j8r" Nov 01 16:07:25.719889 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:25.719860 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-734859-589cbc787-xqhjc" Nov 01 16:07:27.789654 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:27.789616 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9a482b-86b559b58f-frnp7" Nov 01 16:07:28.800274 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:28.800245 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a482b-79b5ff7995-5zhdr" Nov 01 16:07:31.186860 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:31.186823 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h"] Nov 01 16:07:31.202829 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:31.202797 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" Nov 01 16:07:31.211033 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:31.210953 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-59d3a7-openshift-default-dockercfg-67nx4\"" Nov 01 16:07:31.225580 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:31.225545 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h"] Nov 01 16:07:31.328415 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:31.328374 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922af45f-b42b-47de-9724-6f2daa958330-istio-data\") pod \"e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h\" (UID: \"922af45f-b42b-47de-9724-6f2daa958330\") " pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" Nov 01 16:07:31.328589 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:31.328505 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922af45f-b42b-47de-9724-6f2daa958330-istiod-ca-cert\") pod \"e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h\" (UID: \"922af45f-b42b-47de-9724-6f2daa958330\") " pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" Nov 01 16:07:31.328589 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:31.328562 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922af45f-b42b-47de-9724-6f2daa958330-credential-socket\") pod \"e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h\" (UID: \"922af45f-b42b-47de-9724-6f2daa958330\") " pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" Nov 01 16:07:31.328743 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:31.328716 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922af45f-b42b-47de-9724-6f2daa958330-istio-token\") pod \"e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h\" (UID: \"922af45f-b42b-47de-9724-6f2daa958330\") " pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" Nov 01 16:07:31.328799 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:31.328781 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922af45f-b42b-47de-9724-6f2daa958330-istio-podinfo\") pod \"e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h\" (UID: \"922af45f-b42b-47de-9724-6f2daa958330\") " pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" Nov 01 16:07:31.328908 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:31.328879 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922af45f-b42b-47de-9724-6f2daa958330-workload-socket\") pod \"e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h\" (UID: \"922af45f-b42b-47de-9724-6f2daa958330\") " pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" Nov 01 16:07:31.329065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:31.328935 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922af45f-b42b-47de-9724-6f2daa958330-workload-certs\") pod \"e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h\" (UID: \"922af45f-b42b-47de-9724-6f2daa958330\") " pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" Nov 01 16:07:31.329065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:31.329053 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9xfm\" (UniqueName: \"kubernetes.io/projected/922af45f-b42b-47de-9724-6f2daa958330-kube-api-access-d9xfm\") pod \"e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h\" (UID: \"922af45f-b42b-47de-9724-6f2daa958330\") " pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" Nov 01 16:07:31.329213 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:31.329195 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922af45f-b42b-47de-9724-6f2daa958330-istio-envoy\") pod \"e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h\" (UID: \"922af45f-b42b-47de-9724-6f2daa958330\") " pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" Nov 01 16:07:31.434103 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:31.433898 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9xfm\" (UniqueName: \"kubernetes.io/projected/922af45f-b42b-47de-9724-6f2daa958330-kube-api-access-d9xfm\") pod \"e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h\" (UID: \"922af45f-b42b-47de-9724-6f2daa958330\") " pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" Nov 01 16:07:31.434103 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:31.434071 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922af45f-b42b-47de-9724-6f2daa958330-istio-envoy\") pod \"e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h\" (UID: \"922af45f-b42b-47de-9724-6f2daa958330\") " pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" Nov 01 16:07:31.434357 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:31.434228 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922af45f-b42b-47de-9724-6f2daa958330-istio-data\") pod \"e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h\" (UID: \"922af45f-b42b-47de-9724-6f2daa958330\") " pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" Nov 01 16:07:31.434357 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:31.434302 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922af45f-b42b-47de-9724-6f2daa958330-istiod-ca-cert\") pod \"e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h\" (UID: \"922af45f-b42b-47de-9724-6f2daa958330\") " pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" Nov 01 16:07:31.434475 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:31.434357 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922af45f-b42b-47de-9724-6f2daa958330-credential-socket\") pod \"e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h\" (UID: \"922af45f-b42b-47de-9724-6f2daa958330\") " pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" Nov 01 16:07:31.434532 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:31.434475 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922af45f-b42b-47de-9724-6f2daa958330-istio-token\") pod \"e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h\" (UID: \"922af45f-b42b-47de-9724-6f2daa958330\") " pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" Nov 01 16:07:31.434532 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:31.434521 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922af45f-b42b-47de-9724-6f2daa958330-istio-podinfo\") pod \"e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h\" (UID: \"922af45f-b42b-47de-9724-6f2daa958330\") " pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" Nov 01 16:07:31.434697 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:31.434578 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922af45f-b42b-47de-9724-6f2daa958330-workload-socket\") pod \"e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h\" (UID: \"922af45f-b42b-47de-9724-6f2daa958330\") " pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" Nov 01 16:07:31.434697 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:31.434624 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922af45f-b42b-47de-9724-6f2daa958330-workload-certs\") pod \"e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h\" (UID: \"922af45f-b42b-47de-9724-6f2daa958330\") " pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" Nov 01 16:07:31.443087 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:31.442247 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922af45f-b42b-47de-9724-6f2daa958330-workload-certs\") pod \"e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h\" (UID: \"922af45f-b42b-47de-9724-6f2daa958330\") " pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" Nov 01 16:07:31.443257 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:31.443173 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922af45f-b42b-47de-9724-6f2daa958330-credential-socket\") pod \"e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h\" (UID: \"922af45f-b42b-47de-9724-6f2daa958330\") " pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" Nov 01 16:07:31.478818 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:31.443496 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922af45f-b42b-47de-9724-6f2daa958330-istio-data\") pod \"e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h\" (UID: \"922af45f-b42b-47de-9724-6f2daa958330\") " pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" Nov 01 16:07:31.478818 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:31.455986 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922af45f-b42b-47de-9724-6f2daa958330-istiod-ca-cert\") pod \"e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h\" (UID: \"922af45f-b42b-47de-9724-6f2daa958330\") " pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" Nov 01 16:07:31.478818 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:31.457130 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922af45f-b42b-47de-9724-6f2daa958330-workload-socket\") pod \"e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h\" (UID: \"922af45f-b42b-47de-9724-6f2daa958330\") " pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" Nov 01 16:07:31.490057 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:31.485191 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922af45f-b42b-47de-9724-6f2daa958330-istio-podinfo\") pod \"e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h\" (UID: \"922af45f-b42b-47de-9724-6f2daa958330\") " pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" Nov 01 16:07:31.490057 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:31.486585 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922af45f-b42b-47de-9724-6f2daa958330-istio-envoy\") pod \"e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h\" (UID: \"922af45f-b42b-47de-9724-6f2daa958330\") " pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" Nov 01 16:07:31.517052 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:31.515808 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922af45f-b42b-47de-9724-6f2daa958330-istio-token\") pod \"e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h\" (UID: \"922af45f-b42b-47de-9724-6f2daa958330\") " pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" Nov 01 16:07:31.551040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:31.545238 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9xfm\" (UniqueName: \"kubernetes.io/projected/922af45f-b42b-47de-9724-6f2daa958330-kube-api-access-d9xfm\") pod \"e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h\" (UID: \"922af45f-b42b-47de-9724-6f2daa958330\") " pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" Nov 01 16:07:31.834996 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:31.834843 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" Nov 01 16:07:32.348483 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:32.346994 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h"] Nov 01 16:07:32.349457 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:07:32.349432 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod922af45f_b42b_47de_9724_6f2daa958330.slice/crio-475202ef8ebefb883fddf72d8adc33af5dddef5af686ecb3da3eb9c14bd198e1 WatchSource:0}: Error finding container 475202ef8ebefb883fddf72d8adc33af5dddef5af686ecb3da3eb9c14bd198e1: Status 404 returned error can't find the container with id 475202ef8ebefb883fddf72d8adc33af5dddef5af686ecb3da3eb9c14bd198e1 Nov 01 16:07:32.353345 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:32.353212 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:07:32.353601 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:32.353569 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:07:32.353805 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:32.353785 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:07:33.142804 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:33.142766 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" event={"ID":"922af45f-b42b-47de-9724-6f2daa958330","Type":"ContainerStarted","Data":"0eaeb83660100b9966a6b659e9b68f1b4598fa0e0ed0a6b61c790ed6224edb4b"} Nov 01 16:07:33.142993 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:33.142835 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" event={"ID":"922af45f-b42b-47de-9724-6f2daa958330","Type":"ContainerStarted","Data":"475202ef8ebefb883fddf72d8adc33af5dddef5af686ecb3da3eb9c14bd198e1"} Nov 01 16:07:33.177583 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:33.177530 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" podStartSLOduration=2.177514976 podStartE2EDuration="2.177514976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:07:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:07:33.172997303 +0000 UTC m=+1186.504734952" watchObservedRunningTime="2026-11-01 16:07:33.177514976 +0000 UTC m=+1186.509252625" Nov 01 16:07:33.841536 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:33.841489 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" Nov 01 16:07:33.847715 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:33.847692 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" Nov 01 16:07:34.155080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:34.155011 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" Nov 01 16:07:34.156102 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:34.156078 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" Nov 01 16:07:36.876138 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:36.876101 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-59d3a7-7d8d884767-mc7ft"] Nov 01 16:07:36.885674 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:36.885648 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-59d3a7-7d8d884767-mc7ft" Nov 01 16:07:36.894568 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:36.894373 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-59d3a7\"" Nov 01 16:07:36.894568 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:36.894440 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-59d3a7-dockercfg-qkzwv\"" Nov 01 16:07:36.944354 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:36.944283 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-59d3a7-7d8d884767-mc7ft"] Nov 01 16:07:37.059446 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:37.059407 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78018b59-7aa6-4aa0-b243-edca76667e74-plugins-config-volume\") pod \"payload-processing-e2e-api-b-59d3a7-7d8d884767-mc7ft\" (UID: \"78018b59-7aa6-4aa0-b243-edca76667e74\") " pod="openshift-ingress/payload-processing-e2e-api-b-59d3a7-7d8d884767-mc7ft" Nov 01 16:07:37.059868 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:37.059840 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49v5x\" (UniqueName: \"kubernetes.io/projected/78018b59-7aa6-4aa0-b243-edca76667e74-kube-api-access-49v5x\") pod \"payload-processing-e2e-api-b-59d3a7-7d8d884767-mc7ft\" (UID: \"78018b59-7aa6-4aa0-b243-edca76667e74\") " pod="openshift-ingress/payload-processing-e2e-api-b-59d3a7-7d8d884767-mc7ft" Nov 01 16:07:37.122102 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:37.122070 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-59d3a7-5c65656bb7-gbhbs"] Nov 01 16:07:37.231107 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:37.166536 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78018b59-7aa6-4aa0-b243-edca76667e74-plugins-config-volume\") pod \"payload-processing-e2e-api-b-59d3a7-7d8d884767-mc7ft\" (UID: \"78018b59-7aa6-4aa0-b243-edca76667e74\") " pod="openshift-ingress/payload-processing-e2e-api-b-59d3a7-7d8d884767-mc7ft" Nov 01 16:07:37.231107 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:37.166716 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49v5x\" (UniqueName: \"kubernetes.io/projected/78018b59-7aa6-4aa0-b243-edca76667e74-kube-api-access-49v5x\") pod \"payload-processing-e2e-api-b-59d3a7-7d8d884767-mc7ft\" (UID: \"78018b59-7aa6-4aa0-b243-edca76667e74\") " pod="openshift-ingress/payload-processing-e2e-api-b-59d3a7-7d8d884767-mc7ft" Nov 01 16:07:37.231107 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:37.168137 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78018b59-7aa6-4aa0-b243-edca76667e74-plugins-config-volume\") pod \"payload-processing-e2e-api-b-59d3a7-7d8d884767-mc7ft\" (UID: \"78018b59-7aa6-4aa0-b243-edca76667e74\") " pod="openshift-ingress/payload-processing-e2e-api-b-59d3a7-7d8d884767-mc7ft" Nov 01 16:07:37.305763 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:37.304763 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-59d3a7-5c65656bb7-gbhbs" Nov 01 16:07:37.449214 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:37.427914 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjg84\" (UniqueName: \"kubernetes.io/projected/4a407516-3c42-40a5-9f89-2c6934a1996f-kube-api-access-kjg84\") pod \"payload-pre-processing-e2e-api-b-59d3a7-5c65656bb7-gbhbs\" (UID: \"4a407516-3c42-40a5-9f89-2c6934a1996f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-59d3a7-5c65656bb7-gbhbs" Nov 01 16:07:37.449214 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:37.428280 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a407516-3c42-40a5-9f89-2c6934a1996f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-59d3a7-5c65656bb7-gbhbs\" (UID: \"4a407516-3c42-40a5-9f89-2c6934a1996f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-59d3a7-5c65656bb7-gbhbs" Nov 01 16:07:37.449214 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:37.428459 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-59d3a7-5c65656bb7-gbhbs"] Nov 01 16:07:37.542782 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:37.542655 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjg84\" (UniqueName: \"kubernetes.io/projected/4a407516-3c42-40a5-9f89-2c6934a1996f-kube-api-access-kjg84\") pod \"payload-pre-processing-e2e-api-b-59d3a7-5c65656bb7-gbhbs\" (UID: \"4a407516-3c42-40a5-9f89-2c6934a1996f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-59d3a7-5c65656bb7-gbhbs" Nov 01 16:07:37.543372 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:37.543324 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a407516-3c42-40a5-9f89-2c6934a1996f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-59d3a7-5c65656bb7-gbhbs\" (UID: \"4a407516-3c42-40a5-9f89-2c6934a1996f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-59d3a7-5c65656bb7-gbhbs" Nov 01 16:07:37.544863 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:37.544829 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a407516-3c42-40a5-9f89-2c6934a1996f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-59d3a7-5c65656bb7-gbhbs\" (UID: \"4a407516-3c42-40a5-9f89-2c6934a1996f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-59d3a7-5c65656bb7-gbhbs" Nov 01 16:07:37.545915 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:37.545810 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49v5x\" (UniqueName: \"kubernetes.io/projected/78018b59-7aa6-4aa0-b243-edca76667e74-kube-api-access-49v5x\") pod \"payload-processing-e2e-api-b-59d3a7-7d8d884767-mc7ft\" (UID: \"78018b59-7aa6-4aa0-b243-edca76667e74\") " pod="openshift-ingress/payload-processing-e2e-api-b-59d3a7-7d8d884767-mc7ft" Nov 01 16:07:37.581233 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:37.581201 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-59d3a7-7d8d884767-mc7ft" Nov 01 16:07:37.770383 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:37.770232 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjg84\" (UniqueName: \"kubernetes.io/projected/4a407516-3c42-40a5-9f89-2c6934a1996f-kube-api-access-kjg84\") pod \"payload-pre-processing-e2e-api-b-59d3a7-5c65656bb7-gbhbs\" (UID: \"4a407516-3c42-40a5-9f89-2c6934a1996f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-59d3a7-5c65656bb7-gbhbs" Nov 01 16:07:38.032299 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:38.032264 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-59d3a7-5c65656bb7-gbhbs" Nov 01 16:07:38.060209 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:07:38.060169 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78018b59_7aa6_4aa0_b243_edca76667e74.slice/crio-9545f9f72db7f08c24e8321f99e1748bee353e48e8aacbb5e38907e6e2d6ad06 WatchSource:0}: Error finding container 9545f9f72db7f08c24e8321f99e1748bee353e48e8aacbb5e38907e6e2d6ad06: Status 404 returned error can't find the container with id 9545f9f72db7f08c24e8321f99e1748bee353e48e8aacbb5e38907e6e2d6ad06 Nov 01 16:07:38.063239 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:38.063208 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-59d3a7-7d8d884767-mc7ft"] Nov 01 16:07:38.907913 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:38.907883 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-59d3a7-5c65656bb7-gbhbs"] Nov 01 16:07:38.908273 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:38.908255 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-59d3a7-7d8d884767-mc7ft" Nov 01 16:07:38.908462 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:38.908443 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-59d3a7-7d8d884767-mc7ft" event={"ID":"78018b59-7aa6-4aa0-b243-edca76667e74","Type":"ContainerStarted","Data":"9545f9f72db7f08c24e8321f99e1748bee353e48e8aacbb5e38907e6e2d6ad06"} Nov 01 16:07:38.927143 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:38.922979 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-59d3a7-7d8d884767-mc7ft" podStartSLOduration=2.922955207 podStartE2EDuration="2.922955207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:07:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:07:38.919798704 +0000 UTC m=+1192.251536356" watchObservedRunningTime="2026-11-01 16:07:38.922955207 +0000 UTC m=+1192.254692857" Nov 01 16:07:39.920746 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:39.920711 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-59d3a7-7d8d884767-mc7ft" event={"ID":"78018b59-7aa6-4aa0-b243-edca76667e74","Type":"ContainerStarted","Data":"bb057d61deeb549749194f8e82593da2d47e0cd1fddfd67fd87db4a908ddb2de"} Nov 01 16:07:39.923854 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:39.923826 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-59d3a7-5c65656bb7-gbhbs" event={"ID":"4a407516-3c42-40a5-9f89-2c6934a1996f","Type":"ContainerStarted","Data":"d5f2ba997391283ccce2ca80f740881de5faebc1d41628d5cf8c98239834b7ea"} Nov 01 16:07:39.923999 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:39.923875 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-59d3a7-5c65656bb7-gbhbs" event={"ID":"4a407516-3c42-40a5-9f89-2c6934a1996f","Type":"ContainerStarted","Data":"32605e86e5b9c0dacbf9d5daf0ddeb73c1b6baf7084b6b686c39328dd0d3ceb5"} Nov 01 16:07:39.923999 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:39.923919 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-59d3a7-5c65656bb7-gbhbs" Nov 01 16:07:39.977150 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:39.970564 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-59d3a7-5c65656bb7-gbhbs" podStartSLOduration=3.970545467 podStartE2EDuration="3.970545467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:07:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:07:39.945513453 +0000 UTC m=+1193.277251134" watchObservedRunningTime="2026-11-01 16:07:39.970545467 +0000 UTC m=+1193.302283118" Nov 01 16:07:47.058817 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:47.058789 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtwhw_8254aafa-dd5b-4150-87e4-d165dab6bcb5/ovn-acl-logging/0.log" Nov 01 16:07:47.200667 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:47.186039 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtwhw_8254aafa-dd5b-4150-87e4-d165dab6bcb5/ovn-acl-logging/0.log" Nov 01 16:07:48.553109 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:07:48.553068 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:07:52.043082 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:52.038485 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:52.124046 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:07:52.060795 4363 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.27s" Nov 01 16:07:52.124046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:52.060918 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-59d3a7-7d8d884767-mc7ft" Nov 01 16:07:52.124046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:52.061029 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-59d3a7-5c65656bb7-gbhbs" Nov 01 16:07:52.284041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:52.267652 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:52.736040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:52.717254 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" podUID="f1e164ab-e940-44e0-8829-8626e4162a9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.115:8000/ready\": context deadline exceeded" Nov 01 16:07:52.875197 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:52.875146 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-f01de5-65884f55d5-pbgx8"] Nov 01 16:07:53.035465 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:53.035365 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:07:53.122042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:53.113876 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:53.122042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:53.039118 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:07:53.181202 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:53.050120 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:53.218455 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:53.182898 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:07:53.422197 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:53.422164 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f01de5-65884f55d5-pbgx8" Nov 01 16:07:53.451314 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:53.451287 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-f01de5\"" Nov 01 16:07:53.451779 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:53.451763 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-f01de5-dockercfg-vsmtl\"" Nov 01 16:07:53.852102 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:53.846183 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 01 16:07:53.994206 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:53.883675 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" podUID="f1e164ab-e940-44e0-8829-8626e4162a9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:54.095842 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:54.094274 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpjwn\" (UniqueName: \"kubernetes.io/projected/387b6adf-9404-414e-ae75-0b48e938e77a-kube-api-access-zpjwn\") pod \"payload-processing-e2e-shared-b-w4-f01de5-65884f55d5-pbgx8\" (UID: \"387b6adf-9404-414e-ae75-0b48e938e77a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f01de5-65884f55d5-pbgx8" Nov 01 16:07:54.095842 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:54.094397 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/387b6adf-9404-414e-ae75-0b48e938e77a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-f01de5-65884f55d5-pbgx8\" (UID: \"387b6adf-9404-414e-ae75-0b48e938e77a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f01de5-65884f55d5-pbgx8" Nov 01 16:07:54.130097 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:54.119140 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:54.130097 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:54.119416 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:54.232052 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:54.231187 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:54.232052 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:54.231403 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:07:54.265077 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:54.262157 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded" Nov 01 16:07:54.438224 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:07:54.425674 4363 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.038s" Nov 01 16:07:54.438602 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:54.438569 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 16:07:54.438985 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:54.438969 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 16:07:54.439187 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:54.439164 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" Nov 01 16:07:54.439376 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:54.439346 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:07:54.439544 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:54.439532 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:07:54.439700 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:54.439688 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:07:54.545403 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:54.119352 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:54.592004 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:54.591974 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-f01de5-65884f55d5-pbgx8"] Nov 01 16:07:54.602050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:54.598678 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:07:54.602050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:54.598756 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 16:07:54.606141 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:54.602727 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:07:54.666072 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:54.661187 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpjwn\" (UniqueName: \"kubernetes.io/projected/387b6adf-9404-414e-ae75-0b48e938e77a-kube-api-access-zpjwn\") pod \"payload-processing-e2e-shared-b-w4-f01de5-65884f55d5-pbgx8\" (UID: \"387b6adf-9404-414e-ae75-0b48e938e77a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f01de5-65884f55d5-pbgx8" Nov 01 16:07:54.666072 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:54.661318 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/387b6adf-9404-414e-ae75-0b48e938e77a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-f01de5-65884f55d5-pbgx8\" (UID: \"387b6adf-9404-414e-ae75-0b48e938e77a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f01de5-65884f55d5-pbgx8" Nov 01 16:07:54.763284 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:54.763186 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" Nov 01 16:07:54.764473 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:54.764450 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 16:07:54.803608 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:54.803559 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/387b6adf-9404-414e-ae75-0b48e938e77a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-f01de5-65884f55d5-pbgx8\" (UID: \"387b6adf-9404-414e-ae75-0b48e938e77a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f01de5-65884f55d5-pbgx8" Nov 01 16:07:54.811123 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:54.811078 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:07:54.843913 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:54.843389 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpjwn\" (UniqueName: \"kubernetes.io/projected/387b6adf-9404-414e-ae75-0b48e938e77a-kube-api-access-zpjwn\") pod \"payload-processing-e2e-shared-b-w4-f01de5-65884f55d5-pbgx8\" (UID: \"387b6adf-9404-414e-ae75-0b48e938e77a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f01de5-65884f55d5-pbgx8" Nov 01 16:07:54.879745 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:54.879711 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:07:54.968229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:54.967982 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:07:55.055625 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:55.055404 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f01de5-65884f55d5-pbgx8" Nov 01 16:07:55.436636 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:55.434802 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-f01de5-65884f55d5-pbgx8"] Nov 01 16:07:55.815291 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:55.815185 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f01de5-65884f55d5-pbgx8" event={"ID":"387b6adf-9404-414e-ae75-0b48e938e77a","Type":"ContainerStarted","Data":"7e12c927caea0f8d330caf6d5d8dbbf8e12e49f79150bd84d09240a2e5b52fbb"} Nov 01 16:07:55.815291 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:55.815250 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f01de5-65884f55d5-pbgx8" event={"ID":"387b6adf-9404-414e-ae75-0b48e938e77a","Type":"ContainerStarted","Data":"869a02c2ab39ce8338032fdc5ba7ea3b0db3b4e17745f931c603f04bd9b1bada"} Nov 01 16:07:55.815725 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:55.815702 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f01de5-65884f55d5-pbgx8" Nov 01 16:07:55.832655 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:07:55.832608 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f01de5-65884f55d5-pbgx8" podStartSLOduration=4.832595165 podStartE2EDuration="4.832595165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:07:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:07:55.831606611 +0000 UTC m=+1209.163344262" watchObservedRunningTime="2026-11-01 16:07:55.832595165 +0000 UTC m=+1209.164332816" Nov 01 16:08:07.860095 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:07.856257 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f01de5-65884f55d5-pbgx8" Nov 01 16:08:18.312287 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:08:18.312244 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:08:22.665837 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:22.665796 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-61bb54-84479f79f5-rc5cd"] Nov 01 16:08:27.667184 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:27.667119 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-61bb54-84479f79f5-rc5cd" podUID="c7434b5f-5698-447d-a93a-41ffcbe77f9b" containerName="payload-pre-processing" containerID="cri-o://bebd0e2654355afd01582c29307e860c3912fa71aa9a8491ad376d1781ef8e98" gracePeriod=25 Nov 01 16:08:28.110087 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:28.110010 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-61bb54-84479f79f5-rc5cd" Nov 01 16:08:28.196051 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:28.195931 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7434b5f-5698-447d-a93a-41ffcbe77f9b-kube-api-access-n4hxp\" (UniqueName: \"kubernetes.io/projected/c7434b5f-5698-447d-a93a-41ffcbe77f9b-kube-api-access-n4hxp\") pod \"c7434b5f-5698-447d-a93a-41ffcbe77f9b\" (UID: \"c7434b5f-5698-447d-a93a-41ffcbe77f9b\") " Nov 01 16:08:28.196273 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:28.196245 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7434b5f-5698-447d-a93a-41ffcbe77f9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7434b5f-5698-447d-a93a-41ffcbe77f9b-plugins-config-volume\") pod \"c7434b5f-5698-447d-a93a-41ffcbe77f9b\" (UID: \"c7434b5f-5698-447d-a93a-41ffcbe77f9b\") " Nov 01 16:08:28.197880 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:28.197852 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7434b5f-5698-447d-a93a-41ffcbe77f9b-plugins-config-volume" pod "c7434b5f-5698-447d-a93a-41ffcbe77f9b" (UID: "c7434b5f-5698-447d-a93a-41ffcbe77f9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:08:28.256343 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:28.256295 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7434b5f-5698-447d-a93a-41ffcbe77f9b-kube-api-access-n4hxp" pod "c7434b5f-5698-447d-a93a-41ffcbe77f9b" (UID: "c7434b5f-5698-447d-a93a-41ffcbe77f9b"). InnerVolumeSpecName "kube-api-access-n4hxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:08:28.300065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:28.299919 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7434b5f-5698-447d-a93a-41ffcbe77f9b-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:28.300065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:28.299963 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4hxp\" (UniqueName: \"kubernetes.io/projected/c7434b5f-5698-447d-a93a-41ffcbe77f9b-kube-api-access-n4hxp\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:28.490663 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:28.490632 4363 generic.go:356] "Generic (PLEG): container finished" podID="c7434b5f-5698-447d-a93a-41ffcbe77f9b" containerID="bebd0e2654355afd01582c29307e860c3912fa71aa9a8491ad376d1781ef8e98" exitCode=0 Nov 01 16:08:28.490840 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:28.490691 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-61bb54-84479f79f5-rc5cd" Nov 01 16:08:28.490840 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:28.490721 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-61bb54-84479f79f5-rc5cd" event={"ID":"c7434b5f-5698-447d-a93a-41ffcbe77f9b","Type":"ContainerDied","Data":"bebd0e2654355afd01582c29307e860c3912fa71aa9a8491ad376d1781ef8e98"} Nov 01 16:08:28.490840 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:28.490809 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-61bb54-84479f79f5-rc5cd" event={"ID":"c7434b5f-5698-447d-a93a-41ffcbe77f9b","Type":"ContainerDied","Data":"504b4f8729c9dbae255fc33e7a0b8eb240dcbdf586d6388020120c2da48f69e4"} Nov 01 16:08:28.490840 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:28.490838 4363 scope.go:122] "RemoveContainer" containerID="bebd0e2654355afd01582c29307e860c3912fa71aa9a8491ad376d1781ef8e98" Nov 01 16:08:28.510692 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:28.510670 4363 scope.go:122] "RemoveContainer" containerID="bebd0e2654355afd01582c29307e860c3912fa71aa9a8491ad376d1781ef8e98" Nov 01 16:08:28.511063 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:08:28.511037 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bebd0e2654355afd01582c29307e860c3912fa71aa9a8491ad376d1781ef8e98\": container with ID starting with bebd0e2654355afd01582c29307e860c3912fa71aa9a8491ad376d1781ef8e98 not found: ID does not exist" containerID="bebd0e2654355afd01582c29307e860c3912fa71aa9a8491ad376d1781ef8e98" Nov 01 16:08:28.511154 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:28.511076 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bebd0e2654355afd01582c29307e860c3912fa71aa9a8491ad376d1781ef8e98"} err="failed to get container status \"bebd0e2654355afd01582c29307e860c3912fa71aa9a8491ad376d1781ef8e98\": rpc error: code = NotFound desc = could not find container \"bebd0e2654355afd01582c29307e860c3912fa71aa9a8491ad376d1781ef8e98\": container with ID starting with bebd0e2654355afd01582c29307e860c3912fa71aa9a8491ad376d1781ef8e98 not found: ID does not exist" Nov 01 16:08:28.520216 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:28.520188 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-61bb54-84479f79f5-rc5cd"] Nov 01 16:08:28.526903 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:28.526882 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-61bb54-84479f79f5-rc5cd"] Nov 01 16:08:28.801750 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:28.801663 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7434b5f-5698-447d-a93a-41ffcbe77f9b" path="/var/lib/kubelet/pods/c7434b5f-5698-447d-a93a-41ffcbe77f9b/volumes" Nov 01 16:08:31.234947 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:31.234912 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5c6b75-58cdb8f699-tzrc2"] Nov 01 16:08:31.236060 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:31.236034 4363 cpu_manager.go:414] "containerMap: removing container" podUID="c7434b5f-5698-447d-a93a-41ffcbe77f9b" containerName="payload-pre-processing" Nov 01 16:08:31.236060 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:31.236059 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7434b5f-5698-447d-a93a-41ffcbe77f9b" containerName="payload-pre-processing" Nov 01 16:08:31.236800 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:31.236777 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7434b5f-5698-447d-a93a-41ffcbe77f9b" containerName="payload-pre-processing" Nov 01 16:08:31.255675 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:31.255643 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5c6b75-9f88c8787-vm79j"] Nov 01 16:08:31.256245 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:31.256224 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5c6b75-58cdb8f699-tzrc2" Nov 01 16:08:31.264707 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:31.264682 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-5c6b75\"" Nov 01 16:08:31.266147 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:31.266123 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5c6b75-58cdb8f699-tzrc2"] Nov 01 16:08:31.266263 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:31.266229 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c6b75-9f88c8787-vm79j" Nov 01 16:08:31.271239 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:31.271218 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-5c6b75-dockercfg-bl66v\"" Nov 01 16:08:31.277513 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:31.277394 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5c6b75-9f88c8787-vm79j"] Nov 01 16:08:31.433039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:31.419421 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgn95\" (UniqueName: \"kubernetes.io/projected/df7c1364-8484-4cb7-9010-b7acbd09ccba-kube-api-access-lgn95\") pod \"payload-pre-processing-e2e-api-a-5c6b75-9f88c8787-vm79j\" (UID: \"df7c1364-8484-4cb7-9010-b7acbd09ccba\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c6b75-9f88c8787-vm79j" Nov 01 16:08:31.433039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:31.419717 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df7c1364-8484-4cb7-9010-b7acbd09ccba-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5c6b75-9f88c8787-vm79j\" (UID: \"df7c1364-8484-4cb7-9010-b7acbd09ccba\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c6b75-9f88c8787-vm79j" Nov 01 16:08:31.433039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:31.419859 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/729e8251-2e65-4826-a535-a19b551e04c7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5c6b75-58cdb8f699-tzrc2\" (UID: \"729e8251-2e65-4826-a535-a19b551e04c7\") " pod="openshift-ingress/payload-processing-e2e-api-a-5c6b75-58cdb8f699-tzrc2" Nov 01 16:08:31.433039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:31.419951 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdtww\" (UniqueName: \"kubernetes.io/projected/729e8251-2e65-4826-a535-a19b551e04c7-kube-api-access-fdtww\") pod \"payload-processing-e2e-api-a-5c6b75-58cdb8f699-tzrc2\" (UID: \"729e8251-2e65-4826-a535-a19b551e04c7\") " pod="openshift-ingress/payload-processing-e2e-api-a-5c6b75-58cdb8f699-tzrc2" Nov 01 16:08:31.521685 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:31.521585 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/729e8251-2e65-4826-a535-a19b551e04c7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5c6b75-58cdb8f699-tzrc2\" (UID: \"729e8251-2e65-4826-a535-a19b551e04c7\") " pod="openshift-ingress/payload-processing-e2e-api-a-5c6b75-58cdb8f699-tzrc2" Nov 01 16:08:31.522927 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:31.522902 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdtww\" (UniqueName: \"kubernetes.io/projected/729e8251-2e65-4826-a535-a19b551e04c7-kube-api-access-fdtww\") pod \"payload-processing-e2e-api-a-5c6b75-58cdb8f699-tzrc2\" (UID: \"729e8251-2e65-4826-a535-a19b551e04c7\") " pod="openshift-ingress/payload-processing-e2e-api-a-5c6b75-58cdb8f699-tzrc2" Nov 01 16:08:31.523603 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:31.523578 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgn95\" (UniqueName: \"kubernetes.io/projected/df7c1364-8484-4cb7-9010-b7acbd09ccba-kube-api-access-lgn95\") pod \"payload-pre-processing-e2e-api-a-5c6b75-9f88c8787-vm79j\" (UID: \"df7c1364-8484-4cb7-9010-b7acbd09ccba\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c6b75-9f88c8787-vm79j" Nov 01 16:08:31.524072 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:31.524051 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df7c1364-8484-4cb7-9010-b7acbd09ccba-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5c6b75-9f88c8787-vm79j\" (UID: \"df7c1364-8484-4cb7-9010-b7acbd09ccba\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c6b75-9f88c8787-vm79j" Nov 01 16:08:31.525081 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:31.525059 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df7c1364-8484-4cb7-9010-b7acbd09ccba-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5c6b75-9f88c8787-vm79j\" (UID: \"df7c1364-8484-4cb7-9010-b7acbd09ccba\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c6b75-9f88c8787-vm79j" Nov 01 16:08:31.525213 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:31.522755 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/729e8251-2e65-4826-a535-a19b551e04c7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5c6b75-58cdb8f699-tzrc2\" (UID: \"729e8251-2e65-4826-a535-a19b551e04c7\") " pod="openshift-ingress/payload-processing-e2e-api-a-5c6b75-58cdb8f699-tzrc2" Nov 01 16:08:31.565279 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:31.563185 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgn95\" (UniqueName: \"kubernetes.io/projected/df7c1364-8484-4cb7-9010-b7acbd09ccba-kube-api-access-lgn95\") pod \"payload-pre-processing-e2e-api-a-5c6b75-9f88c8787-vm79j\" (UID: \"df7c1364-8484-4cb7-9010-b7acbd09ccba\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c6b75-9f88c8787-vm79j" Nov 01 16:08:31.598940 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:31.598909 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdtww\" (UniqueName: \"kubernetes.io/projected/729e8251-2e65-4826-a535-a19b551e04c7-kube-api-access-fdtww\") pod \"payload-processing-e2e-api-a-5c6b75-58cdb8f699-tzrc2\" (UID: \"729e8251-2e65-4826-a535-a19b551e04c7\") " pod="openshift-ingress/payload-processing-e2e-api-a-5c6b75-58cdb8f699-tzrc2" Nov 01 16:08:31.613385 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:31.613351 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c6b75-9f88c8787-vm79j" Nov 01 16:08:31.891103 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:31.883682 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5c6b75-58cdb8f699-tzrc2" Nov 01 16:08:32.473461 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:32.473424 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5c6b75-9f88c8787-vm79j"] Nov 01 16:08:32.510101 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:08:32.510068 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf7c1364_8484_4cb7_9010_b7acbd09ccba.slice/crio-9b8e8adeba0ac81eab170e3844d7e9907cb144712ec4eae77d6b9214fb54ce86 WatchSource:0}: Error finding container 9b8e8adeba0ac81eab170e3844d7e9907cb144712ec4eae77d6b9214fb54ce86: Status 404 returned error can't find the container with id 9b8e8adeba0ac81eab170e3844d7e9907cb144712ec4eae77d6b9214fb54ce86 Nov 01 16:08:32.554151 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:32.554126 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5c6b75-58cdb8f699-tzrc2"] Nov 01 16:08:32.554657 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:08:32.554624 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod729e8251_2e65_4826_a535_a19b551e04c7.slice/crio-ba2bc9bc8bb701171068cae37e7622b505884a40b34087c529027d100157478c WatchSource:0}: Error finding container ba2bc9bc8bb701171068cae37e7622b505884a40b34087c529027d100157478c: Status 404 returned error can't find the container with id ba2bc9bc8bb701171068cae37e7622b505884a40b34087c529027d100157478c Nov 01 16:08:32.814835 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:32.814799 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c6b75-9f88c8787-vm79j" Nov 01 16:08:32.814997 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:32.814852 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c6b75-9f88c8787-vm79j" event={"ID":"df7c1364-8484-4cb7-9010-b7acbd09ccba","Type":"ContainerStarted","Data":"c8f18e85231ccdb24839f583a9dc8bba3207c70dc65e9e8c75594f9f4004a707"} Nov 01 16:08:32.814997 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:32.814871 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c6b75-9f88c8787-vm79j" event={"ID":"df7c1364-8484-4cb7-9010-b7acbd09ccba","Type":"ContainerStarted","Data":"9b8e8adeba0ac81eab170e3844d7e9907cb144712ec4eae77d6b9214fb54ce86"} Nov 01 16:08:32.816108 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:32.816012 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5c6b75-58cdb8f699-tzrc2" event={"ID":"729e8251-2e65-4826-a535-a19b551e04c7","Type":"ContainerStarted","Data":"307bbc000ccbf043f74a3fd2618d3fd25241b1b413b8ec0cb7562d6621d7689a"} Nov 01 16:08:32.816228 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:32.816135 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5c6b75-58cdb8f699-tzrc2" event={"ID":"729e8251-2e65-4826-a535-a19b551e04c7","Type":"ContainerStarted","Data":"ba2bc9bc8bb701171068cae37e7622b505884a40b34087c529027d100157478c"} Nov 01 16:08:32.816228 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:32.816179 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-5c6b75-58cdb8f699-tzrc2" Nov 01 16:08:32.836233 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:32.836151 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c6b75-9f88c8787-vm79j" podStartSLOduration=1.836139485 podStartE2EDuration="1.836139485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:08:32.83122141 +0000 UTC m=+1246.162959064" watchObservedRunningTime="2026-11-01 16:08:32.836139485 +0000 UTC m=+1246.167877135" Nov 01 16:08:32.846419 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:32.846375 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-5c6b75-58cdb8f699-tzrc2" podStartSLOduration=1.846363806 podStartE2EDuration="1.846363806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:08:32.846087677 +0000 UTC m=+1246.177825327" watchObservedRunningTime="2026-11-01 16:08:32.846363806 +0000 UTC m=+1246.178101456" Nov 01 16:08:43.118252 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:43.118214 4363 generic.go:356] "Generic (PLEG): container finished" podID="922af45f-b42b-47de-9724-6f2daa958330" containerID="0eaeb83660100b9966a6b659e9b68f1b4598fa0e0ed0a6b61c790ed6224edb4b" exitCode=0 Nov 01 16:08:43.118975 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:43.118923 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" event={"ID":"922af45f-b42b-47de-9724-6f2daa958330","Type":"ContainerDied","Data":"0eaeb83660100b9966a6b659e9b68f1b4598fa0e0ed0a6b61c790ed6224edb4b"} Nov 01 16:08:43.120555 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:43.120533 4363 scope.go:122] "RemoveContainer" containerID="0eaeb83660100b9966a6b659e9b68f1b4598fa0e0ed0a6b61c790ed6224edb4b" Nov 01 16:08:43.124132 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:43.124095 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:08:43.125077 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:43.125005 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:08:43.125294 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:43.125271 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:08:43.648046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:43.647936 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-61bb54-7b8559df57-hkdqs"] Nov 01 16:08:43.648239 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:43.648199 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61bb54-7b8559df57-hkdqs" podUID="eac30ab3-746f-430c-864b-6b469263e9e1" containerName="maas-api" containerID="cri-o://cf1d129fd09a543b233ed3ec2be6a52341ab9ead21272a6d884ca4904d869c32" gracePeriod=30 Nov 01 16:08:43.693243 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:43.692534 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61bb54-7b8559df57-hkdqs" podUID="eac30ab3-746f-430c-864b-6b469263e9e1" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.117:8443/health\": dial tcp 10.134.0.117:8443: connect: connection refused" Nov 01 16:08:44.108114 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:44.039197 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-5c6b75-58cdb8f699-tzrc2" Nov 01 16:08:44.108114 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:44.091535 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c6b75-9f88c8787-vm79j" Nov 01 16:08:44.306723 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:44.306629 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" event={"ID":"922af45f-b42b-47de-9724-6f2daa958330","Type":"ContainerStarted","Data":"fcdfb6c868bbc4f02d17cd5267156ea0c447011e705da88207029c6f3bb277ef"} Nov 01 16:08:44.312628 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:44.312001 4363 generic.go:356] "Generic (PLEG): container finished" podID="eac30ab3-746f-430c-864b-6b469263e9e1" containerID="cf1d129fd09a543b233ed3ec2be6a52341ab9ead21272a6d884ca4904d869c32" exitCode=0 Nov 01 16:08:44.312798 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:44.312719 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61bb54-7b8559df57-hkdqs" event={"ID":"eac30ab3-746f-430c-864b-6b469263e9e1","Type":"ContainerDied","Data":"cf1d129fd09a543b233ed3ec2be6a52341ab9ead21272a6d884ca4904d869c32"} Nov 01 16:08:44.763890 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:44.763858 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61bb54-7b8559df57-hkdqs" Nov 01 16:08:44.843039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:44.836656 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eac30ab3-746f-430c-864b-6b469263e9e1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eac30ab3-746f-430c-864b-6b469263e9e1-openshift-service-ca\") pod \"eac30ab3-746f-430c-864b-6b469263e9e1\" (UID: \"eac30ab3-746f-430c-864b-6b469263e9e1\") " Nov 01 16:08:44.843039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:44.836947 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eac30ab3-746f-430c-864b-6b469263e9e1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eac30ab3-746f-430c-864b-6b469263e9e1-maas-api-tls\") pod \"eac30ab3-746f-430c-864b-6b469263e9e1\" (UID: \"eac30ab3-746f-430c-864b-6b469263e9e1\") " Nov 01 16:08:44.845501 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:44.843460 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eac30ab3-746f-430c-864b-6b469263e9e1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eac30ab3-746f-430c-864b-6b469263e9e1-metrics-certs\") pod \"eac30ab3-746f-430c-864b-6b469263e9e1\" (UID: \"eac30ab3-746f-430c-864b-6b469263e9e1\") " Nov 01 16:08:44.845501 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:44.843584 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eac30ab3-746f-430c-864b-6b469263e9e1-kube-api-access-q9x4n\" (UniqueName: \"kubernetes.io/projected/eac30ab3-746f-430c-864b-6b469263e9e1-kube-api-access-q9x4n\") pod \"eac30ab3-746f-430c-864b-6b469263e9e1\" (UID: \"eac30ab3-746f-430c-864b-6b469263e9e1\") " Nov 01 16:08:44.869144 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:44.865588 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eac30ab3-746f-430c-864b-6b469263e9e1-openshift-service-ca" pod "eac30ab3-746f-430c-864b-6b469263e9e1" (UID: "eac30ab3-746f-430c-864b-6b469263e9e1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:08:44.962075 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:44.961659 4363 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eac30ab3-746f-430c-864b-6b469263e9e1-openshift-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:44.995905 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:44.995864 4363 patch_prober.go:29] interesting pod/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 01 16:08:44.996330 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:44.996253 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" podUID="922af45f-b42b-47de-9724-6f2daa958330" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 01 16:08:45.077049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:45.072645 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eac30ab3-746f-430c-864b-6b469263e9e1-kube-api-access-q9x4n" pod "eac30ab3-746f-430c-864b-6b469263e9e1" (UID: "eac30ab3-746f-430c-864b-6b469263e9e1"). InnerVolumeSpecName "kube-api-access-q9x4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:08:45.106037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:45.104903 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eac30ab3-746f-430c-864b-6b469263e9e1-metrics-certs" pod "eac30ab3-746f-430c-864b-6b469263e9e1" (UID: "eac30ab3-746f-430c-864b-6b469263e9e1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:08:45.361034 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:45.353799 4363 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eac30ab3-746f-430c-864b-6b469263e9e1-metrics-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:45.361034 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:45.353840 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9x4n\" (UniqueName: \"kubernetes.io/projected/eac30ab3-746f-430c-864b-6b469263e9e1-kube-api-access-q9x4n\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:45.636096 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:45.621419 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eac30ab3-746f-430c-864b-6b469263e9e1-maas-api-tls" pod "eac30ab3-746f-430c-864b-6b469263e9e1" (UID: "eac30ab3-746f-430c-864b-6b469263e9e1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:08:45.770062 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:45.720362 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eac30ab3-746f-430c-864b-6b469263e9e1-maas-api-tls\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:45.855192 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:45.847365 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61bb54-7b8559df57-hkdqs" Nov 01 16:08:45.874656 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:45.866087 4363 patch_prober.go:29] interesting pod/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 01 16:08:45.874656 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:45.866145 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" podUID="922af45f-b42b-47de-9724-6f2daa958330" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 01 16:08:45.895315 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:08:45.892330 4363 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.097s" Nov 01 16:08:45.895315 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:45.892387 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" Nov 01 16:08:45.895315 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:45.892544 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61bb54-7b8559df57-hkdqs" event={"ID":"eac30ab3-746f-430c-864b-6b469263e9e1","Type":"ContainerDied","Data":"a9d42f6cfdce7aef8c70595a200f85a1611e8d58be23a10bf6ac0af7cec9679f"} Nov 01 16:08:45.895315 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:45.892604 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5c6b75-7478fffbb9-st6hq"] Nov 01 16:08:45.913612 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:45.895735 4363 scope.go:122] "RemoveContainer" containerID="cf1d129fd09a543b233ed3ec2be6a52341ab9ead21272a6d884ca4904d869c32" Nov 01 16:08:45.913612 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:45.898742 4363 cpu_manager.go:414] "containerMap: removing container" podUID="eac30ab3-746f-430c-864b-6b469263e9e1" containerName="maas-api" Nov 01 16:08:45.913612 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:45.898764 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eac30ab3-746f-430c-864b-6b469263e9e1" containerName="maas-api" Nov 01 16:08:45.913612 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:45.900296 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="eac30ab3-746f-430c-864b-6b469263e9e1" containerName="maas-api" Nov 01 16:08:45.948432 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:45.948406 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5c6b75-7478fffbb9-st6hq"] Nov 01 16:08:45.948727 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:45.948711 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6b75-7478fffbb9-st6hq" Nov 01 16:08:45.957535 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:45.957513 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-5c6b75-serving-cert\"" Nov 01 16:08:45.967697 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:45.967677 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-61bb54-7b8559df57-hkdqs"] Nov 01 16:08:45.973233 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:45.973211 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-61bb54-7b8559df57-hkdqs"] Nov 01 16:08:46.046405 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:46.046369 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25-metrics-certs\") pod \"maas-api-e2e-api-a-5c6b75-7478fffbb9-st6hq\" (UID: \"a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6b75-7478fffbb9-st6hq" Nov 01 16:08:46.046815 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:46.046710 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25-openshift-service-ca\") pod \"maas-api-e2e-api-a-5c6b75-7478fffbb9-st6hq\" (UID: \"a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6b75-7478fffbb9-st6hq" Nov 01 16:08:46.047085 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:46.047060 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccxhl\" (UniqueName: \"kubernetes.io/projected/a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25-kube-api-access-ccxhl\") pod \"maas-api-e2e-api-a-5c6b75-7478fffbb9-st6hq\" (UID: \"a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6b75-7478fffbb9-st6hq" Nov 01 16:08:46.047182 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:46.047163 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25-maas-api-tls\") pod \"maas-api-e2e-api-a-5c6b75-7478fffbb9-st6hq\" (UID: \"a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6b75-7478fffbb9-st6hq" Nov 01 16:08:46.149367 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:46.149285 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25-openshift-service-ca\") pod \"maas-api-e2e-api-a-5c6b75-7478fffbb9-st6hq\" (UID: \"a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6b75-7478fffbb9-st6hq" Nov 01 16:08:46.149513 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:46.149496 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccxhl\" (UniqueName: \"kubernetes.io/projected/a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25-kube-api-access-ccxhl\") pod \"maas-api-e2e-api-a-5c6b75-7478fffbb9-st6hq\" (UID: \"a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6b75-7478fffbb9-st6hq" Nov 01 16:08:46.149607 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:46.149589 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25-maas-api-tls\") pod \"maas-api-e2e-api-a-5c6b75-7478fffbb9-st6hq\" (UID: \"a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6b75-7478fffbb9-st6hq" Nov 01 16:08:46.149774 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:46.149758 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25-metrics-certs\") pod \"maas-api-e2e-api-a-5c6b75-7478fffbb9-st6hq\" (UID: \"a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6b75-7478fffbb9-st6hq" Nov 01 16:08:46.150061 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:46.150035 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25-openshift-service-ca\") pod \"maas-api-e2e-api-a-5c6b75-7478fffbb9-st6hq\" (UID: \"a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6b75-7478fffbb9-st6hq" Nov 01 16:08:46.152060 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:46.151982 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25-maas-api-tls\") pod \"maas-api-e2e-api-a-5c6b75-7478fffbb9-st6hq\" (UID: \"a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6b75-7478fffbb9-st6hq" Nov 01 16:08:46.152217 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:46.152197 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25-metrics-certs\") pod \"maas-api-e2e-api-a-5c6b75-7478fffbb9-st6hq\" (UID: \"a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6b75-7478fffbb9-st6hq" Nov 01 16:08:46.156381 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:46.156358 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccxhl\" (UniqueName: \"kubernetes.io/projected/a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25-kube-api-access-ccxhl\") pod \"maas-api-e2e-api-a-5c6b75-7478fffbb9-st6hq\" (UID: \"a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6b75-7478fffbb9-st6hq" Nov 01 16:08:46.271781 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:46.271749 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6b75-7478fffbb9-st6hq" Nov 01 16:08:46.343047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:46.341524 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6963-ba3e24"] Nov 01 16:08:46.377070 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:46.372840 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6963-ba3e24" Nov 01 16:08:46.389046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:46.384421 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hqwjk\"" Nov 01 16:08:46.444109 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:46.426981 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6963-ba3e24"] Nov 01 16:08:46.475036 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:46.472822 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tllb\" (UniqueName: \"kubernetes.io/projected/69a265d5-c5d6-4cfd-ae78-daad646b0c6d-kube-api-access-6tllb\") pod \"test-curl-6963-ba3e24\" (UID: \"69a265d5-c5d6-4cfd-ae78-daad646b0c6d\") " pod="opendatahub/test-curl-6963-ba3e24" Nov 01 16:08:46.587267 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:46.586922 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tllb\" (UniqueName: \"kubernetes.io/projected/69a265d5-c5d6-4cfd-ae78-daad646b0c6d-kube-api-access-6tllb\") pod \"test-curl-6963-ba3e24\" (UID: \"69a265d5-c5d6-4cfd-ae78-daad646b0c6d\") " pod="opendatahub/test-curl-6963-ba3e24" Nov 01 16:08:46.640748 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:46.640675 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tllb\" (UniqueName: \"kubernetes.io/projected/69a265d5-c5d6-4cfd-ae78-daad646b0c6d-kube-api-access-6tllb\") pod \"test-curl-6963-ba3e24\" (UID: \"69a265d5-c5d6-4cfd-ae78-daad646b0c6d\") " pod="opendatahub/test-curl-6963-ba3e24" Nov 01 16:08:46.835596 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:46.835164 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eac30ab3-746f-430c-864b-6b469263e9e1" path="/var/lib/kubelet/pods/eac30ab3-746f-430c-864b-6b469263e9e1/volumes" Nov 01 16:08:46.842523 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:46.842489 4363 patch_prober.go:29] interesting pod/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 01 16:08:46.842735 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:46.842550 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" podUID="922af45f-b42b-47de-9724-6f2daa958330" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 01 16:08:46.923508 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:46.922359 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6963-ba3e24" Nov 01 16:08:46.927472 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:46.927439 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5c6b75-7478fffbb9-st6hq"] Nov 01 16:08:46.932585 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:08:46.932544 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6b8a2f8_f6a0_4faa_8cc2_8b86b70fdd25.slice/crio-6352fdf4fd6b691a239668d1d0cf1b31f381c89d3b791dc57009e386879535de WatchSource:0}: Error finding container 6352fdf4fd6b691a239668d1d0cf1b31f381c89d3b791dc57009e386879535de: Status 404 returned error can't find the container with id 6352fdf4fd6b691a239668d1d0cf1b31f381c89d3b791dc57009e386879535de Nov 01 16:08:47.462951 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:08:47.462918 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69a265d5_c5d6_4cfd_ae78_daad646b0c6d.slice/crio-7adfd52fad325f316b4a8a0a6a50b01bc0212303c997995c2d417014cf1e9181 WatchSource:0}: Error finding container 7adfd52fad325f316b4a8a0a6a50b01bc0212303c997995c2d417014cf1e9181: Status 404 returned error can't find the container with id 7adfd52fad325f316b4a8a0a6a50b01bc0212303c997995c2d417014cf1e9181 Nov 01 16:08:47.463823 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:47.463736 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6963-ba3e24"] Nov 01 16:08:47.850093 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:47.844048 4363 patch_prober.go:29] interesting pod/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 01 16:08:47.850093 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:47.844137 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" podUID="922af45f-b42b-47de-9724-6f2daa958330" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 01 16:08:48.009063 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:47.973308 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6b75-7478fffbb9-st6hq" event={"ID":"a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25","Type":"ContainerStarted","Data":"1c04ac4afba3883bed375df88bc96613170c4300de681f166cac80accfc048d6"} Nov 01 16:08:48.009063 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:47.973380 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6b75-7478fffbb9-st6hq" event={"ID":"a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25","Type":"ContainerStarted","Data":"6352fdf4fd6b691a239668d1d0cf1b31f381c89d3b791dc57009e386879535de"} Nov 01 16:08:48.009063 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:47.974655 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6b75-7478fffbb9-st6hq" Nov 01 16:08:48.027862 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:48.025267 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6963-ba3e24" event={"ID":"69a265d5-c5d6-4cfd-ae78-daad646b0c6d","Type":"ContainerStarted","Data":"180c6636e4e91ff98940f07c53c45ddd99d7a15c4eb91105ab0e76158af4f80f"} Nov 01 16:08:48.027862 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:48.025271 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6b75-7478fffbb9-st6hq" podStartSLOduration=3.507550994 podStartE2EDuration="4.025235142s" totalImagesPullingTime="517.684148ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:08:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:08:48.024938907 +0000 UTC m=+1261.356676557" watchObservedRunningTime="2026-11-01 16:08:48.025235142 +0000 UTC m=+1261.356972793" Nov 01 16:08:48.027862 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:48.025319 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6963-ba3e24" event={"ID":"69a265d5-c5d6-4cfd-ae78-daad646b0c6d","Type":"ContainerStarted","Data":"7adfd52fad325f316b4a8a0a6a50b01bc0212303c997995c2d417014cf1e9181"} Nov 01 16:08:48.041745 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:48.041670 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6963-ba3e24" podStartSLOduration=2.041653939 podStartE2EDuration="2.041653939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:08:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:08:48.038416239 +0000 UTC m=+1261.370153912" watchObservedRunningTime="2026-11-01 16:08:48.041653939 +0000 UTC m=+1261.373391589" Nov 01 16:08:48.374198 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:08:48.374160 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:08:48.841572 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:48.841537 4363 patch_prober.go:29] interesting pod/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 01 16:08:48.841959 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:48.841588 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" podUID="922af45f-b42b-47de-9724-6f2daa958330" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 01 16:08:48.947384 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:48.947305 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6963-ba3e24"] Nov 01 16:08:48.953173 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:48.953142 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6963-ba3e24"] Nov 01 16:08:49.155183 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:49.155152 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" Nov 01 16:08:49.864471 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:49.850952 4363 patch_prober.go:29] interesting pod/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 01 16:08:49.864471 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:49.851010 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" podUID="922af45f-b42b-47de-9724-6f2daa958330" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 01 16:08:50.082289 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:50.082100 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6963-ba3e24" podUID="69a265d5-c5d6-4cfd-ae78-daad646b0c6d" containerName="test-curl-6963-ba3e24" containerID="cri-o://180c6636e4e91ff98940f07c53c45ddd99d7a15c4eb91105ab0e76158af4f80f" gracePeriod=2 Nov 01 16:08:50.109535 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:08:50.109484 4363 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6963-ba3e24\" is forbidden: User \"system:node:ip-10-0-128-238.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-238.us-east-2.compute.internal' and this object" podUID="69a265d5-c5d6-4cfd-ae78-daad646b0c6d" pod="opendatahub/test-curl-6963-ba3e24" Nov 01 16:08:50.842101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:50.842055 4363 patch_prober.go:29] interesting pod/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 01 16:08:50.842313 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:50.842109 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" podUID="922af45f-b42b-47de-9724-6f2daa958330" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 01 16:08:52.070107 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:52.070071 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" Nov 01 16:08:52.073011 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:52.072981 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" Nov 01 16:08:52.858725 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:52.858692 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6963-ba3e24" Nov 01 16:08:52.935626 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:52.935590 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69a265d5-c5d6-4cfd-ae78-daad646b0c6d-kube-api-access-6tllb\" (UniqueName: \"kubernetes.io/projected/69a265d5-c5d6-4cfd-ae78-daad646b0c6d-kube-api-access-6tllb\") pod \"69a265d5-c5d6-4cfd-ae78-daad646b0c6d\" (UID: \"69a265d5-c5d6-4cfd-ae78-daad646b0c6d\") " Nov 01 16:08:52.937724 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:52.937694 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69a265d5-c5d6-4cfd-ae78-daad646b0c6d-kube-api-access-6tllb" pod "69a265d5-c5d6-4cfd-ae78-daad646b0c6d" (UID: "69a265d5-c5d6-4cfd-ae78-daad646b0c6d"). InnerVolumeSpecName "kube-api-access-6tllb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:08:53.038548 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:53.038517 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tllb\" (UniqueName: \"kubernetes.io/projected/69a265d5-c5d6-4cfd-ae78-daad646b0c6d-kube-api-access-6tllb\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:53.140758 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:53.140725 4363 generic.go:356] "Generic (PLEG): container finished" podID="69a265d5-c5d6-4cfd-ae78-daad646b0c6d" containerID="180c6636e4e91ff98940f07c53c45ddd99d7a15c4eb91105ab0e76158af4f80f" exitCode=137 Nov 01 16:08:53.141290 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:53.140834 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6963-ba3e24" Nov 01 16:08:53.141290 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:53.140881 4363 scope.go:122] "RemoveContainer" containerID="180c6636e4e91ff98940f07c53c45ddd99d7a15c4eb91105ab0e76158af4f80f" Nov 01 16:08:53.153218 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:08:53.153183 4363 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6963-ba3e24\" is forbidden: User \"system:node:ip-10-0-128-238.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-238.us-east-2.compute.internal' and this object" podUID="69a265d5-c5d6-4cfd-ae78-daad646b0c6d" pod="opendatahub/test-curl-6963-ba3e24" Nov 01 16:08:53.153883 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:53.153866 4363 scope.go:122] "RemoveContainer" containerID="180c6636e4e91ff98940f07c53c45ddd99d7a15c4eb91105ab0e76158af4f80f" Nov 01 16:08:53.154228 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:08:53.154208 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"180c6636e4e91ff98940f07c53c45ddd99d7a15c4eb91105ab0e76158af4f80f\": container with ID starting with 180c6636e4e91ff98940f07c53c45ddd99d7a15c4eb91105ab0e76158af4f80f not found: ID does not exist" containerID="180c6636e4e91ff98940f07c53c45ddd99d7a15c4eb91105ab0e76158af4f80f" Nov 01 16:08:53.154333 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:53.154235 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"180c6636e4e91ff98940f07c53c45ddd99d7a15c4eb91105ab0e76158af4f80f"} err="failed to get container status \"180c6636e4e91ff98940f07c53c45ddd99d7a15c4eb91105ab0e76158af4f80f\": rpc error: code = NotFound desc = could not find container \"180c6636e4e91ff98940f07c53c45ddd99d7a15c4eb91105ab0e76158af4f80f\": container with ID starting with 180c6636e4e91ff98940f07c53c45ddd99d7a15c4eb91105ab0e76158af4f80f not found: ID does not exist" Nov 01 16:08:54.871688 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:54.860210 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69a265d5-c5d6-4cfd-ae78-daad646b0c6d" path="/var/lib/kubelet/pods/69a265d5-c5d6-4cfd-ae78-daad646b0c6d/volumes" Nov 01 16:08:55.090469 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:55.090438 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6b75-7478fffbb9-st6hq" Nov 01 16:08:59.597168 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:59.597138 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-9p2wc"] Nov 01 16:08:59.618355 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:08:59.617356 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-rsvc9"] Nov 01 16:09:02.485420 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:02.485354 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-59d3a7-7d8d884767-mc7ft"] Nov 01 16:09:02.524443 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:02.524406 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-59d3a7-5c65656bb7-gbhbs"] Nov 01 16:09:04.598283 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:04.598241 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-9p2wc" podUID="2143f802-6cb6-409d-9015-84056431984b" containerName="payload-processing" containerID="cri-o://f2270ff47454d6163baf0602be60a8443767475b60f4b1c6e26d4fdc63836527" gracePeriod=25 Nov 01 16:09:04.618455 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:04.618418 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-rsvc9" podUID="be288817-6d19-47a1-a31c-7ee1c4946815" containerName="payload-pre-processing" containerID="cri-o://b0fe2ea799cbccef4323ad290ca30f27445c90c73f6230964ae650cf498081ce" gracePeriod=25 Nov 01 16:09:07.488450 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:07.488370 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-59d3a7-7d8d884767-mc7ft" podUID="78018b59-7aa6-4aa0-b243-edca76667e74" containerName="payload-processing" containerID="cri-o://bb057d61deeb549749194f8e82593da2d47e0cd1fddfd67fd87db4a908ddb2de" gracePeriod=25 Nov 01 16:09:07.532634 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:07.532186 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-59d3a7-5c65656bb7-gbhbs" podUID="4a407516-3c42-40a5-9f89-2c6934a1996f" containerName="payload-pre-processing" containerID="cri-o://d5f2ba997391283ccce2ca80f740881de5faebc1d41628d5cf8c98239834b7ea" gracePeriod=25 Nov 01 16:09:07.645739 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:07.645710 4363 generic.go:356] "Generic (PLEG): container finished" podID="78018b59-7aa6-4aa0-b243-edca76667e74" containerID="bb057d61deeb549749194f8e82593da2d47e0cd1fddfd67fd87db4a908ddb2de" exitCode=0 Nov 01 16:09:07.645879 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:07.645790 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-59d3a7-7d8d884767-mc7ft" event={"ID":"78018b59-7aa6-4aa0-b243-edca76667e74","Type":"ContainerDied","Data":"bb057d61deeb549749194f8e82593da2d47e0cd1fddfd67fd87db4a908ddb2de"} Nov 01 16:09:07.905077 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:07.905040 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-59d3a7-7d8d884767-mc7ft" Nov 01 16:09:07.961678 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:07.961645 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-59d3a7-5c65656bb7-gbhbs" Nov 01 16:09:08.042911 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:08.042873 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a407516-3c42-40a5-9f89-2c6934a1996f-kube-api-access-kjg84\" (UniqueName: \"kubernetes.io/projected/4a407516-3c42-40a5-9f89-2c6934a1996f-kube-api-access-kjg84\") pod \"4a407516-3c42-40a5-9f89-2c6934a1996f\" (UID: \"4a407516-3c42-40a5-9f89-2c6934a1996f\") " Nov 01 16:09:08.043160 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:08.043135 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78018b59-7aa6-4aa0-b243-edca76667e74-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78018b59-7aa6-4aa0-b243-edca76667e74-plugins-config-volume\") pod \"78018b59-7aa6-4aa0-b243-edca76667e74\" (UID: \"78018b59-7aa6-4aa0-b243-edca76667e74\") " Nov 01 16:09:08.043292 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:08.043274 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78018b59-7aa6-4aa0-b243-edca76667e74-kube-api-access-49v5x\" (UniqueName: \"kubernetes.io/projected/78018b59-7aa6-4aa0-b243-edca76667e74-kube-api-access-49v5x\") pod \"78018b59-7aa6-4aa0-b243-edca76667e74\" (UID: \"78018b59-7aa6-4aa0-b243-edca76667e74\") " Nov 01 16:09:08.043735 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:08.043411 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a407516-3c42-40a5-9f89-2c6934a1996f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a407516-3c42-40a5-9f89-2c6934a1996f-plugins-config-volume\") pod \"4a407516-3c42-40a5-9f89-2c6934a1996f\" (UID: \"4a407516-3c42-40a5-9f89-2c6934a1996f\") " Nov 01 16:09:08.043735 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:08.043608 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78018b59-7aa6-4aa0-b243-edca76667e74-plugins-config-volume" pod "78018b59-7aa6-4aa0-b243-edca76667e74" (UID: "78018b59-7aa6-4aa0-b243-edca76667e74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:09:08.043992 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:08.043966 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a407516-3c42-40a5-9f89-2c6934a1996f-plugins-config-volume" pod "4a407516-3c42-40a5-9f89-2c6934a1996f" (UID: "4a407516-3c42-40a5-9f89-2c6934a1996f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:09:08.044443 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:08.044418 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a407516-3c42-40a5-9f89-2c6934a1996f-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:08.044550 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:08.044444 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78018b59-7aa6-4aa0-b243-edca76667e74-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:08.045793 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:08.045764 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a407516-3c42-40a5-9f89-2c6934a1996f-kube-api-access-kjg84" pod "4a407516-3c42-40a5-9f89-2c6934a1996f" (UID: "4a407516-3c42-40a5-9f89-2c6934a1996f"). InnerVolumeSpecName "kube-api-access-kjg84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:09:08.046357 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:08.046328 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78018b59-7aa6-4aa0-b243-edca76667e74-kube-api-access-49v5x" pod "78018b59-7aa6-4aa0-b243-edca76667e74" (UID: "78018b59-7aa6-4aa0-b243-edca76667e74"). InnerVolumeSpecName "kube-api-access-49v5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:09:08.167082 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:08.161826 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49v5x\" (UniqueName: \"kubernetes.io/projected/78018b59-7aa6-4aa0-b243-edca76667e74-kube-api-access-49v5x\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:08.167082 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:08.161868 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjg84\" (UniqueName: \"kubernetes.io/projected/4a407516-3c42-40a5-9f89-2c6934a1996f-kube-api-access-kjg84\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:08.654520 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:08.654489 4363 generic.go:356] "Generic (PLEG): container finished" podID="4a407516-3c42-40a5-9f89-2c6934a1996f" containerID="d5f2ba997391283ccce2ca80f740881de5faebc1d41628d5cf8c98239834b7ea" exitCode=0 Nov 01 16:09:08.655060 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:08.654533 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-59d3a7-5c65656bb7-gbhbs" Nov 01 16:09:08.655060 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:08.654571 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-59d3a7-5c65656bb7-gbhbs" event={"ID":"4a407516-3c42-40a5-9f89-2c6934a1996f","Type":"ContainerDied","Data":"d5f2ba997391283ccce2ca80f740881de5faebc1d41628d5cf8c98239834b7ea"} Nov 01 16:09:08.655060 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:08.654633 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-59d3a7-5c65656bb7-gbhbs" event={"ID":"4a407516-3c42-40a5-9f89-2c6934a1996f","Type":"ContainerDied","Data":"32605e86e5b9c0dacbf9d5daf0ddeb73c1b6baf7084b6b686c39328dd0d3ceb5"} Nov 01 16:09:08.655060 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:08.654662 4363 scope.go:122] "RemoveContainer" containerID="d5f2ba997391283ccce2ca80f740881de5faebc1d41628d5cf8c98239834b7ea" Nov 01 16:09:08.656738 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:08.656717 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-59d3a7-7d8d884767-mc7ft" Nov 01 16:09:08.656855 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:08.656769 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-59d3a7-7d8d884767-mc7ft" event={"ID":"78018b59-7aa6-4aa0-b243-edca76667e74","Type":"ContainerDied","Data":"9545f9f72db7f08c24e8321f99e1748bee353e48e8aacbb5e38907e6e2d6ad06"} Nov 01 16:09:08.666758 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:08.666739 4363 scope.go:122] "RemoveContainer" containerID="d5f2ba997391283ccce2ca80f740881de5faebc1d41628d5cf8c98239834b7ea" Nov 01 16:09:08.667296 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:09:08.667130 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5f2ba997391283ccce2ca80f740881de5faebc1d41628d5cf8c98239834b7ea\": container with ID starting with d5f2ba997391283ccce2ca80f740881de5faebc1d41628d5cf8c98239834b7ea not found: ID does not exist" containerID="d5f2ba997391283ccce2ca80f740881de5faebc1d41628d5cf8c98239834b7ea" Nov 01 16:09:08.667296 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:08.667170 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5f2ba997391283ccce2ca80f740881de5faebc1d41628d5cf8c98239834b7ea"} err="failed to get container status \"d5f2ba997391283ccce2ca80f740881de5faebc1d41628d5cf8c98239834b7ea\": rpc error: code = NotFound desc = could not find container \"d5f2ba997391283ccce2ca80f740881de5faebc1d41628d5cf8c98239834b7ea\": container with ID starting with d5f2ba997391283ccce2ca80f740881de5faebc1d41628d5cf8c98239834b7ea not found: ID does not exist" Nov 01 16:09:08.667296 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:08.667195 4363 scope.go:122] "RemoveContainer" containerID="bb057d61deeb549749194f8e82593da2d47e0cd1fddfd67fd87db4a908ddb2de" Nov 01 16:09:08.684607 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:08.684580 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-59d3a7-5c65656bb7-gbhbs"] Nov 01 16:09:08.689292 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:08.689269 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-59d3a7-5c65656bb7-gbhbs"] Nov 01 16:09:08.697496 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:08.697471 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-59d3a7-7d8d884767-mc7ft"] Nov 01 16:09:08.701910 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:08.701887 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-59d3a7-7d8d884767-mc7ft"] Nov 01 16:09:08.804376 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:08.804292 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a407516-3c42-40a5-9f89-2c6934a1996f" path="/var/lib/kubelet/pods/4a407516-3c42-40a5-9f89-2c6934a1996f/volumes" Nov 01 16:09:08.804878 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:08.804862 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78018b59-7aa6-4aa0-b243-edca76667e74" path="/var/lib/kubelet/pods/78018b59-7aa6-4aa0-b243-edca76667e74/volumes" Nov 01 16:09:12.775540 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:12.775500 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-9p2wc" podUID="2143f802-6cb6-409d-9015-84056431984b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:09:12.776112 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:12.775973 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-rsvc9" podUID="be288817-6d19-47a1-a31c-7ee1c4946815" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:09:15.006042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:15.001404 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt"] Nov 01 16:09:15.006042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:15.001650 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" podUID="07f5c387-a86b-4118-bf29-dad1f686db58" containerName="istio-proxy" containerID="cri-o://481fcb02af97305e92c5bc5a150e826137f71145b69c869d4be82e2f085555d6" gracePeriod=30 Nov 01 16:09:16.452043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:16.426777 4363 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:09:16.452043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:16.426855 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" podUID="07f5c387-a86b-4118-bf29-dad1f686db58" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:09:18.329503 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:09:18.329446 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:09:21.048040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:21.023239 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" podUID="f1e164ab-e940-44e0-8829-8626e4162a9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.115:8000/ready\": context deadline exceeded" Nov 01 16:09:21.486955 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:21.481155 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": context deadline exceeded" Nov 01 16:09:21.570973 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:21.569156 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" Nov 01 16:09:21.597153 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:21.597095 4363 generic.go:356] "Generic (PLEG): container finished" podID="07f5c387-a86b-4118-bf29-dad1f686db58" containerID="481fcb02af97305e92c5bc5a150e826137f71145b69c869d4be82e2f085555d6" exitCode=0 Nov 01 16:09:21.601128 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:21.600279 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:09:21.719040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:21.695183 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07f5c387-a86b-4118-bf29-dad1f686db58-kube-api-access-6djgw\" (UniqueName: \"kubernetes.io/projected/07f5c387-a86b-4118-bf29-dad1f686db58-kube-api-access-6djgw\") pod \"07f5c387-a86b-4118-bf29-dad1f686db58\" (UID: \"07f5c387-a86b-4118-bf29-dad1f686db58\") " Nov 01 16:09:21.884162 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:21.884130 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07f5c387-a86b-4118-bf29-dad1f686db58-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07f5c387-a86b-4118-bf29-dad1f686db58-workload-socket\") pod \"07f5c387-a86b-4118-bf29-dad1f686db58\" (UID: \"07f5c387-a86b-4118-bf29-dad1f686db58\") " Nov 01 16:09:21.885964 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:21.885938 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07f5c387-a86b-4118-bf29-dad1f686db58-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07f5c387-a86b-4118-bf29-dad1f686db58-workload-certs\") pod \"07f5c387-a86b-4118-bf29-dad1f686db58\" (UID: \"07f5c387-a86b-4118-bf29-dad1f686db58\") " Nov 01 16:09:21.888385 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:21.888364 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07f5c387-a86b-4118-bf29-dad1f686db58-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07f5c387-a86b-4118-bf29-dad1f686db58-istio-envoy\") pod \"07f5c387-a86b-4118-bf29-dad1f686db58\" (UID: \"07f5c387-a86b-4118-bf29-dad1f686db58\") " Nov 01 16:09:21.888872 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:21.888852 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07f5c387-a86b-4118-bf29-dad1f686db58-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07f5c387-a86b-4118-bf29-dad1f686db58-istio-podinfo\") pod \"07f5c387-a86b-4118-bf29-dad1f686db58\" (UID: \"07f5c387-a86b-4118-bf29-dad1f686db58\") " Nov 01 16:09:21.945042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:21.928455 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07f5c387-a86b-4118-bf29-dad1f686db58-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07f5c387-a86b-4118-bf29-dad1f686db58-istiod-ca-cert\") pod \"07f5c387-a86b-4118-bf29-dad1f686db58\" (UID: \"07f5c387-a86b-4118-bf29-dad1f686db58\") " Nov 01 16:09:21.945042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:21.928895 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07f5c387-a86b-4118-bf29-dad1f686db58-istio-token\" (UniqueName: \"kubernetes.io/projected/07f5c387-a86b-4118-bf29-dad1f686db58-istio-token\") pod \"07f5c387-a86b-4118-bf29-dad1f686db58\" (UID: \"07f5c387-a86b-4118-bf29-dad1f686db58\") " Nov 01 16:09:21.945042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:21.928944 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07f5c387-a86b-4118-bf29-dad1f686db58-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07f5c387-a86b-4118-bf29-dad1f686db58-istio-data\") pod \"07f5c387-a86b-4118-bf29-dad1f686db58\" (UID: \"07f5c387-a86b-4118-bf29-dad1f686db58\") " Nov 01 16:09:21.945042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:21.928982 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07f5c387-a86b-4118-bf29-dad1f686db58-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07f5c387-a86b-4118-bf29-dad1f686db58-credential-socket\") pod \"07f5c387-a86b-4118-bf29-dad1f686db58\" (UID: \"07f5c387-a86b-4118-bf29-dad1f686db58\") " Nov 01 16:09:21.958215 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:21.884899 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07f5c387-a86b-4118-bf29-dad1f686db58-workload-socket" pod "07f5c387-a86b-4118-bf29-dad1f686db58" (UID: "07f5c387-a86b-4118-bf29-dad1f686db58"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:09:21.958215 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:21.888309 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07f5c387-a86b-4118-bf29-dad1f686db58-workload-certs" pod "07f5c387-a86b-4118-bf29-dad1f686db58" (UID: "07f5c387-a86b-4118-bf29-dad1f686db58"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:09:21.958215 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:21.955885 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07f5c387-a86b-4118-bf29-dad1f686db58-istiod-ca-cert" pod "07f5c387-a86b-4118-bf29-dad1f686db58" (UID: "07f5c387-a86b-4118-bf29-dad1f686db58"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:09:21.958215 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:21.787124 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:09:22.000307 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:21.959824 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07f5c387-a86b-4118-bf29-dad1f686db58-istio-podinfo" pod "07f5c387-a86b-4118-bf29-dad1f686db58" (UID: "07f5c387-a86b-4118-bf29-dad1f686db58"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:09:22.069042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.060154 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07f5c387-a86b-4118-bf29-dad1f686db58-istio-data" pod "07f5c387-a86b-4118-bf29-dad1f686db58" (UID: "07f5c387-a86b-4118-bf29-dad1f686db58"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:09:22.081647 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.078272 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07f5c387-a86b-4118-bf29-dad1f686db58-credential-socket" pod "07f5c387-a86b-4118-bf29-dad1f686db58" (UID: "07f5c387-a86b-4118-bf29-dad1f686db58"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:09:22.093888 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.093771 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07f5c387-a86b-4118-bf29-dad1f686db58-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:22.093888 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.093804 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07f5c387-a86b-4118-bf29-dad1f686db58-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:22.093888 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.093820 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07f5c387-a86b-4118-bf29-dad1f686db58-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:22.093888 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.093834 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07f5c387-a86b-4118-bf29-dad1f686db58-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:22.093888 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.093848 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07f5c387-a86b-4118-bf29-dad1f686db58-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:22.093888 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.093863 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07f5c387-a86b-4118-bf29-dad1f686db58-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:22.126702 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.122225 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07f5c387-a86b-4118-bf29-dad1f686db58-istio-envoy" pod "07f5c387-a86b-4118-bf29-dad1f686db58" (UID: "07f5c387-a86b-4118-bf29-dad1f686db58"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:09:22.370257 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:09:22.367927 4363 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.578s" Nov 01 16:09:22.370257 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.368149 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" event={"ID":"07f5c387-a86b-4118-bf29-dad1f686db58","Type":"ContainerDied","Data":"481fcb02af97305e92c5bc5a150e826137f71145b69c869d4be82e2f085555d6"} Nov 01 16:09:22.370257 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.368210 4363 scope.go:122] "RemoveContainer" containerID="481fcb02af97305e92c5bc5a150e826137f71145b69c869d4be82e2f085555d6" Nov 01 16:09:22.375116 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.375049 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07f5c387-a86b-4118-bf29-dad1f686db58-kube-api-access-6djgw" pod "07f5c387-a86b-4118-bf29-dad1f686db58" (UID: "07f5c387-a86b-4118-bf29-dad1f686db58"). InnerVolumeSpecName "kube-api-access-6djgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:09:22.407118 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.407079 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07f5c387-a86b-4118-bf29-dad1f686db58-istio-token" pod "07f5c387-a86b-4118-bf29-dad1f686db58" (UID: "07f5c387-a86b-4118-bf29-dad1f686db58"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:09:22.457323 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.457260 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph"] Nov 01 16:09:22.467643 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.467605 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6djgw\" (UniqueName: \"kubernetes.io/projected/07f5c387-a86b-4118-bf29-dad1f686db58-kube-api-access-6djgw\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:22.467822 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.467660 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07f5c387-a86b-4118-bf29-dad1f686db58-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:22.468698 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.468668 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07f5c387-a86b-4118-bf29-dad1f686db58-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:22.497229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.495872 4363 cpu_manager.go:414] "containerMap: removing container" podUID="4a407516-3c42-40a5-9f89-2c6934a1996f" containerName="payload-pre-processing" Nov 01 16:09:22.497229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.495904 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a407516-3c42-40a5-9f89-2c6934a1996f" containerName="payload-pre-processing" Nov 01 16:09:22.497229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.495999 4363 cpu_manager.go:414] "containerMap: removing container" podUID="78018b59-7aa6-4aa0-b243-edca76667e74" containerName="payload-processing" Nov 01 16:09:22.497229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.496007 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78018b59-7aa6-4aa0-b243-edca76667e74" containerName="payload-processing" Nov 01 16:09:22.497229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.496108 4363 cpu_manager.go:414] "containerMap: removing container" podUID="69a265d5-c5d6-4cfd-ae78-daad646b0c6d" containerName="test-curl-6963-ba3e24" Nov 01 16:09:22.497229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.496118 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69a265d5-c5d6-4cfd-ae78-daad646b0c6d" containerName="test-curl-6963-ba3e24" Nov 01 16:09:22.497229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.496152 4363 cpu_manager.go:414] "containerMap: removing container" podUID="07f5c387-a86b-4118-bf29-dad1f686db58" containerName="istio-proxy" Nov 01 16:09:22.497229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.496160 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07f5c387-a86b-4118-bf29-dad1f686db58" containerName="istio-proxy" Nov 01 16:09:22.497229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.496648 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="69a265d5-c5d6-4cfd-ae78-daad646b0c6d" containerName="test-curl-6963-ba3e24" Nov 01 16:09:22.497229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.496708 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="07f5c387-a86b-4118-bf29-dad1f686db58" containerName="istio-proxy" Nov 01 16:09:22.497229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.496740 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a407516-3c42-40a5-9f89-2c6934a1996f" containerName="payload-pre-processing" Nov 01 16:09:22.497229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.496757 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="78018b59-7aa6-4aa0-b243-edca76667e74" containerName="payload-processing" Nov 01 16:09:22.513807 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.513773 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph"] Nov 01 16:09:22.513978 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.513937 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" Nov 01 16:09:22.521696 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.521663 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-80c4c8-openshift-default-dockercfg-rmtgb\"" Nov 01 16:09:22.575974 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.570353 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxtnd\" (UniqueName: \"kubernetes.io/projected/a77a07cf-0f15-4724-8170-da59b2a1e5f7-kube-api-access-hxtnd\") pod \"e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph\" (UID: \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\") " pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" Nov 01 16:09:22.575974 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.570547 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a77a07cf-0f15-4724-8170-da59b2a1e5f7-istio-envoy\") pod \"e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph\" (UID: \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\") " pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" Nov 01 16:09:22.575974 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.570575 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a77a07cf-0f15-4724-8170-da59b2a1e5f7-istio-podinfo\") pod \"e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph\" (UID: \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\") " pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" Nov 01 16:09:22.575974 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.571441 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77a07cf-0f15-4724-8170-da59b2a1e5f7-credential-socket\") pod \"e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph\" (UID: \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\") " pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" Nov 01 16:09:22.575974 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.571481 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a77a07cf-0f15-4724-8170-da59b2a1e5f7-istio-data\") pod \"e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph\" (UID: \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\") " pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" Nov 01 16:09:22.575974 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.571507 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a77a07cf-0f15-4724-8170-da59b2a1e5f7-istio-token\") pod \"e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph\" (UID: \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\") " pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" Nov 01 16:09:22.575974 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.571674 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77a07cf-0f15-4724-8170-da59b2a1e5f7-workload-socket\") pod \"e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph\" (UID: \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\") " pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" Nov 01 16:09:22.575974 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.571719 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a77a07cf-0f15-4724-8170-da59b2a1e5f7-workload-certs\") pod \"e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph\" (UID: \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\") " pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" Nov 01 16:09:22.575974 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.571784 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a77a07cf-0f15-4724-8170-da59b2a1e5f7-istiod-ca-cert\") pod \"e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph\" (UID: \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\") " pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" Nov 01 16:09:22.626148 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.625769 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" event={"ID":"07f5c387-a86b-4118-bf29-dad1f686db58","Type":"ContainerDied","Data":"255bfbe9db72ad5d4c3e2bb935980f3e002244ae055d15a9feeb875f875172a1"} Nov 01 16:09:22.626148 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.625937 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt" Nov 01 16:09:22.666994 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.666952 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt"] Nov 01 16:09:22.672179 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.672152 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-kv5zt"] Nov 01 16:09:22.673390 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.673362 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77a07cf-0f15-4724-8170-da59b2a1e5f7-workload-socket\") pod \"e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph\" (UID: \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\") " pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" Nov 01 16:09:22.673461 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.673424 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a77a07cf-0f15-4724-8170-da59b2a1e5f7-workload-certs\") pod \"e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph\" (UID: \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\") " pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" Nov 01 16:09:22.673508 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.673466 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a77a07cf-0f15-4724-8170-da59b2a1e5f7-istiod-ca-cert\") pod \"e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph\" (UID: \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\") " pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" Nov 01 16:09:22.673709 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.673684 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxtnd\" (UniqueName: \"kubernetes.io/projected/a77a07cf-0f15-4724-8170-da59b2a1e5f7-kube-api-access-hxtnd\") pod \"e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph\" (UID: \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\") " pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" Nov 01 16:09:22.673867 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.673794 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77a07cf-0f15-4724-8170-da59b2a1e5f7-workload-socket\") pod \"e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph\" (UID: \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\") " pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" Nov 01 16:09:22.673867 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.673830 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a77a07cf-0f15-4724-8170-da59b2a1e5f7-workload-certs\") pod \"e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph\" (UID: \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\") " pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" Nov 01 16:09:22.673972 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.673916 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a77a07cf-0f15-4724-8170-da59b2a1e5f7-istio-envoy\") pod \"e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph\" (UID: \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\") " pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" Nov 01 16:09:22.673972 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.673942 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a77a07cf-0f15-4724-8170-da59b2a1e5f7-istio-podinfo\") pod \"e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph\" (UID: \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\") " pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" Nov 01 16:09:22.674137 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.674055 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77a07cf-0f15-4724-8170-da59b2a1e5f7-credential-socket\") pod \"e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph\" (UID: \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\") " pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" Nov 01 16:09:22.674137 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.674081 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a77a07cf-0f15-4724-8170-da59b2a1e5f7-istio-data\") pod \"e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph\" (UID: \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\") " pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" Nov 01 16:09:22.674137 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.674106 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a77a07cf-0f15-4724-8170-da59b2a1e5f7-istio-token\") pod \"e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph\" (UID: \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\") " pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" Nov 01 16:09:22.674325 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.674304 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a77a07cf-0f15-4724-8170-da59b2a1e5f7-istiod-ca-cert\") pod \"e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph\" (UID: \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\") " pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" Nov 01 16:09:22.674487 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.674455 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a77a07cf-0f15-4724-8170-da59b2a1e5f7-istio-data\") pod \"e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph\" (UID: \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\") " pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" Nov 01 16:09:22.674771 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.674750 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77a07cf-0f15-4724-8170-da59b2a1e5f7-credential-socket\") pod \"e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph\" (UID: \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\") " pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" Nov 01 16:09:22.676324 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.676293 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a77a07cf-0f15-4724-8170-da59b2a1e5f7-istio-envoy\") pod \"e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph\" (UID: \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\") " pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" Nov 01 16:09:22.676454 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.676408 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a77a07cf-0f15-4724-8170-da59b2a1e5f7-istio-podinfo\") pod \"e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph\" (UID: \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\") " pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" Nov 01 16:09:22.690060 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.682846 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxtnd\" (UniqueName: \"kubernetes.io/projected/a77a07cf-0f15-4724-8170-da59b2a1e5f7-kube-api-access-hxtnd\") pod \"e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph\" (UID: \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\") " pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" Nov 01 16:09:22.690060 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.684173 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a77a07cf-0f15-4724-8170-da59b2a1e5f7-istio-token\") pod \"e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph\" (UID: \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\") " pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" Nov 01 16:09:22.777479 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.777394 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-rsvc9" podUID="be288817-6d19-47a1-a31c-7ee1c4946815" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:09:22.777619 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.777404 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-9p2wc" podUID="2143f802-6cb6-409d-9015-84056431984b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:09:22.803219 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.803185 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07f5c387-a86b-4118-bf29-dad1f686db58" path="/var/lib/kubelet/pods/07f5c387-a86b-4118-bf29-dad1f686db58/volumes" Nov 01 16:09:22.838915 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:22.838886 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" Nov 01 16:09:23.214268 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:23.201387 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:09:23.214268 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:23.201487 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:09:23.214268 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:23.201530 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:09:23.214268 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:23.210847 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph"] Nov 01 16:09:23.639337 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:23.639298 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" event={"ID":"a77a07cf-0f15-4724-8170-da59b2a1e5f7","Type":"ContainerStarted","Data":"0285c96e956edbf8d64bb79c684597d677be6dfff08e15028b990bad54fcdbd6"} Nov 01 16:09:23.639526 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:23.639388 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" event={"ID":"a77a07cf-0f15-4724-8170-da59b2a1e5f7","Type":"ContainerStarted","Data":"6ac0e2455fb12c304350247a81450487db355c9996c21de1f88cd99bf45529d4"} Nov 01 16:09:23.660547 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:23.660488 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" podStartSLOduration=17.660465937 podStartE2EDuration="17.660465937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:09:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:09:23.659145339 +0000 UTC m=+1296.990882988" watchObservedRunningTime="2026-11-01 16:09:23.660465937 +0000 UTC m=+1296.992203588" Nov 01 16:09:23.843319 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:23.843286 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" Nov 01 16:09:25.015040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:25.014875 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" Nov 01 16:09:25.141880 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:25.141799 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-9p2wc" Nov 01 16:09:25.197565 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:25.197534 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-rsvc9" Nov 01 16:09:25.235811 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:25.235780 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2143f802-6cb6-409d-9015-84056431984b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2143f802-6cb6-409d-9015-84056431984b-plugins-config-volume\") pod \"2143f802-6cb6-409d-9015-84056431984b\" (UID: \"2143f802-6cb6-409d-9015-84056431984b\") " Nov 01 16:09:25.235980 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:25.235870 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2143f802-6cb6-409d-9015-84056431984b-kube-api-access-x2vkk\" (UniqueName: \"kubernetes.io/projected/2143f802-6cb6-409d-9015-84056431984b-kube-api-access-x2vkk\") pod \"2143f802-6cb6-409d-9015-84056431984b\" (UID: \"2143f802-6cb6-409d-9015-84056431984b\") " Nov 01 16:09:25.236355 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:25.236320 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2143f802-6cb6-409d-9015-84056431984b-plugins-config-volume" pod "2143f802-6cb6-409d-9015-84056431984b" (UID: "2143f802-6cb6-409d-9015-84056431984b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:09:25.236923 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:25.236896 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2143f802-6cb6-409d-9015-84056431984b-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:25.238296 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:25.238269 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2143f802-6cb6-409d-9015-84056431984b-kube-api-access-x2vkk" pod "2143f802-6cb6-409d-9015-84056431984b" (UID: "2143f802-6cb6-409d-9015-84056431984b"). InnerVolumeSpecName "kube-api-access-x2vkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:09:25.338763 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:25.338695 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be288817-6d19-47a1-a31c-7ee1c4946815-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be288817-6d19-47a1-a31c-7ee1c4946815-plugins-config-volume\") pod \"be288817-6d19-47a1-a31c-7ee1c4946815\" (UID: \"be288817-6d19-47a1-a31c-7ee1c4946815\") " Nov 01 16:09:25.338955 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:25.338839 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be288817-6d19-47a1-a31c-7ee1c4946815-kube-api-access-dtg6p\" (UniqueName: \"kubernetes.io/projected/be288817-6d19-47a1-a31c-7ee1c4946815-kube-api-access-dtg6p\") pod \"be288817-6d19-47a1-a31c-7ee1c4946815\" (UID: \"be288817-6d19-47a1-a31c-7ee1c4946815\") " Nov 01 16:09:25.339224 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:25.339195 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be288817-6d19-47a1-a31c-7ee1c4946815-plugins-config-volume" pod "be288817-6d19-47a1-a31c-7ee1c4946815" (UID: "be288817-6d19-47a1-a31c-7ee1c4946815"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:09:25.339734 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:25.339706 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be288817-6d19-47a1-a31c-7ee1c4946815-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:25.339839 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:25.339738 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2vkk\" (UniqueName: \"kubernetes.io/projected/2143f802-6cb6-409d-9015-84056431984b-kube-api-access-x2vkk\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:25.341182 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:25.341151 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be288817-6d19-47a1-a31c-7ee1c4946815-kube-api-access-dtg6p" pod "be288817-6d19-47a1-a31c-7ee1c4946815" (UID: "be288817-6d19-47a1-a31c-7ee1c4946815"). InnerVolumeSpecName "kube-api-access-dtg6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:09:25.441600 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:25.441515 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtg6p\" (UniqueName: \"kubernetes.io/projected/be288817-6d19-47a1-a31c-7ee1c4946815-kube-api-access-dtg6p\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:25.655717 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:25.655687 4363 generic.go:356] "Generic (PLEG): container finished" podID="2143f802-6cb6-409d-9015-84056431984b" containerID="f2270ff47454d6163baf0602be60a8443767475b60f4b1c6e26d4fdc63836527" exitCode=0 Nov 01 16:09:25.655876 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:25.655743 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-9p2wc" Nov 01 16:09:25.655876 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:25.655778 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-9p2wc" event={"ID":"2143f802-6cb6-409d-9015-84056431984b","Type":"ContainerDied","Data":"f2270ff47454d6163baf0602be60a8443767475b60f4b1c6e26d4fdc63836527"} Nov 01 16:09:25.655876 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:25.655850 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-9p2wc" event={"ID":"2143f802-6cb6-409d-9015-84056431984b","Type":"ContainerDied","Data":"de379ebc012df6ab08094249daefce0118f68b53ee338e0aea4f2adcf0775919"} Nov 01 16:09:25.655876 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:25.655872 4363 scope.go:122] "RemoveContainer" containerID="f2270ff47454d6163baf0602be60a8443767475b60f4b1c6e26d4fdc63836527" Nov 01 16:09:25.657497 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:25.657458 4363 generic.go:356] "Generic (PLEG): container finished" podID="be288817-6d19-47a1-a31c-7ee1c4946815" containerID="b0fe2ea799cbccef4323ad290ca30f27445c90c73f6230964ae650cf498081ce" exitCode=0 Nov 01 16:09:25.657893 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:25.657500 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-rsvc9" Nov 01 16:09:25.657893 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:25.657536 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-rsvc9" event={"ID":"be288817-6d19-47a1-a31c-7ee1c4946815","Type":"ContainerDied","Data":"b0fe2ea799cbccef4323ad290ca30f27445c90c73f6230964ae650cf498081ce"} Nov 01 16:09:25.657893 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:25.657589 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-rsvc9" event={"ID":"be288817-6d19-47a1-a31c-7ee1c4946815","Type":"ContainerDied","Data":"f1cddc1770522cf18134a31b522b78c7a09bd142a14e441c24c2d3c53f53a29d"} Nov 01 16:09:25.657893 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:25.657890 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" Nov 01 16:09:25.659394 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:25.659363 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" Nov 01 16:09:25.673113 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:25.673088 4363 scope.go:122] "RemoveContainer" containerID="f2270ff47454d6163baf0602be60a8443767475b60f4b1c6e26d4fdc63836527" Nov 01 16:09:25.673671 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:09:25.673637 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2270ff47454d6163baf0602be60a8443767475b60f4b1c6e26d4fdc63836527\": container with ID starting with f2270ff47454d6163baf0602be60a8443767475b60f4b1c6e26d4fdc63836527 not found: ID does not exist" containerID="f2270ff47454d6163baf0602be60a8443767475b60f4b1c6e26d4fdc63836527" Nov 01 16:09:25.673671 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:25.673686 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2270ff47454d6163baf0602be60a8443767475b60f4b1c6e26d4fdc63836527"} err="failed to get container status \"f2270ff47454d6163baf0602be60a8443767475b60f4b1c6e26d4fdc63836527\": rpc error: code = NotFound desc = could not find container \"f2270ff47454d6163baf0602be60a8443767475b60f4b1c6e26d4fdc63836527\": container with ID starting with f2270ff47454d6163baf0602be60a8443767475b60f4b1c6e26d4fdc63836527 not found: ID does not exist" Nov 01 16:09:25.673881 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:25.673714 4363 scope.go:122] "RemoveContainer" containerID="b0fe2ea799cbccef4323ad290ca30f27445c90c73f6230964ae650cf498081ce" Nov 01 16:09:25.731118 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:25.727210 4363 scope.go:122] "RemoveContainer" containerID="b0fe2ea799cbccef4323ad290ca30f27445c90c73f6230964ae650cf498081ce" Nov 01 16:09:25.774495 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:09:25.739234 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0fe2ea799cbccef4323ad290ca30f27445c90c73f6230964ae650cf498081ce\": container with ID starting with b0fe2ea799cbccef4323ad290ca30f27445c90c73f6230964ae650cf498081ce not found: ID does not exist" containerID="b0fe2ea799cbccef4323ad290ca30f27445c90c73f6230964ae650cf498081ce" Nov 01 16:09:25.774495 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:25.739284 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0fe2ea799cbccef4323ad290ca30f27445c90c73f6230964ae650cf498081ce"} err="failed to get container status \"b0fe2ea799cbccef4323ad290ca30f27445c90c73f6230964ae650cf498081ce\": rpc error: code = NotFound desc = could not find container \"b0fe2ea799cbccef4323ad290ca30f27445c90c73f6230964ae650cf498081ce\": container with ID starting with b0fe2ea799cbccef4323ad290ca30f27445c90c73f6230964ae650cf498081ce not found: ID does not exist" Nov 01 16:09:25.960738 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:25.960677 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-9p2wc"] Nov 01 16:09:25.975927 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:25.975812 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-9p2wc"] Nov 01 16:09:25.983481 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:25.983442 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-rsvc9"] Nov 01 16:09:25.988237 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:25.988210 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-rsvc9"] Nov 01 16:09:26.802774 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:26.802737 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2143f802-6cb6-409d-9015-84056431984b" path="/var/lib/kubelet/pods/2143f802-6cb6-409d-9015-84056431984b/volumes" Nov 01 16:09:26.803329 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:26.803313 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be288817-6d19-47a1-a31c-7ee1c4946815" path="/var/lib/kubelet/pods/be288817-6d19-47a1-a31c-7ee1c4946815/volumes" Nov 01 16:09:27.208865 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:27.208834 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-80c4c8-75974fbc49-zqjkh"] Nov 01 16:09:27.209835 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:27.209812 4363 cpu_manager.go:414] "containerMap: removing container" podUID="2143f802-6cb6-409d-9015-84056431984b" containerName="payload-processing" Nov 01 16:09:27.209835 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:27.209832 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2143f802-6cb6-409d-9015-84056431984b" containerName="payload-processing" Nov 01 16:09:27.210072 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:27.209848 4363 cpu_manager.go:414] "containerMap: removing container" podUID="be288817-6d19-47a1-a31c-7ee1c4946815" containerName="payload-pre-processing" Nov 01 16:09:27.210072 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:27.209853 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be288817-6d19-47a1-a31c-7ee1c4946815" containerName="payload-pre-processing" Nov 01 16:09:27.210188 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:27.210103 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="2143f802-6cb6-409d-9015-84056431984b" containerName="payload-processing" Nov 01 16:09:27.210188 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:27.210120 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="be288817-6d19-47a1-a31c-7ee1c4946815" containerName="payload-pre-processing" Nov 01 16:09:27.215709 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:27.215687 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-80c4c8-75974fbc49-zqjkh" Nov 01 16:09:27.220937 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:27.220909 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-80c4c8\"" Nov 01 16:09:27.226388 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:27.224036 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-80c4c8-75974fbc49-zqjkh"] Nov 01 16:09:27.232683 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:27.232658 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-80c4c8-5779b74587-k4gkg"] Nov 01 16:09:27.248625 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:27.247333 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-80c4c8-5779b74587-k4gkg" Nov 01 16:09:27.258510 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:27.250886 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-80c4c8-dockercfg-hwh76\"" Nov 01 16:09:27.259500 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:27.259466 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-80c4c8-5779b74587-k4gkg"] Nov 01 16:09:27.326410 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:27.326369 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hksd6\" (UniqueName: \"kubernetes.io/projected/0d6b57c1-aa75-406f-aa95-f455f8690078-kube-api-access-hksd6\") pod \"payload-pre-processing-e2e-api-b-80c4c8-5779b74587-k4gkg\" (UID: \"0d6b57c1-aa75-406f-aa95-f455f8690078\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-80c4c8-5779b74587-k4gkg" Nov 01 16:09:27.327224 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:27.326492 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnc7b\" (UniqueName: \"kubernetes.io/projected/568c7dd7-469a-42a0-bb33-85167782b280-kube-api-access-xnc7b\") pod \"payload-processing-e2e-api-b-80c4c8-75974fbc49-zqjkh\" (UID: \"568c7dd7-469a-42a0-bb33-85167782b280\") " pod="openshift-ingress/payload-processing-e2e-api-b-80c4c8-75974fbc49-zqjkh" Nov 01 16:09:27.327224 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:27.326583 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d6b57c1-aa75-406f-aa95-f455f8690078-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-80c4c8-5779b74587-k4gkg\" (UID: \"0d6b57c1-aa75-406f-aa95-f455f8690078\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-80c4c8-5779b74587-k4gkg" Nov 01 16:09:27.327224 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:27.326927 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/568c7dd7-469a-42a0-bb33-85167782b280-plugins-config-volume\") pod \"payload-processing-e2e-api-b-80c4c8-75974fbc49-zqjkh\" (UID: \"568c7dd7-469a-42a0-bb33-85167782b280\") " pod="openshift-ingress/payload-processing-e2e-api-b-80c4c8-75974fbc49-zqjkh" Nov 01 16:09:27.431045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:27.430473 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hksd6\" (UniqueName: \"kubernetes.io/projected/0d6b57c1-aa75-406f-aa95-f455f8690078-kube-api-access-hksd6\") pod \"payload-pre-processing-e2e-api-b-80c4c8-5779b74587-k4gkg\" (UID: \"0d6b57c1-aa75-406f-aa95-f455f8690078\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-80c4c8-5779b74587-k4gkg" Nov 01 16:09:27.431045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:27.430594 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnc7b\" (UniqueName: \"kubernetes.io/projected/568c7dd7-469a-42a0-bb33-85167782b280-kube-api-access-xnc7b\") pod \"payload-processing-e2e-api-b-80c4c8-75974fbc49-zqjkh\" (UID: \"568c7dd7-469a-42a0-bb33-85167782b280\") " pod="openshift-ingress/payload-processing-e2e-api-b-80c4c8-75974fbc49-zqjkh" Nov 01 16:09:27.431045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:27.430644 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d6b57c1-aa75-406f-aa95-f455f8690078-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-80c4c8-5779b74587-k4gkg\" (UID: \"0d6b57c1-aa75-406f-aa95-f455f8690078\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-80c4c8-5779b74587-k4gkg" Nov 01 16:09:27.431383 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:27.431091 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/568c7dd7-469a-42a0-bb33-85167782b280-plugins-config-volume\") pod \"payload-processing-e2e-api-b-80c4c8-75974fbc49-zqjkh\" (UID: \"568c7dd7-469a-42a0-bb33-85167782b280\") " pod="openshift-ingress/payload-processing-e2e-api-b-80c4c8-75974fbc49-zqjkh" Nov 01 16:09:27.456052 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:27.432041 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/568c7dd7-469a-42a0-bb33-85167782b280-plugins-config-volume\") pod \"payload-processing-e2e-api-b-80c4c8-75974fbc49-zqjkh\" (UID: \"568c7dd7-469a-42a0-bb33-85167782b280\") " pod="openshift-ingress/payload-processing-e2e-api-b-80c4c8-75974fbc49-zqjkh" Nov 01 16:09:27.456052 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:27.433331 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d6b57c1-aa75-406f-aa95-f455f8690078-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-80c4c8-5779b74587-k4gkg\" (UID: \"0d6b57c1-aa75-406f-aa95-f455f8690078\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-80c4c8-5779b74587-k4gkg" Nov 01 16:09:27.484082 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:27.478759 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnc7b\" (UniqueName: \"kubernetes.io/projected/568c7dd7-469a-42a0-bb33-85167782b280-kube-api-access-xnc7b\") pod \"payload-processing-e2e-api-b-80c4c8-75974fbc49-zqjkh\" (UID: \"568c7dd7-469a-42a0-bb33-85167782b280\") " pod="openshift-ingress/payload-processing-e2e-api-b-80c4c8-75974fbc49-zqjkh" Nov 01 16:09:27.484082 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:27.482662 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hksd6\" (UniqueName: \"kubernetes.io/projected/0d6b57c1-aa75-406f-aa95-f455f8690078-kube-api-access-hksd6\") pod \"payload-pre-processing-e2e-api-b-80c4c8-5779b74587-k4gkg\" (UID: \"0d6b57c1-aa75-406f-aa95-f455f8690078\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-80c4c8-5779b74587-k4gkg" Nov 01 16:09:27.562041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:27.556073 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-80c4c8-75974fbc49-zqjkh" Nov 01 16:09:27.646056 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:27.642843 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-80c4c8-5779b74587-k4gkg" Nov 01 16:09:28.287072 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:09:28.286993 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod568c7dd7_469a_42a0_bb33_85167782b280.slice/crio-16e268748326a91b4bf67e7a93f3e2b00de82a2190b1b141cbb59cc82745d43f WatchSource:0}: Error finding container 16e268748326a91b4bf67e7a93f3e2b00de82a2190b1b141cbb59cc82745d43f: Status 404 returned error can't find the container with id 16e268748326a91b4bf67e7a93f3e2b00de82a2190b1b141cbb59cc82745d43f Nov 01 16:09:28.288340 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:28.288305 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-80c4c8-75974fbc49-zqjkh"] Nov 01 16:09:28.463041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:28.456229 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-80c4c8-5779b74587-k4gkg"] Nov 01 16:09:29.146049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:29.143424 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-80c4c8-75974fbc49-zqjkh" event={"ID":"568c7dd7-469a-42a0-bb33-85167782b280","Type":"ContainerStarted","Data":"5935d2652e6a2b3f3bc191cb5f9396949c0c35dde1d6775f28ee392d49102511"} Nov 01 16:09:29.146049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:29.145131 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-80c4c8-75974fbc49-zqjkh" event={"ID":"568c7dd7-469a-42a0-bb33-85167782b280","Type":"ContainerStarted","Data":"16e268748326a91b4bf67e7a93f3e2b00de82a2190b1b141cbb59cc82745d43f"} Nov 01 16:09:29.146313 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:29.146129 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-80c4c8-75974fbc49-zqjkh" Nov 01 16:09:29.167488 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:29.160544 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-80c4c8-5779b74587-k4gkg" event={"ID":"0d6b57c1-aa75-406f-aa95-f455f8690078","Type":"ContainerStarted","Data":"5d429f533209d90a11bb526cdbb9aaaa0565efec86ae650726e1a4273e8e835f"} Nov 01 16:09:29.167488 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:29.160635 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-80c4c8-5779b74587-k4gkg" event={"ID":"0d6b57c1-aa75-406f-aa95-f455f8690078","Type":"ContainerStarted","Data":"dbed6a283a6c11a2163bb80693edd2fb807aa427ea5477e84e5ee68009b352c1"} Nov 01 16:09:29.180038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:29.168573 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-80c4c8-75974fbc49-zqjkh" podStartSLOduration=2.168553369 podStartE2EDuration="2.168553369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:09:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:09:29.165389136 +0000 UTC m=+1302.497126790" watchObservedRunningTime="2026-11-01 16:09:29.168553369 +0000 UTC m=+1302.500291024" Nov 01 16:09:29.180038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:29.176255 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-80c4c8-5779b74587-k4gkg" Nov 01 16:09:29.201111 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:29.199263 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-80c4c8-5779b74587-k4gkg" podStartSLOduration=2.199240075 podStartE2EDuration="2.199240075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:09:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:09:29.198632348 +0000 UTC m=+1302.530369999" watchObservedRunningTime="2026-11-01 16:09:29.199240075 +0000 UTC m=+1302.530977728" Nov 01 16:09:30.083297 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:30.083264 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6"] Nov 01 16:09:30.083774 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:30.083699 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" podUID="241f70eb-0584-4e76-9616-152a84654bde" containerName="istio-proxy" containerID="cri-o://b2911525b548edd8ce427ad4371687a1e03dab1155706e6b512f000fdd921854" gracePeriod=30 Nov 01 16:09:35.686453 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:35.686425 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" Nov 01 16:09:35.848559 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:35.844235 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/241f70eb-0584-4e76-9616-152a84654bde-istio-token\" (UniqueName: \"kubernetes.io/projected/241f70eb-0584-4e76-9616-152a84654bde-istio-token\") pod \"241f70eb-0584-4e76-9616-152a84654bde\" (UID: \"241f70eb-0584-4e76-9616-152a84654bde\") " Nov 01 16:09:35.848559 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:35.844312 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/241f70eb-0584-4e76-9616-152a84654bde-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/241f70eb-0584-4e76-9616-152a84654bde-workload-socket\") pod \"241f70eb-0584-4e76-9616-152a84654bde\" (UID: \"241f70eb-0584-4e76-9616-152a84654bde\") " Nov 01 16:09:35.848559 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:35.844442 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/241f70eb-0584-4e76-9616-152a84654bde-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/241f70eb-0584-4e76-9616-152a84654bde-istio-podinfo\") pod \"241f70eb-0584-4e76-9616-152a84654bde\" (UID: \"241f70eb-0584-4e76-9616-152a84654bde\") " Nov 01 16:09:35.848559 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:35.844494 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/241f70eb-0584-4e76-9616-152a84654bde-kube-api-access-v8vln\" (UniqueName: \"kubernetes.io/projected/241f70eb-0584-4e76-9616-152a84654bde-kube-api-access-v8vln\") pod \"241f70eb-0584-4e76-9616-152a84654bde\" (UID: \"241f70eb-0584-4e76-9616-152a84654bde\") " Nov 01 16:09:35.848559 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:35.844546 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/241f70eb-0584-4e76-9616-152a84654bde-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/241f70eb-0584-4e76-9616-152a84654bde-istio-data\") pod \"241f70eb-0584-4e76-9616-152a84654bde\" (UID: \"241f70eb-0584-4e76-9616-152a84654bde\") " Nov 01 16:09:35.848559 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:35.844759 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/241f70eb-0584-4e76-9616-152a84654bde-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/241f70eb-0584-4e76-9616-152a84654bde-workload-certs\") pod \"241f70eb-0584-4e76-9616-152a84654bde\" (UID: \"241f70eb-0584-4e76-9616-152a84654bde\") " Nov 01 16:09:35.848559 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:35.844840 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/241f70eb-0584-4e76-9616-152a84654bde-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/241f70eb-0584-4e76-9616-152a84654bde-istio-envoy\") pod \"241f70eb-0584-4e76-9616-152a84654bde\" (UID: \"241f70eb-0584-4e76-9616-152a84654bde\") " Nov 01 16:09:35.848559 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:35.844930 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/241f70eb-0584-4e76-9616-152a84654bde-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/241f70eb-0584-4e76-9616-152a84654bde-credential-socket\") pod \"241f70eb-0584-4e76-9616-152a84654bde\" (UID: \"241f70eb-0584-4e76-9616-152a84654bde\") " Nov 01 16:09:35.848559 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:35.845129 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/241f70eb-0584-4e76-9616-152a84654bde-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/241f70eb-0584-4e76-9616-152a84654bde-istiod-ca-cert\") pod \"241f70eb-0584-4e76-9616-152a84654bde\" (UID: \"241f70eb-0584-4e76-9616-152a84654bde\") " Nov 01 16:09:35.855737 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:35.853097 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/241f70eb-0584-4e76-9616-152a84654bde-kube-api-access-v8vln" pod "241f70eb-0584-4e76-9616-152a84654bde" (UID: "241f70eb-0584-4e76-9616-152a84654bde"). InnerVolumeSpecName "kube-api-access-v8vln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:09:35.858359 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:35.858319 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/241f70eb-0584-4e76-9616-152a84654bde-workload-socket" pod "241f70eb-0584-4e76-9616-152a84654bde" (UID: "241f70eb-0584-4e76-9616-152a84654bde"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:09:35.858503 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:35.858490 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/241f70eb-0584-4e76-9616-152a84654bde-workload-certs" pod "241f70eb-0584-4e76-9616-152a84654bde" (UID: "241f70eb-0584-4e76-9616-152a84654bde"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:09:35.858630 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:35.858609 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/241f70eb-0584-4e76-9616-152a84654bde-istio-data" pod "241f70eb-0584-4e76-9616-152a84654bde" (UID: "241f70eb-0584-4e76-9616-152a84654bde"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:09:35.883050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:35.875268 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/241f70eb-0584-4e76-9616-152a84654bde-credential-socket" pod "241f70eb-0584-4e76-9616-152a84654bde" (UID: "241f70eb-0584-4e76-9616-152a84654bde"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:09:35.887326 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:35.886283 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/241f70eb-0584-4e76-9616-152a84654bde-istiod-ca-cert" pod "241f70eb-0584-4e76-9616-152a84654bde" (UID: "241f70eb-0584-4e76-9616-152a84654bde"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:09:35.892639 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:35.889247 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/241f70eb-0584-4e76-9616-152a84654bde-istio-token" pod "241f70eb-0584-4e76-9616-152a84654bde" (UID: "241f70eb-0584-4e76-9616-152a84654bde"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:09:35.892639 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:35.890440 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/241f70eb-0584-4e76-9616-152a84654bde-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:35.892639 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:35.890465 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/241f70eb-0584-4e76-9616-152a84654bde-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:35.892639 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:35.890481 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/241f70eb-0584-4e76-9616-152a84654bde-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:35.892639 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:35.890496 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8vln\" (UniqueName: \"kubernetes.io/projected/241f70eb-0584-4e76-9616-152a84654bde-kube-api-access-v8vln\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:35.892639 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:35.890513 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/241f70eb-0584-4e76-9616-152a84654bde-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:35.892639 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:35.890528 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/241f70eb-0584-4e76-9616-152a84654bde-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:35.892639 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:35.890542 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/241f70eb-0584-4e76-9616-152a84654bde-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:35.977245 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:35.977138 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/241f70eb-0584-4e76-9616-152a84654bde-istio-podinfo" pod "241f70eb-0584-4e76-9616-152a84654bde" (UID: "241f70eb-0584-4e76-9616-152a84654bde"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:09:35.985760 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:35.985721 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/241f70eb-0584-4e76-9616-152a84654bde-istio-envoy" pod "241f70eb-0584-4e76-9616-152a84654bde" (UID: "241f70eb-0584-4e76-9616-152a84654bde"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:09:36.005047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:36.002569 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/241f70eb-0584-4e76-9616-152a84654bde-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:36.005047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:36.002629 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/241f70eb-0584-4e76-9616-152a84654bde-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:36.431604 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:36.431565 4363 generic.go:356] "Generic (PLEG): container finished" podID="241f70eb-0584-4e76-9616-152a84654bde" containerID="b2911525b548edd8ce427ad4371687a1e03dab1155706e6b512f000fdd921854" exitCode=0 Nov 01 16:09:36.431762 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:36.431617 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" event={"ID":"241f70eb-0584-4e76-9616-152a84654bde","Type":"ContainerDied","Data":"b2911525b548edd8ce427ad4371687a1e03dab1155706e6b512f000fdd921854"} Nov 01 16:09:36.431762 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:36.431631 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" Nov 01 16:09:36.431762 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:36.431671 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6" event={"ID":"241f70eb-0584-4e76-9616-152a84654bde","Type":"ContainerDied","Data":"cbc11ec904a709780a0d421efd11fd91c0dd08c36fc918a3ccdddb6bb905b4d1"} Nov 01 16:09:36.431762 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:36.431699 4363 scope.go:122] "RemoveContainer" containerID="b2911525b548edd8ce427ad4371687a1e03dab1155706e6b512f000fdd921854" Nov 01 16:09:36.443260 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:36.443243 4363 scope.go:122] "RemoveContainer" containerID="b2911525b548edd8ce427ad4371687a1e03dab1155706e6b512f000fdd921854" Nov 01 16:09:36.443578 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:09:36.443556 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2911525b548edd8ce427ad4371687a1e03dab1155706e6b512f000fdd921854\": container with ID starting with b2911525b548edd8ce427ad4371687a1e03dab1155706e6b512f000fdd921854 not found: ID does not exist" containerID="b2911525b548edd8ce427ad4371687a1e03dab1155706e6b512f000fdd921854" Nov 01 16:09:36.443639 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:36.443587 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2911525b548edd8ce427ad4371687a1e03dab1155706e6b512f000fdd921854"} err="failed to get container status \"b2911525b548edd8ce427ad4371687a1e03dab1155706e6b512f000fdd921854\": rpc error: code = NotFound desc = could not find container \"b2911525b548edd8ce427ad4371687a1e03dab1155706e6b512f000fdd921854\": container with ID starting with b2911525b548edd8ce427ad4371687a1e03dab1155706e6b512f000fdd921854 not found: ID does not exist" Nov 01 16:09:36.459462 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:36.459437 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6"] Nov 01 16:09:36.463776 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:36.463757 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-gsxq6"] Nov 01 16:09:36.804081 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:36.803995 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="241f70eb-0584-4e76-9616-152a84654bde" path="/var/lib/kubelet/pods/241f70eb-0584-4e76-9616-152a84654bde/volumes" Nov 01 16:09:39.695420 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:39.695361 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h"] Nov 01 16:09:39.695979 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:39.695959 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-lqtdt"] Nov 01 16:09:39.698437 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:39.698412 4363 cpu_manager.go:414] "containerMap: removing container" podUID="241f70eb-0584-4e76-9616-152a84654bde" containerName="istio-proxy" Nov 01 16:09:39.711032 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:39.710989 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="241f70eb-0584-4e76-9616-152a84654bde" containerName="istio-proxy" Nov 01 16:09:39.712097 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:39.711896 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="241f70eb-0584-4e76-9616-152a84654bde" containerName="istio-proxy" Nov 01 16:09:39.720438 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:39.698867 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" podUID="922af45f-b42b-47de-9724-6f2daa958330" containerName="istio-proxy" containerID="cri-o://fcdfb6c868bbc4f02d17cd5267156ea0c447011e705da88207029c6f3bb277ef" gracePeriod=30 Nov 01 16:09:39.720916 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:39.720891 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-lqtdt" Nov 01 16:09:39.795049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:39.791281 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-76063d10-dockercfg-bdgjz\"" Nov 01 16:09:39.795049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:39.791599 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-76063d10\"" Nov 01 16:09:39.839290 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:39.839253 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-lqtdt"] Nov 01 16:09:39.922951 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:39.904845 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-vrmjz"] Nov 01 16:09:39.964238 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:39.893654 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:09:40.088040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:40.082060 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-vrmjz" Nov 01 16:09:40.185159 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:40.185111 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:09:40.214040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:40.211633 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37d0ec61-3743-48db-bed1-7ba33650158a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-lqtdt\" (UID: \"37d0ec61-3743-48db-bed1-7ba33650158a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-lqtdt" Nov 01 16:09:40.214040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:40.211702 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzlvr\" (UniqueName: \"kubernetes.io/projected/37d0ec61-3743-48db-bed1-7ba33650158a-kube-api-access-tzlvr\") pod \"payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-lqtdt\" (UID: \"37d0ec61-3743-48db-bed1-7ba33650158a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-lqtdt" Nov 01 16:09:40.232314 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:40.224689 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-80c4c8-5779b74587-k4gkg" Nov 01 16:09:40.241045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:40.237129 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-vrmjz"] Nov 01 16:09:40.334041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:40.314113 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37d0ec61-3743-48db-bed1-7ba33650158a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-lqtdt\" (UID: \"37d0ec61-3743-48db-bed1-7ba33650158a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-lqtdt" Nov 01 16:09:40.335119 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:40.335086 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzlvr\" (UniqueName: \"kubernetes.io/projected/37d0ec61-3743-48db-bed1-7ba33650158a-kube-api-access-tzlvr\") pod \"payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-lqtdt\" (UID: \"37d0ec61-3743-48db-bed1-7ba33650158a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-lqtdt" Nov 01 16:09:40.345177 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:40.345132 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37d0ec61-3743-48db-bed1-7ba33650158a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-lqtdt\" (UID: \"37d0ec61-3743-48db-bed1-7ba33650158a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-lqtdt" Nov 01 16:09:40.444534 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:40.438075 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a30818-8812-4f4d-a5c3-764d879deb52-plugins-config-volume\") pod \"payload-processing-e2e-mt-76063d10-f66f6f4bf-vrmjz\" (UID: \"11a30818-8812-4f4d-a5c3-764d879deb52\") " pod="openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-vrmjz" Nov 01 16:09:40.444534 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:40.438606 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc5kk\" (UniqueName: \"kubernetes.io/projected/11a30818-8812-4f4d-a5c3-764d879deb52-kube-api-access-wc5kk\") pod \"payload-processing-e2e-mt-76063d10-f66f6f4bf-vrmjz\" (UID: \"11a30818-8812-4f4d-a5c3-764d879deb52\") " pod="openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-vrmjz" Nov 01 16:09:40.525812 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:40.524551 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzlvr\" (UniqueName: \"kubernetes.io/projected/37d0ec61-3743-48db-bed1-7ba33650158a-kube-api-access-tzlvr\") pod \"payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-lqtdt\" (UID: \"37d0ec61-3743-48db-bed1-7ba33650158a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-lqtdt" Nov 01 16:09:40.552052 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:40.547262 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc5kk\" (UniqueName: \"kubernetes.io/projected/11a30818-8812-4f4d-a5c3-764d879deb52-kube-api-access-wc5kk\") pod \"payload-processing-e2e-mt-76063d10-f66f6f4bf-vrmjz\" (UID: \"11a30818-8812-4f4d-a5c3-764d879deb52\") " pod="openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-vrmjz" Nov 01 16:09:40.552052 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:40.547450 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a30818-8812-4f4d-a5c3-764d879deb52-plugins-config-volume\") pod \"payload-processing-e2e-mt-76063d10-f66f6f4bf-vrmjz\" (UID: \"11a30818-8812-4f4d-a5c3-764d879deb52\") " pod="openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-vrmjz" Nov 01 16:09:40.552052 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:40.549427 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a30818-8812-4f4d-a5c3-764d879deb52-plugins-config-volume\") pod \"payload-processing-e2e-mt-76063d10-f66f6f4bf-vrmjz\" (UID: \"11a30818-8812-4f4d-a5c3-764d879deb52\") " pod="openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-vrmjz" Nov 01 16:09:40.752116 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:40.720939 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-lqtdt" Nov 01 16:09:40.752116 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:40.723618 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc5kk\" (UniqueName: \"kubernetes.io/projected/11a30818-8812-4f4d-a5c3-764d879deb52-kube-api-access-wc5kk\") pod \"payload-processing-e2e-mt-76063d10-f66f6f4bf-vrmjz\" (UID: \"11a30818-8812-4f4d-a5c3-764d879deb52\") " pod="openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-vrmjz" Nov 01 16:09:40.869045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:40.859922 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-vrmjz" Nov 01 16:09:41.112340 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:41.112296 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88"] Nov 01 16:09:41.125098 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:41.125063 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88"] Nov 01 16:09:41.125266 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:41.125200 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" Nov 01 16:09:41.131164 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:41.131137 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f5dd93-openshift-default-dockercfg-h4q9h\"" Nov 01 16:09:41.191515 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:41.191484 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-80c4c8-75974fbc49-zqjkh" Nov 01 16:09:41.277946 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:41.277908 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9cc82b-caf7-4108-b80b-d10135f512b1-credential-socket\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88\" (UID: \"ef9cc82b-caf7-4108-b80b-d10135f512b1\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" Nov 01 16:09:41.277946 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:41.277952 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef9cc82b-caf7-4108-b80b-d10135f512b1-istio-podinfo\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88\" (UID: \"ef9cc82b-caf7-4108-b80b-d10135f512b1\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" Nov 01 16:09:41.278255 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:41.278127 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef9cc82b-caf7-4108-b80b-d10135f512b1-istiod-ca-cert\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88\" (UID: \"ef9cc82b-caf7-4108-b80b-d10135f512b1\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" Nov 01 16:09:41.278255 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:41.278183 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef9cc82b-caf7-4108-b80b-d10135f512b1-workload-certs\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88\" (UID: \"ef9cc82b-caf7-4108-b80b-d10135f512b1\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" Nov 01 16:09:41.278366 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:41.278301 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef9cc82b-caf7-4108-b80b-d10135f512b1-istio-data\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88\" (UID: \"ef9cc82b-caf7-4108-b80b-d10135f512b1\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" Nov 01 16:09:41.278366 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:41.278324 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef9cc82b-caf7-4108-b80b-d10135f512b1-istio-envoy\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88\" (UID: \"ef9cc82b-caf7-4108-b80b-d10135f512b1\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" Nov 01 16:09:41.278472 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:41.278391 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9cc82b-caf7-4108-b80b-d10135f512b1-workload-socket\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88\" (UID: \"ef9cc82b-caf7-4108-b80b-d10135f512b1\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" Nov 01 16:09:41.278472 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:41.278432 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef9cc82b-caf7-4108-b80b-d10135f512b1-istio-token\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88\" (UID: \"ef9cc82b-caf7-4108-b80b-d10135f512b1\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" Nov 01 16:09:41.278586 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:41.278469 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqtdq\" (UniqueName: \"kubernetes.io/projected/ef9cc82b-caf7-4108-b80b-d10135f512b1-kube-api-access-rqtdq\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88\" (UID: \"ef9cc82b-caf7-4108-b80b-d10135f512b1\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" Nov 01 16:09:41.380640 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:41.380554 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef9cc82b-caf7-4108-b80b-d10135f512b1-istiod-ca-cert\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88\" (UID: \"ef9cc82b-caf7-4108-b80b-d10135f512b1\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" Nov 01 16:09:41.380640 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:41.380610 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef9cc82b-caf7-4108-b80b-d10135f512b1-workload-certs\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88\" (UID: \"ef9cc82b-caf7-4108-b80b-d10135f512b1\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" Nov 01 16:09:41.380922 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:41.380687 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef9cc82b-caf7-4108-b80b-d10135f512b1-istio-data\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88\" (UID: \"ef9cc82b-caf7-4108-b80b-d10135f512b1\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" Nov 01 16:09:41.380922 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:41.380711 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef9cc82b-caf7-4108-b80b-d10135f512b1-istio-envoy\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88\" (UID: \"ef9cc82b-caf7-4108-b80b-d10135f512b1\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" Nov 01 16:09:41.380922 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:41.380775 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9cc82b-caf7-4108-b80b-d10135f512b1-workload-socket\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88\" (UID: \"ef9cc82b-caf7-4108-b80b-d10135f512b1\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" Nov 01 16:09:41.380922 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:41.380821 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef9cc82b-caf7-4108-b80b-d10135f512b1-istio-token\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88\" (UID: \"ef9cc82b-caf7-4108-b80b-d10135f512b1\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" Nov 01 16:09:41.380922 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:41.380853 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqtdq\" (UniqueName: \"kubernetes.io/projected/ef9cc82b-caf7-4108-b80b-d10135f512b1-kube-api-access-rqtdq\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88\" (UID: \"ef9cc82b-caf7-4108-b80b-d10135f512b1\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" Nov 01 16:09:41.381289 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:41.381086 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef9cc82b-caf7-4108-b80b-d10135f512b1-workload-certs\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88\" (UID: \"ef9cc82b-caf7-4108-b80b-d10135f512b1\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" Nov 01 16:09:41.381289 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:41.381143 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef9cc82b-caf7-4108-b80b-d10135f512b1-istio-data\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88\" (UID: \"ef9cc82b-caf7-4108-b80b-d10135f512b1\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" Nov 01 16:09:41.381399 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:41.381307 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9cc82b-caf7-4108-b80b-d10135f512b1-workload-socket\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88\" (UID: \"ef9cc82b-caf7-4108-b80b-d10135f512b1\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" Nov 01 16:09:41.381456 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:41.381410 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef9cc82b-caf7-4108-b80b-d10135f512b1-istiod-ca-cert\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88\" (UID: \"ef9cc82b-caf7-4108-b80b-d10135f512b1\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" Nov 01 16:09:41.381556 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:41.381515 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9cc82b-caf7-4108-b80b-d10135f512b1-credential-socket\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88\" (UID: \"ef9cc82b-caf7-4108-b80b-d10135f512b1\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" Nov 01 16:09:41.381556 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:41.381551 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef9cc82b-caf7-4108-b80b-d10135f512b1-istio-podinfo\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88\" (UID: \"ef9cc82b-caf7-4108-b80b-d10135f512b1\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" Nov 01 16:09:41.381904 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:41.381883 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9cc82b-caf7-4108-b80b-d10135f512b1-credential-socket\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88\" (UID: \"ef9cc82b-caf7-4108-b80b-d10135f512b1\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" Nov 01 16:09:41.384049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:41.383989 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef9cc82b-caf7-4108-b80b-d10135f512b1-istio-envoy\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88\" (UID: \"ef9cc82b-caf7-4108-b80b-d10135f512b1\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" Nov 01 16:09:41.384388 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:41.384370 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef9cc82b-caf7-4108-b80b-d10135f512b1-istio-podinfo\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88\" (UID: \"ef9cc82b-caf7-4108-b80b-d10135f512b1\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" Nov 01 16:09:41.388649 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:41.388629 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef9cc82b-caf7-4108-b80b-d10135f512b1-istio-token\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88\" (UID: \"ef9cc82b-caf7-4108-b80b-d10135f512b1\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" Nov 01 16:09:41.389239 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:41.389220 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqtdq\" (UniqueName: \"kubernetes.io/projected/ef9cc82b-caf7-4108-b80b-d10135f512b1-kube-api-access-rqtdq\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88\" (UID: \"ef9cc82b-caf7-4108-b80b-d10135f512b1\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" Nov 01 16:09:41.408355 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:41.408329 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-lqtdt"] Nov 01 16:09:41.414329 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:09:41.414306 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11a30818_8812_4f4d_a5c3_764d879deb52.slice/crio-f8503eafa00333fd98ea805636d838b0807ae00386b075a263ab1d954823b9f9 WatchSource:0}: Error finding container f8503eafa00333fd98ea805636d838b0807ae00386b075a263ab1d954823b9f9: Status 404 returned error can't find the container with id f8503eafa00333fd98ea805636d838b0807ae00386b075a263ab1d954823b9f9 Nov 01 16:09:41.415653 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:41.415631 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-vrmjz"] Nov 01 16:09:41.452204 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:41.452170 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" Nov 01 16:09:41.906583 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:41.906555 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88"] Nov 01 16:09:41.907005 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:09:41.906925 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef9cc82b_caf7_4108_b80b_d10135f512b1.slice/crio-5b0be85fbec09466d052d8b9174e5500c42e20ce1d9e88005970130848683131 WatchSource:0}: Error finding container 5b0be85fbec09466d052d8b9174e5500c42e20ce1d9e88005970130848683131: Status 404 returned error can't find the container with id 5b0be85fbec09466d052d8b9174e5500c42e20ce1d9e88005970130848683131 Nov 01 16:09:41.910494 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:41.910461 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:09:41.910628 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:41.910525 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:09:41.910628 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:41.910566 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:09:42.100084 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:42.100045 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-vrmjz" event={"ID":"11a30818-8812-4f4d-a5c3-764d879deb52","Type":"ContainerStarted","Data":"fdbf01d14088d583fcd34c8caa42a8299e9f25c8ac450e1ee22d0606f7330c09"} Nov 01 16:09:42.100277 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:42.100121 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-vrmjz" Nov 01 16:09:42.100277 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:42.100145 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-vrmjz" event={"ID":"11a30818-8812-4f4d-a5c3-764d879deb52","Type":"ContainerStarted","Data":"f8503eafa00333fd98ea805636d838b0807ae00386b075a263ab1d954823b9f9"} Nov 01 16:09:42.102896 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:42.102870 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" event={"ID":"ef9cc82b-caf7-4108-b80b-d10135f512b1","Type":"ContainerStarted","Data":"a2fdb4ea49a46cd17108656febd6573d9ebf7f3d7d40960311021f074ae2fca6"} Nov 01 16:09:42.103152 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:42.103126 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" event={"ID":"ef9cc82b-caf7-4108-b80b-d10135f512b1","Type":"ContainerStarted","Data":"5b0be85fbec09466d052d8b9174e5500c42e20ce1d9e88005970130848683131"} Nov 01 16:09:42.106090 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:42.106064 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-lqtdt" event={"ID":"37d0ec61-3743-48db-bed1-7ba33650158a","Type":"ContainerStarted","Data":"49bbd26bead5c5a52af3a09249aaea79894eb6e3b02aae6521f0dfd9cc440fa0"} Nov 01 16:09:42.106210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:42.106112 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-lqtdt" event={"ID":"37d0ec61-3743-48db-bed1-7ba33650158a","Type":"ContainerStarted","Data":"e792f86f3278e6ada6af18578ed125e169599c889cd4d71f9219493193bceba2"} Nov 01 16:09:42.106210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:42.106155 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-lqtdt" Nov 01 16:09:42.119863 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:42.119820 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-vrmjz" podStartSLOduration=3.119806895 podStartE2EDuration="3.119806895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:09:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:09:42.114856765 +0000 UTC m=+1315.446594441" watchObservedRunningTime="2026-11-01 16:09:42.119806895 +0000 UTC m=+1315.451544542" Nov 01 16:09:42.136690 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:42.136628 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" podStartSLOduration=2.136609944 podStartE2EDuration="2.136609944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:09:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:09:42.1343636 +0000 UTC m=+1315.466101252" watchObservedRunningTime="2026-11-01 16:09:42.136609944 +0000 UTC m=+1315.468347595" Nov 01 16:09:42.153115 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:42.153065 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-lqtdt" podStartSLOduration=3.153052198 podStartE2EDuration="3.153052198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:09:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:09:42.148776198 +0000 UTC m=+1315.480513843" watchObservedRunningTime="2026-11-01 16:09:42.153052198 +0000 UTC m=+1315.484789881" Nov 01 16:09:42.462629 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:42.462543 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" Nov 01 16:09:43.510491 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:43.510452 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" Nov 01 16:09:44.172446 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:44.172416 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" Nov 01 16:09:44.173615 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:44.173594 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" Nov 01 16:09:44.399978 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:44.399934 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn"] Nov 01 16:09:44.400501 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:44.400453 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" podUID="c10680c0-ac33-4914-931e-15c3728484bc" containerName="istio-proxy" containerID="cri-o://e323a502222833dc923d26c2a6ec20177c3ac61e1ab12e27f21f15b860a277c0" gracePeriod=30 Nov 01 16:09:44.622087 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:44.610516 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt"] Nov 01 16:09:44.622087 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:44.614162 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" podUID="41d1f840-a59c-4f46-84b0-d04c3cd1d500" containerName="istio-proxy" containerID="cri-o://f11b60de5c029ba956d3e5cef1d1d8fcc91a7654adb0f2f27eaf452d502bf650" gracePeriod=30 Nov 01 16:09:44.947523 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:44.947446 4363 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:09:44.947523 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:44.947501 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" podUID="41d1f840-a59c-4f46-84b0-d04c3cd1d500" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:09:45.183907 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.183874 4363 generic.go:356] "Generic (PLEG): container finished" podID="922af45f-b42b-47de-9724-6f2daa958330" containerID="fcdfb6c868bbc4f02d17cd5267156ea0c447011e705da88207029c6f3bb277ef" exitCode=0 Nov 01 16:09:45.184965 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.184882 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" event={"ID":"922af45f-b42b-47de-9724-6f2daa958330","Type":"ContainerDied","Data":"fcdfb6c868bbc4f02d17cd5267156ea0c447011e705da88207029c6f3bb277ef"} Nov 01 16:09:45.185283 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.185257 4363 scope.go:122] "RemoveContainer" containerID="0eaeb83660100b9966a6b659e9b68f1b4598fa0e0ed0a6b61c790ed6224edb4b" Nov 01 16:09:45.368618 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.368592 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" Nov 01 16:09:45.447820 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.447780 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/922af45f-b42b-47de-9724-6f2daa958330-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922af45f-b42b-47de-9724-6f2daa958330-istiod-ca-cert\") pod \"922af45f-b42b-47de-9724-6f2daa958330\" (UID: \"922af45f-b42b-47de-9724-6f2daa958330\") " Nov 01 16:09:45.448096 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.447858 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922af45f-b42b-47de-9724-6f2daa958330-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922af45f-b42b-47de-9724-6f2daa958330-workload-socket\") pod \"922af45f-b42b-47de-9724-6f2daa958330\" (UID: \"922af45f-b42b-47de-9724-6f2daa958330\") " Nov 01 16:09:45.448096 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.447927 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922af45f-b42b-47de-9724-6f2daa958330-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922af45f-b42b-47de-9724-6f2daa958330-istio-envoy\") pod \"922af45f-b42b-47de-9724-6f2daa958330\" (UID: \"922af45f-b42b-47de-9724-6f2daa958330\") " Nov 01 16:09:45.448096 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.447957 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/922af45f-b42b-47de-9724-6f2daa958330-istio-token\" (UniqueName: \"kubernetes.io/projected/922af45f-b42b-47de-9724-6f2daa958330-istio-token\") pod \"922af45f-b42b-47de-9724-6f2daa958330\" (UID: \"922af45f-b42b-47de-9724-6f2daa958330\") " Nov 01 16:09:45.448297 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.448103 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/922af45f-b42b-47de-9724-6f2daa958330-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922af45f-b42b-47de-9724-6f2daa958330-istio-podinfo\") pod \"922af45f-b42b-47de-9724-6f2daa958330\" (UID: \"922af45f-b42b-47de-9724-6f2daa958330\") " Nov 01 16:09:45.448297 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.448153 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922af45f-b42b-47de-9724-6f2daa958330-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922af45f-b42b-47de-9724-6f2daa958330-credential-socket\") pod \"922af45f-b42b-47de-9724-6f2daa958330\" (UID: \"922af45f-b42b-47de-9724-6f2daa958330\") " Nov 01 16:09:45.448297 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.448270 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/922af45f-b42b-47de-9724-6f2daa958330-kube-api-access-d9xfm\" (UniqueName: \"kubernetes.io/projected/922af45f-b42b-47de-9724-6f2daa958330-kube-api-access-d9xfm\") pod \"922af45f-b42b-47de-9724-6f2daa958330\" (UID: \"922af45f-b42b-47de-9724-6f2daa958330\") " Nov 01 16:09:45.451080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.451042 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/922af45f-b42b-47de-9724-6f2daa958330-istiod-ca-cert" pod "922af45f-b42b-47de-9724-6f2daa958330" (UID: "922af45f-b42b-47de-9724-6f2daa958330"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:09:45.451187 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.451101 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922af45f-b42b-47de-9724-6f2daa958330-credential-socket" pod "922af45f-b42b-47de-9724-6f2daa958330" (UID: "922af45f-b42b-47de-9724-6f2daa958330"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:09:45.451513 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.451484 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922af45f-b42b-47de-9724-6f2daa958330-workload-socket" pod "922af45f-b42b-47de-9724-6f2daa958330" (UID: "922af45f-b42b-47de-9724-6f2daa958330"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:09:45.451733 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.451710 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922af45f-b42b-47de-9724-6f2daa958330-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922af45f-b42b-47de-9724-6f2daa958330-istio-data\") pod \"922af45f-b42b-47de-9724-6f2daa958330\" (UID: \"922af45f-b42b-47de-9724-6f2daa958330\") " Nov 01 16:09:45.451977 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.451956 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922af45f-b42b-47de-9724-6f2daa958330-istio-data" pod "922af45f-b42b-47de-9724-6f2daa958330" (UID: "922af45f-b42b-47de-9724-6f2daa958330"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:09:45.452084 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.452041 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922af45f-b42b-47de-9724-6f2daa958330-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922af45f-b42b-47de-9724-6f2daa958330-workload-certs\") pod \"922af45f-b42b-47de-9724-6f2daa958330\" (UID: \"922af45f-b42b-47de-9724-6f2daa958330\") " Nov 01 16:09:45.453130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.453105 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922af45f-b42b-47de-9724-6f2daa958330-workload-certs" pod "922af45f-b42b-47de-9724-6f2daa958330" (UID: "922af45f-b42b-47de-9724-6f2daa958330"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:09:45.454502 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.454455 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/922af45f-b42b-47de-9724-6f2daa958330-istio-token" pod "922af45f-b42b-47de-9724-6f2daa958330" (UID: "922af45f-b42b-47de-9724-6f2daa958330"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:09:45.454805 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.454784 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922af45f-b42b-47de-9724-6f2daa958330-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:45.455009 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.454809 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922af45f-b42b-47de-9724-6f2daa958330-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:45.455009 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.454823 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922af45f-b42b-47de-9724-6f2daa958330-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:45.455009 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.454837 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922af45f-b42b-47de-9724-6f2daa958330-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:45.455009 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.454850 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922af45f-b42b-47de-9724-6f2daa958330-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:45.455009 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.454864 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922af45f-b42b-47de-9724-6f2daa958330-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:45.474581 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.474535 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/922af45f-b42b-47de-9724-6f2daa958330-istio-podinfo" pod "922af45f-b42b-47de-9724-6f2daa958330" (UID: "922af45f-b42b-47de-9724-6f2daa958330"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:09:45.474847 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.474803 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922af45f-b42b-47de-9724-6f2daa958330-istio-envoy" pod "922af45f-b42b-47de-9724-6f2daa958330" (UID: "922af45f-b42b-47de-9724-6f2daa958330"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:09:45.482241 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.482199 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/922af45f-b42b-47de-9724-6f2daa958330-kube-api-access-d9xfm" pod "922af45f-b42b-47de-9724-6f2daa958330" (UID: "922af45f-b42b-47de-9724-6f2daa958330"). InnerVolumeSpecName "kube-api-access-d9xfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:09:45.501829 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.501607 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p"] Nov 01 16:09:45.508579 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.508541 4363 cpu_manager.go:414] "containerMap: removing container" podUID="922af45f-b42b-47de-9724-6f2daa958330" containerName="istio-proxy" Nov 01 16:09:45.508579 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.508572 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="922af45f-b42b-47de-9724-6f2daa958330" containerName="istio-proxy" Nov 01 16:09:45.508872 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.508856 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="922af45f-b42b-47de-9724-6f2daa958330" containerName="istio-proxy" Nov 01 16:09:45.508961 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.508884 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="922af45f-b42b-47de-9724-6f2daa958330" containerName="istio-proxy" Nov 01 16:09:45.514081 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.514011 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p"] Nov 01 16:09:45.514193 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.514168 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" Nov 01 16:09:45.524574 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.524545 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-022f62-openshift-default-dockercfg-7n4cf\"" Nov 01 16:09:45.557270 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.557101 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb765068-ab2a-42f7-9f46-66cfa9745814-credential-socket\") pod \"e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p\" (UID: \"eb765068-ab2a-42f7-9f46-66cfa9745814\") " pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" Nov 01 16:09:45.557270 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.557239 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb765068-ab2a-42f7-9f46-66cfa9745814-workload-certs\") pod \"e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p\" (UID: \"eb765068-ab2a-42f7-9f46-66cfa9745814\") " pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" Nov 01 16:09:45.557651 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.557415 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb765068-ab2a-42f7-9f46-66cfa9745814-istio-data\") pod \"e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p\" (UID: \"eb765068-ab2a-42f7-9f46-66cfa9745814\") " pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" Nov 01 16:09:45.557651 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.557476 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb765068-ab2a-42f7-9f46-66cfa9745814-workload-socket\") pod \"e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p\" (UID: \"eb765068-ab2a-42f7-9f46-66cfa9745814\") " pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" Nov 01 16:09:45.557651 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.557512 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb765068-ab2a-42f7-9f46-66cfa9745814-istio-envoy\") pod \"e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p\" (UID: \"eb765068-ab2a-42f7-9f46-66cfa9745814\") " pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" Nov 01 16:09:45.557827 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.557670 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb765068-ab2a-42f7-9f46-66cfa9745814-istio-token\") pod \"e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p\" (UID: \"eb765068-ab2a-42f7-9f46-66cfa9745814\") " pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" Nov 01 16:09:45.557891 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.557831 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb765068-ab2a-42f7-9f46-66cfa9745814-istiod-ca-cert\") pod \"e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p\" (UID: \"eb765068-ab2a-42f7-9f46-66cfa9745814\") " pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" Nov 01 16:09:45.557891 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.557877 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb765068-ab2a-42f7-9f46-66cfa9745814-istio-podinfo\") pod \"e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p\" (UID: \"eb765068-ab2a-42f7-9f46-66cfa9745814\") " pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" Nov 01 16:09:45.558006 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.557926 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-458gr\" (UniqueName: \"kubernetes.io/projected/eb765068-ab2a-42f7-9f46-66cfa9745814-kube-api-access-458gr\") pod \"e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p\" (UID: \"eb765068-ab2a-42f7-9f46-66cfa9745814\") " pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" Nov 01 16:09:45.558130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.558090 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922af45f-b42b-47de-9724-6f2daa958330-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:45.558130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.558110 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922af45f-b42b-47de-9724-6f2daa958330-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:45.558130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.558127 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9xfm\" (UniqueName: \"kubernetes.io/projected/922af45f-b42b-47de-9724-6f2daa958330-kube-api-access-d9xfm\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:45.723097 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.720638 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb765068-ab2a-42f7-9f46-66cfa9745814-istio-podinfo\") pod \"e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p\" (UID: \"eb765068-ab2a-42f7-9f46-66cfa9745814\") " pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" Nov 01 16:09:45.723097 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.720723 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-458gr\" (UniqueName: \"kubernetes.io/projected/eb765068-ab2a-42f7-9f46-66cfa9745814-kube-api-access-458gr\") pod \"e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p\" (UID: \"eb765068-ab2a-42f7-9f46-66cfa9745814\") " pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" Nov 01 16:09:45.723097 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.720782 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb765068-ab2a-42f7-9f46-66cfa9745814-credential-socket\") pod \"e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p\" (UID: \"eb765068-ab2a-42f7-9f46-66cfa9745814\") " pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" Nov 01 16:09:45.723097 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.720845 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb765068-ab2a-42f7-9f46-66cfa9745814-workload-certs\") pod \"e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p\" (UID: \"eb765068-ab2a-42f7-9f46-66cfa9745814\") " pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" Nov 01 16:09:45.723097 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.721063 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb765068-ab2a-42f7-9f46-66cfa9745814-istio-data\") pod \"e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p\" (UID: \"eb765068-ab2a-42f7-9f46-66cfa9745814\") " pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" Nov 01 16:09:45.723097 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.721130 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb765068-ab2a-42f7-9f46-66cfa9745814-workload-socket\") pod \"e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p\" (UID: \"eb765068-ab2a-42f7-9f46-66cfa9745814\") " pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" Nov 01 16:09:45.723097 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.721167 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb765068-ab2a-42f7-9f46-66cfa9745814-istio-envoy\") pod \"e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p\" (UID: \"eb765068-ab2a-42f7-9f46-66cfa9745814\") " pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" Nov 01 16:09:45.723097 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.721261 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb765068-ab2a-42f7-9f46-66cfa9745814-istio-token\") pod \"e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p\" (UID: \"eb765068-ab2a-42f7-9f46-66cfa9745814\") " pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" Nov 01 16:09:45.723097 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.721441 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb765068-ab2a-42f7-9f46-66cfa9745814-istiod-ca-cert\") pod \"e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p\" (UID: \"eb765068-ab2a-42f7-9f46-66cfa9745814\") " pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" Nov 01 16:09:45.723097 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.722329 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb765068-ab2a-42f7-9f46-66cfa9745814-istiod-ca-cert\") pod \"e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p\" (UID: \"eb765068-ab2a-42f7-9f46-66cfa9745814\") " pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" Nov 01 16:09:45.734046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.728568 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb765068-ab2a-42f7-9f46-66cfa9745814-istio-data\") pod \"e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p\" (UID: \"eb765068-ab2a-42f7-9f46-66cfa9745814\") " pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" Nov 01 16:09:45.734046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.729187 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb765068-ab2a-42f7-9f46-66cfa9745814-credential-socket\") pod \"e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p\" (UID: \"eb765068-ab2a-42f7-9f46-66cfa9745814\") " pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" Nov 01 16:09:45.761995 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.747789 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb765068-ab2a-42f7-9f46-66cfa9745814-istio-podinfo\") pod \"e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p\" (UID: \"eb765068-ab2a-42f7-9f46-66cfa9745814\") " pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" Nov 01 16:09:45.761995 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.748459 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb765068-ab2a-42f7-9f46-66cfa9745814-workload-certs\") pod \"e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p\" (UID: \"eb765068-ab2a-42f7-9f46-66cfa9745814\") " pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" Nov 01 16:09:45.761995 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.750566 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb765068-ab2a-42f7-9f46-66cfa9745814-workload-socket\") pod \"e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p\" (UID: \"eb765068-ab2a-42f7-9f46-66cfa9745814\") " pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" Nov 01 16:09:45.796039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.793603 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb765068-ab2a-42f7-9f46-66cfa9745814-istio-envoy\") pod \"e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p\" (UID: \"eb765068-ab2a-42f7-9f46-66cfa9745814\") " pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" Nov 01 16:09:45.796039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.795041 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-458gr\" (UniqueName: \"kubernetes.io/projected/eb765068-ab2a-42f7-9f46-66cfa9745814-kube-api-access-458gr\") pod \"e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p\" (UID: \"eb765068-ab2a-42f7-9f46-66cfa9745814\") " pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" Nov 01 16:09:45.815169 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.815105 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb765068-ab2a-42f7-9f46-66cfa9745814-istio-token\") pod \"e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p\" (UID: \"eb765068-ab2a-42f7-9f46-66cfa9745814\") " pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" Nov 01 16:09:45.845039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:45.843340 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" Nov 01 16:09:46.218971 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:46.205651 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" Nov 01 16:09:46.218971 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:46.216352 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h" event={"ID":"922af45f-b42b-47de-9724-6f2daa958330","Type":"ContainerDied","Data":"475202ef8ebefb883fddf72d8adc33af5dddef5af686ecb3da3eb9c14bd198e1"} Nov 01 16:09:46.218971 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:46.216440 4363 scope.go:122] "RemoveContainer" containerID="fcdfb6c868bbc4f02d17cd5267156ea0c447011e705da88207029c6f3bb277ef" Nov 01 16:09:46.286636 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:46.286549 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h"] Nov 01 16:09:46.300969 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:46.298073 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-59d3a7-openshift-default-dcfc79fcc-4tr8h"] Nov 01 16:09:46.651920 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:46.650215 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p"] Nov 01 16:09:46.692129 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:46.687106 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:09:46.692129 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:46.687202 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:09:46.692129 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:46.687246 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:09:46.988327 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:46.978574 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="922af45f-b42b-47de-9724-6f2daa958330" path="/var/lib/kubelet/pods/922af45f-b42b-47de-9724-6f2daa958330/volumes" Nov 01 16:09:47.215395 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:47.215305 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" event={"ID":"eb765068-ab2a-42f7-9f46-66cfa9745814","Type":"ContainerStarted","Data":"a63f5e962708e8ed721205a200627275b54ed1319bcb23a29768210f81cbf0ce"} Nov 01 16:09:47.215543 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:47.215404 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" event={"ID":"eb765068-ab2a-42f7-9f46-66cfa9745814","Type":"ContainerStarted","Data":"381a5272707ac081ee454043c777ffd8186669485bcad4b3ce705e1cd43195da"} Nov 01 16:09:47.239070 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:47.238994 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" podStartSLOduration=2.23897519 podStartE2EDuration="2.23897519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:09:47.23478116 +0000 UTC m=+1320.566518842" watchObservedRunningTime="2026-11-01 16:09:47.23897519 +0000 UTC m=+1320.570712844" Nov 01 16:09:47.843746 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:47.843712 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" Nov 01 16:09:48.312080 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:09:48.311981 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:09:48.850258 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:48.850229 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" Nov 01 16:09:49.278041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:49.274445 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" Nov 01 16:09:49.287493 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:49.287456 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" Nov 01 16:09:49.816984 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:49.816829 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" Nov 01 16:09:49.921745 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:49.912899 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c10680c0-ac33-4914-931e-15c3728484bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c10680c0-ac33-4914-931e-15c3728484bc-istiod-ca-cert\") pod \"c10680c0-ac33-4914-931e-15c3728484bc\" (UID: \"c10680c0-ac33-4914-931e-15c3728484bc\") " Nov 01 16:09:49.921745 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:49.913004 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c10680c0-ac33-4914-931e-15c3728484bc-kube-api-access-57klw\" (UniqueName: \"kubernetes.io/projected/c10680c0-ac33-4914-931e-15c3728484bc-kube-api-access-57klw\") pod \"c10680c0-ac33-4914-931e-15c3728484bc\" (UID: \"c10680c0-ac33-4914-931e-15c3728484bc\") " Nov 01 16:09:49.921745 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:49.913127 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c10680c0-ac33-4914-931e-15c3728484bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c10680c0-ac33-4914-931e-15c3728484bc-istio-data\") pod \"c10680c0-ac33-4914-931e-15c3728484bc\" (UID: \"c10680c0-ac33-4914-931e-15c3728484bc\") " Nov 01 16:09:49.921745 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:49.913178 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c10680c0-ac33-4914-931e-15c3728484bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c10680c0-ac33-4914-931e-15c3728484bc-istio-envoy\") pod \"c10680c0-ac33-4914-931e-15c3728484bc\" (UID: \"c10680c0-ac33-4914-931e-15c3728484bc\") " Nov 01 16:09:49.921745 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:49.913216 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c10680c0-ac33-4914-931e-15c3728484bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10680c0-ac33-4914-931e-15c3728484bc-credential-socket\") pod \"c10680c0-ac33-4914-931e-15c3728484bc\" (UID: \"c10680c0-ac33-4914-931e-15c3728484bc\") " Nov 01 16:09:49.921745 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:49.913269 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c10680c0-ac33-4914-931e-15c3728484bc-istio-token\" (UniqueName: \"kubernetes.io/projected/c10680c0-ac33-4914-931e-15c3728484bc-istio-token\") pod \"c10680c0-ac33-4914-931e-15c3728484bc\" (UID: \"c10680c0-ac33-4914-931e-15c3728484bc\") " Nov 01 16:09:49.921745 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:49.913330 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c10680c0-ac33-4914-931e-15c3728484bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c10680c0-ac33-4914-931e-15c3728484bc-workload-certs\") pod \"c10680c0-ac33-4914-931e-15c3728484bc\" (UID: \"c10680c0-ac33-4914-931e-15c3728484bc\") " Nov 01 16:09:49.921745 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:49.913648 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c10680c0-ac33-4914-931e-15c3728484bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10680c0-ac33-4914-931e-15c3728484bc-workload-socket\") pod \"c10680c0-ac33-4914-931e-15c3728484bc\" (UID: \"c10680c0-ac33-4914-931e-15c3728484bc\") " Nov 01 16:09:49.921745 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:49.913707 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c10680c0-ac33-4914-931e-15c3728484bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c10680c0-ac33-4914-931e-15c3728484bc-istio-podinfo\") pod \"c10680c0-ac33-4914-931e-15c3728484bc\" (UID: \"c10680c0-ac33-4914-931e-15c3728484bc\") " Nov 01 16:09:49.921745 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:49.915557 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c10680c0-ac33-4914-931e-15c3728484bc-credential-socket" pod "c10680c0-ac33-4914-931e-15c3728484bc" (UID: "c10680c0-ac33-4914-931e-15c3728484bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:09:49.921745 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:49.916424 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c10680c0-ac33-4914-931e-15c3728484bc-istiod-ca-cert" pod "c10680c0-ac33-4914-931e-15c3728484bc" (UID: "c10680c0-ac33-4914-931e-15c3728484bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:09:49.921745 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:49.918222 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c10680c0-ac33-4914-931e-15c3728484bc-workload-certs" pod "c10680c0-ac33-4914-931e-15c3728484bc" (UID: "c10680c0-ac33-4914-931e-15c3728484bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:09:49.921745 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:49.918501 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c10680c0-ac33-4914-931e-15c3728484bc-workload-socket" pod "c10680c0-ac33-4914-931e-15c3728484bc" (UID: "c10680c0-ac33-4914-931e-15c3728484bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:09:49.921745 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:49.919010 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c10680c0-ac33-4914-931e-15c3728484bc-istio-data" pod "c10680c0-ac33-4914-931e-15c3728484bc" (UID: "c10680c0-ac33-4914-931e-15c3728484bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:09:49.950518 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:49.950463 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c10680c0-ac33-4914-931e-15c3728484bc-istio-token" pod "c10680c0-ac33-4914-931e-15c3728484bc" (UID: "c10680c0-ac33-4914-931e-15c3728484bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:09:49.953182 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:49.953149 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c10680c0-ac33-4914-931e-15c3728484bc-istio-envoy" pod "c10680c0-ac33-4914-931e-15c3728484bc" (UID: "c10680c0-ac33-4914-931e-15c3728484bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:09:49.953471 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:49.953421 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c10680c0-ac33-4914-931e-15c3728484bc-kube-api-access-57klw" pod "c10680c0-ac33-4914-931e-15c3728484bc" (UID: "c10680c0-ac33-4914-931e-15c3728484bc"). InnerVolumeSpecName "kube-api-access-57klw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:09:49.958625 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:49.958599 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c10680c0-ac33-4914-931e-15c3728484bc-istio-podinfo" pod "c10680c0-ac33-4914-931e-15c3728484bc" (UID: "c10680c0-ac33-4914-931e-15c3728484bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:09:49.963064 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:49.962979 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" Nov 01 16:09:50.021065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.015456 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41d1f840-a59c-4f46-84b0-d04c3cd1d500-istio-token\" (UniqueName: \"kubernetes.io/projected/41d1f840-a59c-4f46-84b0-d04c3cd1d500-istio-token\") pod \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\" (UID: \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\") " Nov 01 16:09:50.021065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.015525 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41d1f840-a59c-4f46-84b0-d04c3cd1d500-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d1f840-a59c-4f46-84b0-d04c3cd1d500-workload-socket\") pod \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\" (UID: \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\") " Nov 01 16:09:50.021065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.015739 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41d1f840-a59c-4f46-84b0-d04c3cd1d500-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41d1f840-a59c-4f46-84b0-d04c3cd1d500-istio-podinfo\") pod \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\" (UID: \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\") " Nov 01 16:09:50.021065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.015833 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41d1f840-a59c-4f46-84b0-d04c3cd1d500-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41d1f840-a59c-4f46-84b0-d04c3cd1d500-istio-data\") pod \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\" (UID: \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\") " Nov 01 16:09:50.021065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.015910 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41d1f840-a59c-4f46-84b0-d04c3cd1d500-kube-api-access-c8zv6\" (UniqueName: \"kubernetes.io/projected/41d1f840-a59c-4f46-84b0-d04c3cd1d500-kube-api-access-c8zv6\") pod \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\" (UID: \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\") " Nov 01 16:09:50.021065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.016053 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41d1f840-a59c-4f46-84b0-d04c3cd1d500-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41d1f840-a59c-4f46-84b0-d04c3cd1d500-istiod-ca-cert\") pod \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\" (UID: \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\") " Nov 01 16:09:50.021065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.016086 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41d1f840-a59c-4f46-84b0-d04c3cd1d500-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d1f840-a59c-4f46-84b0-d04c3cd1d500-credential-socket\") pod \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\" (UID: \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\") " Nov 01 16:09:50.021065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.016137 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41d1f840-a59c-4f46-84b0-d04c3cd1d500-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41d1f840-a59c-4f46-84b0-d04c3cd1d500-workload-certs\") pod \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\" (UID: \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\") " Nov 01 16:09:50.021065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.016202 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41d1f840-a59c-4f46-84b0-d04c3cd1d500-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41d1f840-a59c-4f46-84b0-d04c3cd1d500-istio-envoy\") pod \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\" (UID: \"41d1f840-a59c-4f46-84b0-d04c3cd1d500\") " Nov 01 16:09:50.021065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.017295 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c10680c0-ac33-4914-931e-15c3728484bc-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:50.021065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.017320 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10680c0-ac33-4914-931e-15c3728484bc-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:50.021065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.017364 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c10680c0-ac33-4914-931e-15c3728484bc-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:50.021065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.017379 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c10680c0-ac33-4914-931e-15c3728484bc-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:50.021065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.017396 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57klw\" (UniqueName: \"kubernetes.io/projected/c10680c0-ac33-4914-931e-15c3728484bc-kube-api-access-57klw\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:50.021065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.017413 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c10680c0-ac33-4914-931e-15c3728484bc-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:50.021065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.017427 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c10680c0-ac33-4914-931e-15c3728484bc-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:50.021065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.017441 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10680c0-ac33-4914-931e-15c3728484bc-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:50.021065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.017456 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c10680c0-ac33-4914-931e-15c3728484bc-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:50.021065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.017975 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d1f840-a59c-4f46-84b0-d04c3cd1d500-istio-data" pod "41d1f840-a59c-4f46-84b0-d04c3cd1d500" (UID: "41d1f840-a59c-4f46-84b0-d04c3cd1d500"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:09:50.021065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.018571 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d1f840-a59c-4f46-84b0-d04c3cd1d500-credential-socket" pod "41d1f840-a59c-4f46-84b0-d04c3cd1d500" (UID: "41d1f840-a59c-4f46-84b0-d04c3cd1d500"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:09:50.021065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.018869 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d1f840-a59c-4f46-84b0-d04c3cd1d500-workload-socket" pod "41d1f840-a59c-4f46-84b0-d04c3cd1d500" (UID: "41d1f840-a59c-4f46-84b0-d04c3cd1d500"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:09:50.021065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.019176 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d1f840-a59c-4f46-84b0-d04c3cd1d500-workload-certs" pod "41d1f840-a59c-4f46-84b0-d04c3cd1d500" (UID: "41d1f840-a59c-4f46-84b0-d04c3cd1d500"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:09:50.021065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.019454 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41d1f840-a59c-4f46-84b0-d04c3cd1d500-istiod-ca-cert" pod "41d1f840-a59c-4f46-84b0-d04c3cd1d500" (UID: "41d1f840-a59c-4f46-84b0-d04c3cd1d500"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:09:50.021065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.020705 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41d1f840-a59c-4f46-84b0-d04c3cd1d500-istio-token" pod "41d1f840-a59c-4f46-84b0-d04c3cd1d500" (UID: "41d1f840-a59c-4f46-84b0-d04c3cd1d500"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:09:50.021065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.020864 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d1f840-a59c-4f46-84b0-d04c3cd1d500-istio-envoy" pod "41d1f840-a59c-4f46-84b0-d04c3cd1d500" (UID: "41d1f840-a59c-4f46-84b0-d04c3cd1d500"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:09:50.022425 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.022355 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41d1f840-a59c-4f46-84b0-d04c3cd1d500-kube-api-access-c8zv6" pod "41d1f840-a59c-4f46-84b0-d04c3cd1d500" (UID: "41d1f840-a59c-4f46-84b0-d04c3cd1d500"). InnerVolumeSpecName "kube-api-access-c8zv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:09:50.024158 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.024124 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41d1f840-a59c-4f46-84b0-d04c3cd1d500-istio-podinfo" pod "41d1f840-a59c-4f46-84b0-d04c3cd1d500" (UID: "41d1f840-a59c-4f46-84b0-d04c3cd1d500"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:09:50.129130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.126871 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41d1f840-a59c-4f46-84b0-d04c3cd1d500-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:50.129130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.126905 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41d1f840-a59c-4f46-84b0-d04c3cd1d500-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:50.129130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.126923 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8zv6\" (UniqueName: \"kubernetes.io/projected/41d1f840-a59c-4f46-84b0-d04c3cd1d500-kube-api-access-c8zv6\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:50.129130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.126940 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41d1f840-a59c-4f46-84b0-d04c3cd1d500-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:50.129130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.126954 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d1f840-a59c-4f46-84b0-d04c3cd1d500-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:50.129130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.126968 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41d1f840-a59c-4f46-84b0-d04c3cd1d500-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:50.129130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.126982 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41d1f840-a59c-4f46-84b0-d04c3cd1d500-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:50.129130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.126995 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41d1f840-a59c-4f46-84b0-d04c3cd1d500-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:50.129130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.127010 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41d1f840-a59c-4f46-84b0-d04c3cd1d500-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:50.137084 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.136833 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-2zh4r"] Nov 01 16:09:50.138466 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.138436 4363 cpu_manager.go:414] "containerMap: removing container" podUID="922af45f-b42b-47de-9724-6f2daa958330" containerName="istio-proxy" Nov 01 16:09:50.138466 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.138463 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="922af45f-b42b-47de-9724-6f2daa958330" containerName="istio-proxy" Nov 01 16:09:50.138665 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.138515 4363 cpu_manager.go:414] "containerMap: removing container" podUID="c10680c0-ac33-4914-931e-15c3728484bc" containerName="istio-proxy" Nov 01 16:09:50.138665 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.138524 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c10680c0-ac33-4914-931e-15c3728484bc" containerName="istio-proxy" Nov 01 16:09:50.138665 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.138595 4363 cpu_manager.go:414] "containerMap: removing container" podUID="41d1f840-a59c-4f46-84b0-d04c3cd1d500" containerName="istio-proxy" Nov 01 16:09:50.138665 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.138604 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41d1f840-a59c-4f46-84b0-d04c3cd1d500" containerName="istio-proxy" Nov 01 16:09:50.138851 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.138782 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="41d1f840-a59c-4f46-84b0-d04c3cd1d500" containerName="istio-proxy" Nov 01 16:09:50.138851 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.138803 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="c10680c0-ac33-4914-931e-15c3728484bc" containerName="istio-proxy" Nov 01 16:09:50.143695 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.143672 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-2zh4r" Nov 01 16:09:50.156067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.156036 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-2zh4r"] Nov 01 16:09:50.175463 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.175376 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-76063d10-serving-cert\"" Nov 01 16:09:50.231324 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.231283 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21e3bc0d-84f1-4ae1-baaa-97509d961b89-metrics-certs\") pod \"maas-api-e2e-mt-76063d10-78c6f5c5f9-2zh4r\" (UID: \"21e3bc0d-84f1-4ae1-baaa-97509d961b89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-2zh4r" Nov 01 16:09:50.231506 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.231494 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h9md\" (UniqueName: \"kubernetes.io/projected/21e3bc0d-84f1-4ae1-baaa-97509d961b89-kube-api-access-8h9md\") pod \"maas-api-e2e-mt-76063d10-78c6f5c5f9-2zh4r\" (UID: \"21e3bc0d-84f1-4ae1-baaa-97509d961b89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-2zh4r" Nov 01 16:09:50.231606 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.231591 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21e3bc0d-84f1-4ae1-baaa-97509d961b89-maas-api-tls\") pod \"maas-api-e2e-mt-76063d10-78c6f5c5f9-2zh4r\" (UID: \"21e3bc0d-84f1-4ae1-baaa-97509d961b89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-2zh4r" Nov 01 16:09:50.231712 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.231695 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e3bc0d-84f1-4ae1-baaa-97509d961b89-openshift-service-ca\") pod \"maas-api-e2e-mt-76063d10-78c6f5c5f9-2zh4r\" (UID: \"21e3bc0d-84f1-4ae1-baaa-97509d961b89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-2zh4r" Nov 01 16:09:50.296066 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.286424 4363 generic.go:356] "Generic (PLEG): container finished" podID="c10680c0-ac33-4914-931e-15c3728484bc" containerID="e323a502222833dc923d26c2a6ec20177c3ac61e1ab12e27f21f15b860a277c0" exitCode=0 Nov 01 16:09:50.296066 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.286474 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" Nov 01 16:09:50.296066 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.286517 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" event={"ID":"c10680c0-ac33-4914-931e-15c3728484bc","Type":"ContainerDied","Data":"e323a502222833dc923d26c2a6ec20177c3ac61e1ab12e27f21f15b860a277c0"} Nov 01 16:09:50.296066 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.286583 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn" event={"ID":"c10680c0-ac33-4914-931e-15c3728484bc","Type":"ContainerDied","Data":"9db9da70022d60f8ae7c4ff0da869d7671a396e30779f97fce6d04d6a6ddab01"} Nov 01 16:09:50.296066 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.286617 4363 scope.go:122] "RemoveContainer" containerID="e323a502222833dc923d26c2a6ec20177c3ac61e1ab12e27f21f15b860a277c0" Nov 01 16:09:50.296066 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.288963 4363 generic.go:356] "Generic (PLEG): container finished" podID="41d1f840-a59c-4f46-84b0-d04c3cd1d500" containerID="f11b60de5c029ba956d3e5cef1d1d8fcc91a7654adb0f2f27eaf452d502bf650" exitCode=0 Nov 01 16:09:50.296066 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.289065 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" event={"ID":"41d1f840-a59c-4f46-84b0-d04c3cd1d500","Type":"ContainerDied","Data":"f11b60de5c029ba956d3e5cef1d1d8fcc91a7654adb0f2f27eaf452d502bf650"} Nov 01 16:09:50.296066 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.289113 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" event={"ID":"41d1f840-a59c-4f46-84b0-d04c3cd1d500","Type":"ContainerDied","Data":"cd1d0422ce395e279f51c9dd3135b764e149cfb737775c01377443f90a73ffd5"} Nov 01 16:09:50.296066 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.289195 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt" Nov 01 16:09:50.321315 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.304818 4363 scope.go:122] "RemoveContainer" containerID="e323a502222833dc923d26c2a6ec20177c3ac61e1ab12e27f21f15b860a277c0" Nov 01 16:09:50.321315 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:09:50.305466 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e323a502222833dc923d26c2a6ec20177c3ac61e1ab12e27f21f15b860a277c0\": container with ID starting with e323a502222833dc923d26c2a6ec20177c3ac61e1ab12e27f21f15b860a277c0 not found: ID does not exist" containerID="e323a502222833dc923d26c2a6ec20177c3ac61e1ab12e27f21f15b860a277c0" Nov 01 16:09:50.321315 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.305507 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e323a502222833dc923d26c2a6ec20177c3ac61e1ab12e27f21f15b860a277c0"} err="failed to get container status \"e323a502222833dc923d26c2a6ec20177c3ac61e1ab12e27f21f15b860a277c0\": rpc error: code = NotFound desc = could not find container \"e323a502222833dc923d26c2a6ec20177c3ac61e1ab12e27f21f15b860a277c0\": container with ID starting with e323a502222833dc923d26c2a6ec20177c3ac61e1ab12e27f21f15b860a277c0 not found: ID does not exist" Nov 01 16:09:50.321315 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.305531 4363 scope.go:122] "RemoveContainer" containerID="f11b60de5c029ba956d3e5cef1d1d8fcc91a7654adb0f2f27eaf452d502bf650" Nov 01 16:09:50.383110 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.372236 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21e3bc0d-84f1-4ae1-baaa-97509d961b89-metrics-certs\") pod \"maas-api-e2e-mt-76063d10-78c6f5c5f9-2zh4r\" (UID: \"21e3bc0d-84f1-4ae1-baaa-97509d961b89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-2zh4r" Nov 01 16:09:50.383110 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.372499 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h9md\" (UniqueName: \"kubernetes.io/projected/21e3bc0d-84f1-4ae1-baaa-97509d961b89-kube-api-access-8h9md\") pod \"maas-api-e2e-mt-76063d10-78c6f5c5f9-2zh4r\" (UID: \"21e3bc0d-84f1-4ae1-baaa-97509d961b89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-2zh4r" Nov 01 16:09:50.383110 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.372636 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21e3bc0d-84f1-4ae1-baaa-97509d961b89-maas-api-tls\") pod \"maas-api-e2e-mt-76063d10-78c6f5c5f9-2zh4r\" (UID: \"21e3bc0d-84f1-4ae1-baaa-97509d961b89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-2zh4r" Nov 01 16:09:50.383110 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.372803 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e3bc0d-84f1-4ae1-baaa-97509d961b89-openshift-service-ca\") pod \"maas-api-e2e-mt-76063d10-78c6f5c5f9-2zh4r\" (UID: \"21e3bc0d-84f1-4ae1-baaa-97509d961b89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-2zh4r" Nov 01 16:09:50.383110 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.373772 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e3bc0d-84f1-4ae1-baaa-97509d961b89-openshift-service-ca\") pod \"maas-api-e2e-mt-76063d10-78c6f5c5f9-2zh4r\" (UID: \"21e3bc0d-84f1-4ae1-baaa-97509d961b89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-2zh4r" Nov 01 16:09:50.454092 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.416970 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h9md\" (UniqueName: \"kubernetes.io/projected/21e3bc0d-84f1-4ae1-baaa-97509d961b89-kube-api-access-8h9md\") pod \"maas-api-e2e-mt-76063d10-78c6f5c5f9-2zh4r\" (UID: \"21e3bc0d-84f1-4ae1-baaa-97509d961b89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-2zh4r" Nov 01 16:09:50.454092 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.423923 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21e3bc0d-84f1-4ae1-baaa-97509d961b89-maas-api-tls\") pod \"maas-api-e2e-mt-76063d10-78c6f5c5f9-2zh4r\" (UID: \"21e3bc0d-84f1-4ae1-baaa-97509d961b89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-2zh4r" Nov 01 16:09:50.454092 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.442406 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21e3bc0d-84f1-4ae1-baaa-97509d961b89-metrics-certs\") pod \"maas-api-e2e-mt-76063d10-78c6f5c5f9-2zh4r\" (UID: \"21e3bc0d-84f1-4ae1-baaa-97509d961b89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-2zh4r" Nov 01 16:09:50.454092 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.447589 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn"] Nov 01 16:09:50.523781 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.491366 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-2zh4r" Nov 01 16:09:50.523781 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.495064 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-pnlqn"] Nov 01 16:09:50.523781 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.502374 4363 scope.go:122] "RemoveContainer" containerID="f11b60de5c029ba956d3e5cef1d1d8fcc91a7654adb0f2f27eaf452d502bf650" Nov 01 16:09:50.523781 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:09:50.502899 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f11b60de5c029ba956d3e5cef1d1d8fcc91a7654adb0f2f27eaf452d502bf650\": container with ID starting with f11b60de5c029ba956d3e5cef1d1d8fcc91a7654adb0f2f27eaf452d502bf650 not found: ID does not exist" containerID="f11b60de5c029ba956d3e5cef1d1d8fcc91a7654adb0f2f27eaf452d502bf650" Nov 01 16:09:50.523781 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.502953 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f11b60de5c029ba956d3e5cef1d1d8fcc91a7654adb0f2f27eaf452d502bf650"} err="failed to get container status \"f11b60de5c029ba956d3e5cef1d1d8fcc91a7654adb0f2f27eaf452d502bf650\": rpc error: code = NotFound desc = could not find container \"f11b60de5c029ba956d3e5cef1d1d8fcc91a7654adb0f2f27eaf452d502bf650\": container with ID starting with f11b60de5c029ba956d3e5cef1d1d8fcc91a7654adb0f2f27eaf452d502bf650 not found: ID does not exist" Nov 01 16:09:50.538403 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.538166 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt"] Nov 01 16:09:50.593168 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.577156 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-xdtnt"] Nov 01 16:09:50.664641 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.664601 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-022f62-6b68c9f45f-ls8hp"] Nov 01 16:09:50.702246 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.698782 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-022f62-6b68c9f45f-ls8hp" Nov 01 16:09:50.702246 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.699501 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-022f62-6b6cb577c9-w46bj"] Nov 01 16:09:50.707569 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.705467 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-022f62\"" Nov 01 16:09:50.707569 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.706352 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-022f62-6b68c9f45f-ls8hp"] Nov 01 16:09:50.707569 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.706454 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-022f62-6b6cb577c9-w46bj" Nov 01 16:09:50.707840 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.707690 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-022f62-6b6cb577c9-w46bj"] Nov 01 16:09:50.711383 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.710895 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-022f62-dockercfg-9zncp\"" Nov 01 16:09:50.800559 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.800517 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jct8\" (UniqueName: \"kubernetes.io/projected/ea6d9e7a-1a13-4ead-9cb2-c3408313dd09-kube-api-access-8jct8\") pod \"payload-processing-e2e-api-a-022f62-6b6cb577c9-w46bj\" (UID: \"ea6d9e7a-1a13-4ead-9cb2-c3408313dd09\") " pod="openshift-ingress/payload-processing-e2e-api-a-022f62-6b6cb577c9-w46bj" Nov 01 16:09:50.800814 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.800592 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrpsz\" (UniqueName: \"kubernetes.io/projected/067dcb6b-1a6d-4818-b42d-9e3617e2302c-kube-api-access-nrpsz\") pod \"payload-pre-processing-e2e-api-a-022f62-6b68c9f45f-ls8hp\" (UID: \"067dcb6b-1a6d-4818-b42d-9e3617e2302c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-022f62-6b68c9f45f-ls8hp" Nov 01 16:09:50.800814 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.800667 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea6d9e7a-1a13-4ead-9cb2-c3408313dd09-plugins-config-volume\") pod \"payload-processing-e2e-api-a-022f62-6b6cb577c9-w46bj\" (UID: \"ea6d9e7a-1a13-4ead-9cb2-c3408313dd09\") " pod="openshift-ingress/payload-processing-e2e-api-a-022f62-6b6cb577c9-w46bj" Nov 01 16:09:50.800814 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.800791 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/067dcb6b-1a6d-4818-b42d-9e3617e2302c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-022f62-6b68c9f45f-ls8hp\" (UID: \"067dcb6b-1a6d-4818-b42d-9e3617e2302c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-022f62-6b68c9f45f-ls8hp" Nov 01 16:09:50.801681 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.801656 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41d1f840-a59c-4f46-84b0-d04c3cd1d500" path="/var/lib/kubelet/pods/41d1f840-a59c-4f46-84b0-d04c3cd1d500/volumes" Nov 01 16:09:50.802209 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.802195 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c10680c0-ac33-4914-931e-15c3728484bc" path="/var/lib/kubelet/pods/c10680c0-ac33-4914-931e-15c3728484bc/volumes" Nov 01 16:09:50.894279 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.894252 4363 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 16:09:50.894806 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.894776 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-2zh4r"] Nov 01 16:09:50.906613 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.906431 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/067dcb6b-1a6d-4818-b42d-9e3617e2302c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-022f62-6b68c9f45f-ls8hp\" (UID: \"067dcb6b-1a6d-4818-b42d-9e3617e2302c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-022f62-6b68c9f45f-ls8hp" Nov 01 16:09:50.906984 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.906795 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jct8\" (UniqueName: \"kubernetes.io/projected/ea6d9e7a-1a13-4ead-9cb2-c3408313dd09-kube-api-access-8jct8\") pod \"payload-processing-e2e-api-a-022f62-6b6cb577c9-w46bj\" (UID: \"ea6d9e7a-1a13-4ead-9cb2-c3408313dd09\") " pod="openshift-ingress/payload-processing-e2e-api-a-022f62-6b6cb577c9-w46bj" Nov 01 16:09:50.906984 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.906862 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrpsz\" (UniqueName: \"kubernetes.io/projected/067dcb6b-1a6d-4818-b42d-9e3617e2302c-kube-api-access-nrpsz\") pod \"payload-pre-processing-e2e-api-a-022f62-6b68c9f45f-ls8hp\" (UID: \"067dcb6b-1a6d-4818-b42d-9e3617e2302c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-022f62-6b68c9f45f-ls8hp" Nov 01 16:09:50.906984 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.906938 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea6d9e7a-1a13-4ead-9cb2-c3408313dd09-plugins-config-volume\") pod \"payload-processing-e2e-api-a-022f62-6b6cb577c9-w46bj\" (UID: \"ea6d9e7a-1a13-4ead-9cb2-c3408313dd09\") " pod="openshift-ingress/payload-processing-e2e-api-a-022f62-6b6cb577c9-w46bj" Nov 01 16:09:50.907572 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.907545 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/067dcb6b-1a6d-4818-b42d-9e3617e2302c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-022f62-6b68c9f45f-ls8hp\" (UID: \"067dcb6b-1a6d-4818-b42d-9e3617e2302c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-022f62-6b68c9f45f-ls8hp" Nov 01 16:09:50.907848 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.907826 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea6d9e7a-1a13-4ead-9cb2-c3408313dd09-plugins-config-volume\") pod \"payload-processing-e2e-api-a-022f62-6b6cb577c9-w46bj\" (UID: \"ea6d9e7a-1a13-4ead-9cb2-c3408313dd09\") " pod="openshift-ingress/payload-processing-e2e-api-a-022f62-6b6cb577c9-w46bj" Nov 01 16:09:50.960785 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.960628 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrpsz\" (UniqueName: \"kubernetes.io/projected/067dcb6b-1a6d-4818-b42d-9e3617e2302c-kube-api-access-nrpsz\") pod \"payload-pre-processing-e2e-api-a-022f62-6b68c9f45f-ls8hp\" (UID: \"067dcb6b-1a6d-4818-b42d-9e3617e2302c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-022f62-6b68c9f45f-ls8hp" Nov 01 16:09:50.965039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:50.961127 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jct8\" (UniqueName: \"kubernetes.io/projected/ea6d9e7a-1a13-4ead-9cb2-c3408313dd09-kube-api-access-8jct8\") pod \"payload-processing-e2e-api-a-022f62-6b6cb577c9-w46bj\" (UID: \"ea6d9e7a-1a13-4ead-9cb2-c3408313dd09\") " pod="openshift-ingress/payload-processing-e2e-api-a-022f62-6b6cb577c9-w46bj" Nov 01 16:09:51.042040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:51.041273 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-022f62-6b68c9f45f-ls8hp" Nov 01 16:09:51.059074 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:51.052303 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-022f62-6b6cb577c9-w46bj" Nov 01 16:09:51.321060 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:51.315557 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-2zh4r" event={"ID":"21e3bc0d-84f1-4ae1-baaa-97509d961b89","Type":"ContainerStarted","Data":"ff6f2ce3352e0a8602571bb0ee3bcb50fe1d7314fd8670211d4b9a5282611571"} Nov 01 16:09:52.010831 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:52.010800 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-022f62-6b68c9f45f-ls8hp"] Nov 01 16:09:52.012496 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:09:52.012459 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod067dcb6b_1a6d_4818_b42d_9e3617e2302c.slice/crio-2eaeb656b7269da709a52b21f437662be80699a6b2b12ba12979507639f66afe WatchSource:0}: Error finding container 2eaeb656b7269da709a52b21f437662be80699a6b2b12ba12979507639f66afe: Status 404 returned error can't find the container with id 2eaeb656b7269da709a52b21f437662be80699a6b2b12ba12979507639f66afe Nov 01 16:09:52.019337 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:52.019294 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-022f62-6b6cb577c9-w46bj"] Nov 01 16:09:52.032142 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:09:52.032113 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea6d9e7a_1a13_4ead_9cb2_c3408313dd09.slice/crio-356c2ae8f22f0949612771e8407d9a4265f78b687ed31db782d6e164068dba3b WatchSource:0}: Error finding container 356c2ae8f22f0949612771e8407d9a4265f78b687ed31db782d6e164068dba3b: Status 404 returned error can't find the container with id 356c2ae8f22f0949612771e8407d9a4265f78b687ed31db782d6e164068dba3b Nov 01 16:09:52.340155 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:52.340059 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-2zh4r" event={"ID":"21e3bc0d-84f1-4ae1-baaa-97509d961b89","Type":"ContainerStarted","Data":"b85a4b4298ecdae0a78c4771a0e57de23e4a8040b699d19fc80295f83ce9823a"} Nov 01 16:09:52.340445 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:52.340419 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-2zh4r" Nov 01 16:09:52.341928 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:52.341900 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-022f62-6b6cb577c9-w46bj" event={"ID":"ea6d9e7a-1a13-4ead-9cb2-c3408313dd09","Type":"ContainerStarted","Data":"086287d47ac13eebc9a7aa898b2ad79ea2c0adf687902102e7867023bd9552bc"} Nov 01 16:09:52.342055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:52.341956 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-022f62-6b6cb577c9-w46bj" Nov 01 16:09:52.342055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:52.341980 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-022f62-6b6cb577c9-w46bj" event={"ID":"ea6d9e7a-1a13-4ead-9cb2-c3408313dd09","Type":"ContainerStarted","Data":"356c2ae8f22f0949612771e8407d9a4265f78b687ed31db782d6e164068dba3b"} Nov 01 16:09:52.343848 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:52.343823 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-022f62-6b68c9f45f-ls8hp" event={"ID":"067dcb6b-1a6d-4818-b42d-9e3617e2302c","Type":"ContainerStarted","Data":"da77b63b1905450ae9d679325b0b975eabcecd922378298d69e5c7ade35272f4"} Nov 01 16:09:52.343955 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:52.343870 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-022f62-6b68c9f45f-ls8hp" event={"ID":"067dcb6b-1a6d-4818-b42d-9e3617e2302c","Type":"ContainerStarted","Data":"2eaeb656b7269da709a52b21f437662be80699a6b2b12ba12979507639f66afe"} Nov 01 16:09:52.343955 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:52.343917 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-022f62-6b68c9f45f-ls8hp" Nov 01 16:09:52.360172 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:52.360120 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-2zh4r" podStartSLOduration=1.7062672559999998 podStartE2EDuration="2.36010652s" totalImagesPullingTime="653.839264ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:09:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:09:52.356063233 +0000 UTC m=+1325.687800887" watchObservedRunningTime="2026-11-01 16:09:52.36010652 +0000 UTC m=+1325.691844170" Nov 01 16:09:52.372343 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:52.372299 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-022f62-6b6cb577c9-w46bj" podStartSLOduration=2.372283994 podStartE2EDuration="2.372283994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:09:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:09:52.371118258 +0000 UTC m=+1325.702855909" watchObservedRunningTime="2026-11-01 16:09:52.372283994 +0000 UTC m=+1325.704021645" Nov 01 16:09:53.216613 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:53.212223 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-vrmjz" Nov 01 16:09:53.216613 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:53.213225 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-lqtdt" Nov 01 16:09:53.291057 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:53.288629 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-022f62-6b68c9f45f-ls8hp" podStartSLOduration=3.288611289 podStartE2EDuration="3.288611289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:09:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:09:52.386129441 +0000 UTC m=+1325.717867090" watchObservedRunningTime="2026-11-01 16:09:53.288611289 +0000 UTC m=+1326.620348940" Nov 01 16:09:58.358319 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:58.358288 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-2zh4r" Nov 01 16:09:59.628045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:59.626214 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk"] Nov 01 16:09:59.628045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:59.626502 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" podUID="4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459" containerName="istio-proxy" containerID="cri-o://5d4172fa160490c368e0f1fde63fb6de21ef76011542bd53966bb6c2abe80c2d" gracePeriod=30 Nov 01 16:09:59.646846 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:59.645973 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k"] Nov 01 16:09:59.646846 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:09:59.646249 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" podUID="4070ca0f-1230-4eed-992c-b3f6bce6a4bb" containerName="istio-proxy" containerID="cri-o://0f8a65fdecd9f9f6a94b9d39b2f3d7f99e0466c5a33d37a06bb6903f4f8bf4b3" gracePeriod=30 Nov 01 16:10:00.017081 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:00.015678 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc"] Nov 01 16:10:00.017081 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:00.015951 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" podUID="bfe59fad-a9c8-4073-81c6-11f4eda63086" containerName="istio-proxy" containerID="cri-o://c472bea28321ed20bfc49b0b7f82215ac7384e18f1ac96ffcc1fceb0eba005d3" gracePeriod=30 Nov 01 16:10:00.023971 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:00.023576 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws"] Nov 01 16:10:00.024180 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:00.024137 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" podUID="07c0b57d-ed0c-4366-9716-256714694541" containerName="istio-proxy" containerID="cri-o://af74ce07a3b12fa506a685bf3df5118f906d5acf180c7cfbf31991680d62a347" gracePeriod=30 Nov 01 16:10:00.032270 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:00.031697 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8"] Nov 01 16:10:00.032270 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:00.031903 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" podUID="74344814-e7cb-4b6f-b295-bb441f192023" containerName="istio-proxy" containerID="cri-o://00a54faa1bcc3176a4d7bb7c40007969ddd95f826cee12a7c991e871bfdf9a2e" gracePeriod=30 Nov 01 16:10:00.080758 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:00.076281 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz"] Nov 01 16:10:00.080758 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:00.076529 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" podUID="408aa757-573f-4cd4-b81d-c26c9af252f2" containerName="istio-proxy" containerID="cri-o://becc68c69d5083a9e078509dcb7c90251f33191fb317ea61e3a59037fa7cee65" gracePeriod=30 Nov 01 16:10:00.087561 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:00.087530 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl"] Nov 01 16:10:00.087862 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:00.087760 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" podUID="0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0" containerName="istio-proxy" containerID="cri-o://5caefde19962b8bdfbe8aa21d27864e87a211f1a46aec0201e9fed1c1d85b500" gracePeriod=30 Nov 01 16:10:00.153864 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:00.153828 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79"] Nov 01 16:10:00.154123 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:00.154093 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" podUID="edcab30a-6e4e-4019-9fd7-96fe7b085711" containerName="istio-proxy" containerID="cri-o://15bfb16861b677eac440a82beb7cd7f2a9964e25b9351c73f69ca970eeb0ebed" gracePeriod=30 Nov 01 16:10:00.159555 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:00.159524 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8"] Nov 01 16:10:00.160011 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:00.159959 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" podUID="3b704396-3411-40af-95fe-4b67a41ae69f" containerName="istio-proxy" containerID="cri-o://6d837305a83abbd54c37311b0b33ff057014a6ad06193106ea3fe7f41ed5e19f" gracePeriod=30 Nov 01 16:10:03.356391 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:03.356358 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-022f62-6b6cb577c9-w46bj" Nov 01 16:10:03.356883 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:03.356780 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-022f62-6b68c9f45f-ls8hp" Nov 01 16:10:05.453265 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.453117 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" Nov 01 16:10:05.460067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.457352 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" Nov 01 16:10:05.595779 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.595749 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-istio-envoy\") pod \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\" (UID: \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\") " Nov 01 16:10:05.595985 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.595945 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-istio-podinfo\") pod \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\" (UID: \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\") " Nov 01 16:10:05.596097 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.596011 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-istio-podinfo\") pod \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\" (UID: \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\") " Nov 01 16:10:05.596222 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.596100 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-istiod-ca-cert\") pod \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\" (UID: \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\") " Nov 01 16:10:05.596222 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.596137 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-istiod-ca-cert\") pod \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\" (UID: \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\") " Nov 01 16:10:05.596222 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.596181 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-istio-token\" (UniqueName: \"kubernetes.io/projected/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-istio-token\") pod \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\" (UID: \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\") " Nov 01 16:10:05.596222 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.596218 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-workload-certs\") pod \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\" (UID: \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\") " Nov 01 16:10:05.596515 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.596244 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-kube-api-access-xg29p\" (UniqueName: \"kubernetes.io/projected/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-kube-api-access-xg29p\") pod \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\" (UID: \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\") " Nov 01 16:10:05.596515 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.596325 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-istio-envoy\") pod \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\" (UID: \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\") " Nov 01 16:10:05.596515 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.596354 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-istio-data\") pod \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\" (UID: \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\") " Nov 01 16:10:05.596515 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.596437 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-istio-data\") pod \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\" (UID: \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\") " Nov 01 16:10:05.596515 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.596468 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-credential-socket\") pod \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\" (UID: \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\") " Nov 01 16:10:05.596882 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.596559 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-istio-token\" (UniqueName: \"kubernetes.io/projected/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-istio-token\") pod \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\" (UID: \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\") " Nov 01 16:10:05.597428 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.597400 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-workload-socket\") pod \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\" (UID: \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\") " Nov 01 16:10:05.597565 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.597522 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-credential-socket\") pod \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\" (UID: \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\") " Nov 01 16:10:05.597647 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.597566 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-kube-api-access-l55gc\" (UniqueName: \"kubernetes.io/projected/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-kube-api-access-l55gc\") pod \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\" (UID: \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\") " Nov 01 16:10:05.597647 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.597616 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-workload-certs\") pod \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\" (UID: \"4070ca0f-1230-4eed-992c-b3f6bce6a4bb\") " Nov 01 16:10:05.597773 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.597716 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-workload-socket\") pod \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\" (UID: \"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459\") " Nov 01 16:10:05.599052 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.598841 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-istiod-ca-cert" pod "4070ca0f-1230-4eed-992c-b3f6bce6a4bb" (UID: "4070ca0f-1230-4eed-992c-b3f6bce6a4bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:10:05.599272 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.599209 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-workload-socket" pod "4070ca0f-1230-4eed-992c-b3f6bce6a4bb" (UID: "4070ca0f-1230-4eed-992c-b3f6bce6a4bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:05.599432 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.599410 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-credential-socket" pod "4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459" (UID: "4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:05.601838 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.601773 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-kube-api-access-l55gc" pod "4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459" (UID: "4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459"). InnerVolumeSpecName "kube-api-access-l55gc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:05.609741 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.602096 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-workload-certs" pod "4070ca0f-1230-4eed-992c-b3f6bce6a4bb" (UID: "4070ca0f-1230-4eed-992c-b3f6bce6a4bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:05.609741 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.602283 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-workload-socket" pod "4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459" (UID: "4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:05.609741 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.602668 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-workload-certs" pod "4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459" (UID: "4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:05.609741 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.602940 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-istiod-ca-cert" pod "4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459" (UID: "4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:10:05.609741 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.603133 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-istio-data" pod "4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459" (UID: "4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:05.609741 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.603274 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-istio-data" pod "4070ca0f-1230-4eed-992c-b3f6bce6a4bb" (UID: "4070ca0f-1230-4eed-992c-b3f6bce6a4bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:05.610178 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.609748 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-kube-api-access-xg29p" pod "4070ca0f-1230-4eed-992c-b3f6bce6a4bb" (UID: "4070ca0f-1230-4eed-992c-b3f6bce6a4bb"). InnerVolumeSpecName "kube-api-access-xg29p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:05.610234 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.610207 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-credential-socket" pod "4070ca0f-1230-4eed-992c-b3f6bce6a4bb" (UID: "4070ca0f-1230-4eed-992c-b3f6bce6a4bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:05.618067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.610841 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-istio-podinfo" pod "4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459" (UID: "4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:10:05.618067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.614235 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-istio-token" pod "4070ca0f-1230-4eed-992c-b3f6bce6a4bb" (UID: "4070ca0f-1230-4eed-992c-b3f6bce6a4bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:05.623442 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.623392 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-istio-envoy" pod "4070ca0f-1230-4eed-992c-b3f6bce6a4bb" (UID: "4070ca0f-1230-4eed-992c-b3f6bce6a4bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:05.623563 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.623492 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-istio-token" pod "4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459" (UID: "4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:05.623715 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.623687 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-istio-envoy" pod "4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459" (UID: "4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:05.638048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.636902 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-istio-podinfo" pod "4070ca0f-1230-4eed-992c-b3f6bce6a4bb" (UID: "4070ca0f-1230-4eed-992c-b3f6bce6a4bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:10:05.661296 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.660933 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 01 16:10:05.661296 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.660994 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" podUID="edcab30a-6e4e-4019-9fd7-96fe7b085711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 01 16:10:05.677170 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.676256 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" Nov 01 16:10:05.743080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.738451 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.743080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.738483 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.743080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.738497 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.743080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.738525 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.743080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.738538 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.743080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.738552 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.743080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.738565 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.743080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.738592 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.743080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.738605 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg29p\" (UniqueName: \"kubernetes.io/projected/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-kube-api-access-xg29p\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.743080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.738618 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.743080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.738631 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.743080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.738644 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.743080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.738669 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.743080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.738683 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.743080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.738697 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.743080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.738710 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.743080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.738724 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l55gc\" (UniqueName: \"kubernetes.io/projected/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459-kube-api-access-l55gc\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.743080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.738751 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4070ca0f-1230-4eed-992c-b3f6bce6a4bb-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.759267 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.759228 4363 generic.go:356] "Generic (PLEG): container finished" podID="3b704396-3411-40af-95fe-4b67a41ae69f" containerID="6d837305a83abbd54c37311b0b33ff057014a6ad06193106ea3fe7f41ed5e19f" exitCode=0 Nov 01 16:10:05.759395 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.759356 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" event={"ID":"3b704396-3411-40af-95fe-4b67a41ae69f","Type":"ContainerDied","Data":"6d837305a83abbd54c37311b0b33ff057014a6ad06193106ea3fe7f41ed5e19f"} Nov 01 16:10:05.761539 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.761514 4363 generic.go:356] "Generic (PLEG): container finished" podID="bfe59fad-a9c8-4073-81c6-11f4eda63086" containerID="c472bea28321ed20bfc49b0b7f82215ac7384e18f1ac96ffcc1fceb0eba005d3" exitCode=0 Nov 01 16:10:05.761628 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.761567 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" Nov 01 16:10:05.761628 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.761617 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" event={"ID":"bfe59fad-a9c8-4073-81c6-11f4eda63086","Type":"ContainerDied","Data":"c472bea28321ed20bfc49b0b7f82215ac7384e18f1ac96ffcc1fceb0eba005d3"} Nov 01 16:10:05.761701 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.761653 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc" event={"ID":"bfe59fad-a9c8-4073-81c6-11f4eda63086","Type":"ContainerDied","Data":"b37c6e1b05585e753da187a9a3d8f887e92fa32347df50eed90e6c315104a94c"} Nov 01 16:10:05.761701 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.761682 4363 scope.go:122] "RemoveContainer" containerID="c472bea28321ed20bfc49b0b7f82215ac7384e18f1ac96ffcc1fceb0eba005d3" Nov 01 16:10:05.768279 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.765099 4363 generic.go:356] "Generic (PLEG): container finished" podID="0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0" containerID="5caefde19962b8bdfbe8aa21d27864e87a211f1a46aec0201e9fed1c1d85b500" exitCode=0 Nov 01 16:10:05.768279 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.765237 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" event={"ID":"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0","Type":"ContainerDied","Data":"5caefde19962b8bdfbe8aa21d27864e87a211f1a46aec0201e9fed1c1d85b500"} Nov 01 16:10:05.783549 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.771149 4363 generic.go:356] "Generic (PLEG): container finished" podID="edcab30a-6e4e-4019-9fd7-96fe7b085711" containerID="15bfb16861b677eac440a82beb7cd7f2a9964e25b9351c73f69ca970eeb0ebed" exitCode=0 Nov 01 16:10:05.783549 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.771244 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" event={"ID":"edcab30a-6e4e-4019-9fd7-96fe7b085711","Type":"ContainerDied","Data":"15bfb16861b677eac440a82beb7cd7f2a9964e25b9351c73f69ca970eeb0ebed"} Nov 01 16:10:05.783549 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.773438 4363 generic.go:356] "Generic (PLEG): container finished" podID="4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459" containerID="5d4172fa160490c368e0f1fde63fb6de21ef76011542bd53966bb6c2abe80c2d" exitCode=0 Nov 01 16:10:05.783549 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.773517 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" event={"ID":"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459","Type":"ContainerDied","Data":"5d4172fa160490c368e0f1fde63fb6de21ef76011542bd53966bb6c2abe80c2d"} Nov 01 16:10:05.783549 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.773650 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" event={"ID":"4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459","Type":"ContainerDied","Data":"faf56527c1acb87097f900a62733ace0e1bd5a10766c38c2b16d936d04fb7b5b"} Nov 01 16:10:05.783549 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.773745 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk" Nov 01 16:10:05.792202 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.792182 4363 scope.go:122] "RemoveContainer" containerID="c472bea28321ed20bfc49b0b7f82215ac7384e18f1ac96ffcc1fceb0eba005d3" Nov 01 16:10:05.798534 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:10:05.795442 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c472bea28321ed20bfc49b0b7f82215ac7384e18f1ac96ffcc1fceb0eba005d3\": container with ID starting with c472bea28321ed20bfc49b0b7f82215ac7384e18f1ac96ffcc1fceb0eba005d3 not found: ID does not exist" containerID="c472bea28321ed20bfc49b0b7f82215ac7384e18f1ac96ffcc1fceb0eba005d3" Nov 01 16:10:05.798534 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.795478 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c472bea28321ed20bfc49b0b7f82215ac7384e18f1ac96ffcc1fceb0eba005d3"} err="failed to get container status \"c472bea28321ed20bfc49b0b7f82215ac7384e18f1ac96ffcc1fceb0eba005d3\": rpc error: code = NotFound desc = could not find container \"c472bea28321ed20bfc49b0b7f82215ac7384e18f1ac96ffcc1fceb0eba005d3\": container with ID starting with c472bea28321ed20bfc49b0b7f82215ac7384e18f1ac96ffcc1fceb0eba005d3 not found: ID does not exist" Nov 01 16:10:05.798534 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.795502 4363 scope.go:122] "RemoveContainer" containerID="5d4172fa160490c368e0f1fde63fb6de21ef76011542bd53966bb6c2abe80c2d" Nov 01 16:10:05.798534 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.797262 4363 generic.go:356] "Generic (PLEG): container finished" podID="4070ca0f-1230-4eed-992c-b3f6bce6a4bb" containerID="0f8a65fdecd9f9f6a94b9d39b2f3d7f99e0466c5a33d37a06bb6903f4f8bf4b3" exitCode=0 Nov 01 16:10:05.798534 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.797352 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" event={"ID":"4070ca0f-1230-4eed-992c-b3f6bce6a4bb","Type":"ContainerDied","Data":"0f8a65fdecd9f9f6a94b9d39b2f3d7f99e0466c5a33d37a06bb6903f4f8bf4b3"} Nov 01 16:10:05.798534 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.797390 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" event={"ID":"4070ca0f-1230-4eed-992c-b3f6bce6a4bb","Type":"ContainerDied","Data":"ec9a12e1d0eecb78a65553ff14afb9fc6ea6616c9e38eef12467da8500175c6c"} Nov 01 16:10:05.798534 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.797470 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k" Nov 01 16:10:05.822228 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.821317 4363 generic.go:356] "Generic (PLEG): container finished" podID="07c0b57d-ed0c-4366-9716-256714694541" containerID="af74ce07a3b12fa506a685bf3df5118f906d5acf180c7cfbf31991680d62a347" exitCode=0 Nov 01 16:10:05.822228 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.821468 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" event={"ID":"07c0b57d-ed0c-4366-9716-256714694541","Type":"ContainerDied","Data":"af74ce07a3b12fa506a685bf3df5118f906d5acf180c7cfbf31991680d62a347"} Nov 01 16:10:05.823878 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.823854 4363 generic.go:356] "Generic (PLEG): container finished" podID="408aa757-573f-4cd4-b81d-c26c9af252f2" containerID="becc68c69d5083a9e078509dcb7c90251f33191fb317ea61e3a59037fa7cee65" exitCode=0 Nov 01 16:10:05.824004 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.823954 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" event={"ID":"408aa757-573f-4cd4-b81d-c26c9af252f2","Type":"ContainerDied","Data":"becc68c69d5083a9e078509dcb7c90251f33191fb317ea61e3a59037fa7cee65"} Nov 01 16:10:05.825873 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.825692 4363 generic.go:356] "Generic (PLEG): container finished" podID="74344814-e7cb-4b6f-b295-bb441f192023" containerID="00a54faa1bcc3176a4d7bb7c40007969ddd95f826cee12a7c991e871bfdf9a2e" exitCode=0 Nov 01 16:10:05.825873 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.825773 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" event={"ID":"74344814-e7cb-4b6f-b295-bb441f192023","Type":"ContainerDied","Data":"00a54faa1bcc3176a4d7bb7c40007969ddd95f826cee12a7c991e871bfdf9a2e"} Nov 01 16:10:05.840635 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.840603 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfe59fad-a9c8-4073-81c6-11f4eda63086-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe59fad-a9c8-4073-81c6-11f4eda63086-credential-socket\") pod \"bfe59fad-a9c8-4073-81c6-11f4eda63086\" (UID: \"bfe59fad-a9c8-4073-81c6-11f4eda63086\") " Nov 01 16:10:05.840912 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.840733 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfe59fad-a9c8-4073-81c6-11f4eda63086-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfe59fad-a9c8-4073-81c6-11f4eda63086-istio-envoy\") pod \"bfe59fad-a9c8-4073-81c6-11f4eda63086\" (UID: \"bfe59fad-a9c8-4073-81c6-11f4eda63086\") " Nov 01 16:10:05.840912 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.840777 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfe59fad-a9c8-4073-81c6-11f4eda63086-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfe59fad-a9c8-4073-81c6-11f4eda63086-istio-data\") pod \"bfe59fad-a9c8-4073-81c6-11f4eda63086\" (UID: \"bfe59fad-a9c8-4073-81c6-11f4eda63086\") " Nov 01 16:10:05.840912 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.840844 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfe59fad-a9c8-4073-81c6-11f4eda63086-kube-api-access-m8695\" (UniqueName: \"kubernetes.io/projected/bfe59fad-a9c8-4073-81c6-11f4eda63086-kube-api-access-m8695\") pod \"bfe59fad-a9c8-4073-81c6-11f4eda63086\" (UID: \"bfe59fad-a9c8-4073-81c6-11f4eda63086\") " Nov 01 16:10:05.841113 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.840972 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfe59fad-a9c8-4073-81c6-11f4eda63086-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfe59fad-a9c8-4073-81c6-11f4eda63086-workload-certs\") pod \"bfe59fad-a9c8-4073-81c6-11f4eda63086\" (UID: \"bfe59fad-a9c8-4073-81c6-11f4eda63086\") " Nov 01 16:10:05.841234 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.841216 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfe59fad-a9c8-4073-81c6-11f4eda63086-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfe59fad-a9c8-4073-81c6-11f4eda63086-istiod-ca-cert\") pod \"bfe59fad-a9c8-4073-81c6-11f4eda63086\" (UID: \"bfe59fad-a9c8-4073-81c6-11f4eda63086\") " Nov 01 16:10:05.841313 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.841296 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bfe59fad-a9c8-4073-81c6-11f4eda63086-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfe59fad-a9c8-4073-81c6-11f4eda63086-istio-podinfo\") pod \"bfe59fad-a9c8-4073-81c6-11f4eda63086\" (UID: \"bfe59fad-a9c8-4073-81c6-11f4eda63086\") " Nov 01 16:10:05.841437 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.841406 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfe59fad-a9c8-4073-81c6-11f4eda63086-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe59fad-a9c8-4073-81c6-11f4eda63086-workload-socket\") pod \"bfe59fad-a9c8-4073-81c6-11f4eda63086\" (UID: \"bfe59fad-a9c8-4073-81c6-11f4eda63086\") " Nov 01 16:10:05.841554 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.841521 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfe59fad-a9c8-4073-81c6-11f4eda63086-istio-token\" (UniqueName: \"kubernetes.io/projected/bfe59fad-a9c8-4073-81c6-11f4eda63086-istio-token\") pod \"bfe59fad-a9c8-4073-81c6-11f4eda63086\" (UID: \"bfe59fad-a9c8-4073-81c6-11f4eda63086\") " Nov 01 16:10:05.888288 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.844380 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfe59fad-a9c8-4073-81c6-11f4eda63086-istiod-ca-cert" pod "bfe59fad-a9c8-4073-81c6-11f4eda63086" (UID: "bfe59fad-a9c8-4073-81c6-11f4eda63086"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:10:05.888288 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.844647 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfe59fad-a9c8-4073-81c6-11f4eda63086-workload-certs" pod "bfe59fad-a9c8-4073-81c6-11f4eda63086" (UID: "bfe59fad-a9c8-4073-81c6-11f4eda63086"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:05.888288 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.845987 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfe59fad-a9c8-4073-81c6-11f4eda63086-credential-socket" pod "bfe59fad-a9c8-4073-81c6-11f4eda63086" (UID: "bfe59fad-a9c8-4073-81c6-11f4eda63086"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:05.888288 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.846064 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfe59fad-a9c8-4073-81c6-11f4eda63086-istio-data" pod "bfe59fad-a9c8-4073-81c6-11f4eda63086" (UID: "bfe59fad-a9c8-4073-81c6-11f4eda63086"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:05.888288 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.846256 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfe59fad-a9c8-4073-81c6-11f4eda63086-workload-socket" pod "bfe59fad-a9c8-4073-81c6-11f4eda63086" (UID: "bfe59fad-a9c8-4073-81c6-11f4eda63086"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:05.888288 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.853066 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfe59fad-a9c8-4073-81c6-11f4eda63086-istio-token" pod "bfe59fad-a9c8-4073-81c6-11f4eda63086" (UID: "bfe59fad-a9c8-4073-81c6-11f4eda63086"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:05.888288 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.872066 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfe59fad-a9c8-4073-81c6-11f4eda63086-kube-api-access-m8695" pod "bfe59fad-a9c8-4073-81c6-11f4eda63086" (UID: "bfe59fad-a9c8-4073-81c6-11f4eda63086"). InnerVolumeSpecName "kube-api-access-m8695". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:05.888288 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.888208 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfe59fad-a9c8-4073-81c6-11f4eda63086-istio-podinfo" pod "bfe59fad-a9c8-4073-81c6-11f4eda63086" (UID: "bfe59fad-a9c8-4073-81c6-11f4eda63086"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:10:05.896262 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.895221 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfe59fad-a9c8-4073-81c6-11f4eda63086-istio-envoy" pod "bfe59fad-a9c8-4073-81c6-11f4eda63086" (UID: "bfe59fad-a9c8-4073-81c6-11f4eda63086"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:05.913979 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.913400 4363 scope.go:122] "RemoveContainer" containerID="5d4172fa160490c368e0f1fde63fb6de21ef76011542bd53966bb6c2abe80c2d" Nov 01 16:10:05.915286 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:10:05.914825 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d4172fa160490c368e0f1fde63fb6de21ef76011542bd53966bb6c2abe80c2d\": container with ID starting with 5d4172fa160490c368e0f1fde63fb6de21ef76011542bd53966bb6c2abe80c2d not found: ID does not exist" containerID="5d4172fa160490c368e0f1fde63fb6de21ef76011542bd53966bb6c2abe80c2d" Nov 01 16:10:05.915286 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.914862 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d4172fa160490c368e0f1fde63fb6de21ef76011542bd53966bb6c2abe80c2d"} err="failed to get container status \"5d4172fa160490c368e0f1fde63fb6de21ef76011542bd53966bb6c2abe80c2d\": rpc error: code = NotFound desc = could not find container \"5d4172fa160490c368e0f1fde63fb6de21ef76011542bd53966bb6c2abe80c2d\": container with ID starting with 5d4172fa160490c368e0f1fde63fb6de21ef76011542bd53966bb6c2abe80c2d not found: ID does not exist" Nov 01 16:10:05.915286 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.914886 4363 scope.go:122] "RemoveContainer" containerID="0f8a65fdecd9f9f6a94b9d39b2f3d7f99e0466c5a33d37a06bb6903f4f8bf4b3" Nov 01 16:10:05.917620 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.917596 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" Nov 01 16:10:05.923801 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.923763 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" Nov 01 16:10:05.934167 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.934131 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" Nov 01 16:10:05.936358 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.936334 4363 scope.go:122] "RemoveContainer" containerID="0f8a65fdecd9f9f6a94b9d39b2f3d7f99e0466c5a33d37a06bb6903f4f8bf4b3" Nov 01 16:10:05.936686 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:10:05.936657 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f8a65fdecd9f9f6a94b9d39b2f3d7f99e0466c5a33d37a06bb6903f4f8bf4b3\": container with ID starting with 0f8a65fdecd9f9f6a94b9d39b2f3d7f99e0466c5a33d37a06bb6903f4f8bf4b3 not found: ID does not exist" containerID="0f8a65fdecd9f9f6a94b9d39b2f3d7f99e0466c5a33d37a06bb6903f4f8bf4b3" Nov 01 16:10:05.936802 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.936698 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f8a65fdecd9f9f6a94b9d39b2f3d7f99e0466c5a33d37a06bb6903f4f8bf4b3"} err="failed to get container status \"0f8a65fdecd9f9f6a94b9d39b2f3d7f99e0466c5a33d37a06bb6903f4f8bf4b3\": rpc error: code = NotFound desc = could not find container \"0f8a65fdecd9f9f6a94b9d39b2f3d7f99e0466c5a33d37a06bb6903f4f8bf4b3\": container with ID starting with 0f8a65fdecd9f9f6a94b9d39b2f3d7f99e0466c5a33d37a06bb6903f4f8bf4b3 not found: ID does not exist" Nov 01 16:10:05.939214 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.939192 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k"] Nov 01 16:10:05.944404 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.944377 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfe59fad-a9c8-4073-81c6-11f4eda63086-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.944494 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.944409 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfe59fad-a9c8-4073-81c6-11f4eda63086-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.944494 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.944424 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfe59fad-a9c8-4073-81c6-11f4eda63086-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.944494 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.944439 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe59fad-a9c8-4073-81c6-11f4eda63086-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.944494 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.944453 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfe59fad-a9c8-4073-81c6-11f4eda63086-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.944494 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.944467 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe59fad-a9c8-4073-81c6-11f4eda63086-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.944494 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.944488 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfe59fad-a9c8-4073-81c6-11f4eda63086-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.944716 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.944503 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfe59fad-a9c8-4073-81c6-11f4eda63086-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.944716 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.944519 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8695\" (UniqueName: \"kubernetes.io/projected/bfe59fad-a9c8-4073-81c6-11f4eda63086-kube-api-access-m8695\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.946154 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.946106 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-dq55k"] Nov 01 16:10:05.959044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.958394 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" Nov 01 16:10:05.964421 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.964374 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk"] Nov 01 16:10:05.971515 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.970627 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" Nov 01 16:10:05.972734 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:05.972698 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw8vk"] Nov 01 16:10:06.045552 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.045430 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-istio-podinfo\") pod \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\" (UID: \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\") " Nov 01 16:10:06.045552 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.045485 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/408aa757-573f-4cd4-b81d-c26c9af252f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408aa757-573f-4cd4-b81d-c26c9af252f2-istiod-ca-cert\") pod \"408aa757-573f-4cd4-b81d-c26c9af252f2\" (UID: \"408aa757-573f-4cd4-b81d-c26c9af252f2\") " Nov 01 16:10:06.045552 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.045538 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b704396-3411-40af-95fe-4b67a41ae69f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b704396-3411-40af-95fe-4b67a41ae69f-istio-data\") pod \"3b704396-3411-40af-95fe-4b67a41ae69f\" (UID: \"3b704396-3411-40af-95fe-4b67a41ae69f\") " Nov 01 16:10:06.045844 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.045583 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408aa757-573f-4cd4-b81d-c26c9af252f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408aa757-573f-4cd4-b81d-c26c9af252f2-credential-socket\") pod \"408aa757-573f-4cd4-b81d-c26c9af252f2\" (UID: \"408aa757-573f-4cd4-b81d-c26c9af252f2\") " Nov 01 16:10:06.045844 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.045613 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b704396-3411-40af-95fe-4b67a41ae69f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b704396-3411-40af-95fe-4b67a41ae69f-credential-socket\") pod \"3b704396-3411-40af-95fe-4b67a41ae69f\" (UID: \"3b704396-3411-40af-95fe-4b67a41ae69f\") " Nov 01 16:10:06.045844 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.045687 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408aa757-573f-4cd4-b81d-c26c9af252f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408aa757-573f-4cd4-b81d-c26c9af252f2-istio-envoy\") pod \"408aa757-573f-4cd4-b81d-c26c9af252f2\" (UID: \"408aa757-573f-4cd4-b81d-c26c9af252f2\") " Nov 01 16:10:06.045844 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.045711 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/408aa757-573f-4cd4-b81d-c26c9af252f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408aa757-573f-4cd4-b81d-c26c9af252f2-istio-podinfo\") pod \"408aa757-573f-4cd4-b81d-c26c9af252f2\" (UID: \"408aa757-573f-4cd4-b81d-c26c9af252f2\") " Nov 01 16:10:06.046100 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.045854 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408aa757-573f-4cd4-b81d-c26c9af252f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408aa757-573f-4cd4-b81d-c26c9af252f2-workload-certs\") pod \"408aa757-573f-4cd4-b81d-c26c9af252f2\" (UID: \"408aa757-573f-4cd4-b81d-c26c9af252f2\") " Nov 01 16:10:06.046100 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.045906 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-credential-socket\") pod \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\" (UID: \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\") " Nov 01 16:10:06.046100 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.045911 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408aa757-573f-4cd4-b81d-c26c9af252f2-credential-socket" pod "408aa757-573f-4cd4-b81d-c26c9af252f2" (UID: "408aa757-573f-4cd4-b81d-c26c9af252f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:06.046100 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.045933 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b704396-3411-40af-95fe-4b67a41ae69f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b704396-3411-40af-95fe-4b67a41ae69f-istio-envoy\") pod \"3b704396-3411-40af-95fe-4b67a41ae69f\" (UID: \"3b704396-3411-40af-95fe-4b67a41ae69f\") " Nov 01 16:10:06.046100 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.045983 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-istio-data\") pod \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\" (UID: \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\") " Nov 01 16:10:06.046100 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.046006 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/408aa757-573f-4cd4-b81d-c26c9af252f2-istio-token\" (UniqueName: \"kubernetes.io/projected/408aa757-573f-4cd4-b81d-c26c9af252f2-istio-token\") pod \"408aa757-573f-4cd4-b81d-c26c9af252f2\" (UID: \"408aa757-573f-4cd4-b81d-c26c9af252f2\") " Nov 01 16:10:06.046100 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.046089 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07c0b57d-ed0c-4366-9716-256714694541-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07c0b57d-ed0c-4366-9716-256714694541-credential-socket\") pod \"07c0b57d-ed0c-4366-9716-256714694541\" (UID: \"07c0b57d-ed0c-4366-9716-256714694541\") " Nov 01 16:10:06.046471 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.046119 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07c0b57d-ed0c-4366-9716-256714694541-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07c0b57d-ed0c-4366-9716-256714694541-istio-envoy\") pod \"07c0b57d-ed0c-4366-9716-256714694541\" (UID: \"07c0b57d-ed0c-4366-9716-256714694541\") " Nov 01 16:10:06.046471 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.046146 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07c0b57d-ed0c-4366-9716-256714694541-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07c0b57d-ed0c-4366-9716-256714694541-workload-certs\") pod \"07c0b57d-ed0c-4366-9716-256714694541\" (UID: \"07c0b57d-ed0c-4366-9716-256714694541\") " Nov 01 16:10:06.046471 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.046171 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74344814-e7cb-4b6f-b295-bb441f192023-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74344814-e7cb-4b6f-b295-bb441f192023-istio-envoy\") pod \"74344814-e7cb-4b6f-b295-bb441f192023\" (UID: \"74344814-e7cb-4b6f-b295-bb441f192023\") " Nov 01 16:10:06.046471 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.046240 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07c0b57d-ed0c-4366-9716-256714694541-kube-api-access-rp5lm\" (UniqueName: \"kubernetes.io/projected/07c0b57d-ed0c-4366-9716-256714694541-kube-api-access-rp5lm\") pod \"07c0b57d-ed0c-4366-9716-256714694541\" (UID: \"07c0b57d-ed0c-4366-9716-256714694541\") " Nov 01 16:10:06.046471 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.046268 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07c0b57d-ed0c-4366-9716-256714694541-istio-token\" (UniqueName: \"kubernetes.io/projected/07c0b57d-ed0c-4366-9716-256714694541-istio-token\") pod \"07c0b57d-ed0c-4366-9716-256714694541\" (UID: \"07c0b57d-ed0c-4366-9716-256714694541\") " Nov 01 16:10:06.046471 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.046289 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/408aa757-573f-4cd4-b81d-c26c9af252f2-istiod-ca-cert" pod "408aa757-573f-4cd4-b81d-c26c9af252f2" (UID: "408aa757-573f-4cd4-b81d-c26c9af252f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:10:06.046471 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.046314 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74344814-e7cb-4b6f-b295-bb441f192023-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74344814-e7cb-4b6f-b295-bb441f192023-istio-data\") pod \"74344814-e7cb-4b6f-b295-bb441f192023\" (UID: \"74344814-e7cb-4b6f-b295-bb441f192023\") " Nov 01 16:10:06.046471 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.046350 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b704396-3411-40af-95fe-4b67a41ae69f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b704396-3411-40af-95fe-4b67a41ae69f-workload-socket\") pod \"3b704396-3411-40af-95fe-4b67a41ae69f\" (UID: \"3b704396-3411-40af-95fe-4b67a41ae69f\") " Nov 01 16:10:06.046471 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.046378 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74344814-e7cb-4b6f-b295-bb441f192023-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74344814-e7cb-4b6f-b295-bb441f192023-istio-podinfo\") pod \"74344814-e7cb-4b6f-b295-bb441f192023\" (UID: \"74344814-e7cb-4b6f-b295-bb441f192023\") " Nov 01 16:10:06.046471 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.046405 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b704396-3411-40af-95fe-4b67a41ae69f-kube-api-access-mm6ln\" (UniqueName: \"kubernetes.io/projected/3b704396-3411-40af-95fe-4b67a41ae69f-kube-api-access-mm6ln\") pod \"3b704396-3411-40af-95fe-4b67a41ae69f\" (UID: \"3b704396-3411-40af-95fe-4b67a41ae69f\") " Nov 01 16:10:06.046471 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.046471 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408aa757-573f-4cd4-b81d-c26c9af252f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408aa757-573f-4cd4-b81d-c26c9af252f2-workload-socket\") pod \"408aa757-573f-4cd4-b81d-c26c9af252f2\" (UID: \"408aa757-573f-4cd4-b81d-c26c9af252f2\") " Nov 01 16:10:06.048301 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.046513 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-istio-envoy\") pod \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\" (UID: \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\") " Nov 01 16:10:06.048301 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.046531 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b704396-3411-40af-95fe-4b67a41ae69f-istio-data" pod "3b704396-3411-40af-95fe-4b67a41ae69f" (UID: "3b704396-3411-40af-95fe-4b67a41ae69f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:06.048301 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.046541 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b704396-3411-40af-95fe-4b67a41ae69f-istio-token\" (UniqueName: \"kubernetes.io/projected/3b704396-3411-40af-95fe-4b67a41ae69f-istio-token\") pod \"3b704396-3411-40af-95fe-4b67a41ae69f\" (UID: \"3b704396-3411-40af-95fe-4b67a41ae69f\") " Nov 01 16:10:06.048301 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.046618 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74344814-e7cb-4b6f-b295-bb441f192023-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74344814-e7cb-4b6f-b295-bb441f192023-istiod-ca-cert\") pod \"74344814-e7cb-4b6f-b295-bb441f192023\" (UID: \"74344814-e7cb-4b6f-b295-bb441f192023\") " Nov 01 16:10:06.048301 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.046655 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/408aa757-573f-4cd4-b81d-c26c9af252f2-kube-api-access-q8rzn\" (UniqueName: \"kubernetes.io/projected/408aa757-573f-4cd4-b81d-c26c9af252f2-kube-api-access-q8rzn\") pod \"408aa757-573f-4cd4-b81d-c26c9af252f2\" (UID: \"408aa757-573f-4cd4-b81d-c26c9af252f2\") " Nov 01 16:10:06.048301 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.046691 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07c0b57d-ed0c-4366-9716-256714694541-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07c0b57d-ed0c-4366-9716-256714694541-workload-socket\") pod \"07c0b57d-ed0c-4366-9716-256714694541\" (UID: \"07c0b57d-ed0c-4366-9716-256714694541\") " Nov 01 16:10:06.048301 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.046722 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b704396-3411-40af-95fe-4b67a41ae69f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b704396-3411-40af-95fe-4b67a41ae69f-workload-certs\") pod \"3b704396-3411-40af-95fe-4b67a41ae69f\" (UID: \"3b704396-3411-40af-95fe-4b67a41ae69f\") " Nov 01 16:10:06.048301 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.046762 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b704396-3411-40af-95fe-4b67a41ae69f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b704396-3411-40af-95fe-4b67a41ae69f-istio-podinfo\") pod \"3b704396-3411-40af-95fe-4b67a41ae69f\" (UID: \"3b704396-3411-40af-95fe-4b67a41ae69f\") " Nov 01 16:10:06.048301 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.046816 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74344814-e7cb-4b6f-b295-bb441f192023-istio-token\" (UniqueName: \"kubernetes.io/projected/74344814-e7cb-4b6f-b295-bb441f192023-istio-token\") pod \"74344814-e7cb-4b6f-b295-bb441f192023\" (UID: \"74344814-e7cb-4b6f-b295-bb441f192023\") " Nov 01 16:10:06.048301 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.046915 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-workload-certs\") pod \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\" (UID: \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\") " Nov 01 16:10:06.048301 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.046994 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74344814-e7cb-4b6f-b295-bb441f192023-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74344814-e7cb-4b6f-b295-bb441f192023-workload-socket\") pod \"74344814-e7cb-4b6f-b295-bb441f192023\" (UID: \"74344814-e7cb-4b6f-b295-bb441f192023\") " Nov 01 16:10:06.048301 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.047055 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-kube-api-access-t8qbv\" (UniqueName: \"kubernetes.io/projected/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-kube-api-access-t8qbv\") pod \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\" (UID: \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\") " Nov 01 16:10:06.048301 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.047113 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-istio-token\" (UniqueName: \"kubernetes.io/projected/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-istio-token\") pod \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\" (UID: \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\") " Nov 01 16:10:06.048301 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.047141 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07c0b57d-ed0c-4366-9716-256714694541-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07c0b57d-ed0c-4366-9716-256714694541-istio-podinfo\") pod \"07c0b57d-ed0c-4366-9716-256714694541\" (UID: \"07c0b57d-ed0c-4366-9716-256714694541\") " Nov 01 16:10:06.048301 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.047265 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07c0b57d-ed0c-4366-9716-256714694541-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07c0b57d-ed0c-4366-9716-256714694541-istio-data\") pod \"07c0b57d-ed0c-4366-9716-256714694541\" (UID: \"07c0b57d-ed0c-4366-9716-256714694541\") " Nov 01 16:10:06.048301 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.047300 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-istiod-ca-cert\") pod \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\" (UID: \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\") " Nov 01 16:10:06.048301 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.047417 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74344814-e7cb-4b6f-b295-bb441f192023-kube-api-access-xv45k\" (UniqueName: \"kubernetes.io/projected/74344814-e7cb-4b6f-b295-bb441f192023-kube-api-access-xv45k\") pod \"74344814-e7cb-4b6f-b295-bb441f192023\" (UID: \"74344814-e7cb-4b6f-b295-bb441f192023\") " Nov 01 16:10:06.048301 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.047455 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74344814-e7cb-4b6f-b295-bb441f192023-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74344814-e7cb-4b6f-b295-bb441f192023-credential-socket\") pod \"74344814-e7cb-4b6f-b295-bb441f192023\" (UID: \"74344814-e7cb-4b6f-b295-bb441f192023\") " Nov 01 16:10:06.048301 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.047495 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b704396-3411-40af-95fe-4b67a41ae69f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b704396-3411-40af-95fe-4b67a41ae69f-istiod-ca-cert\") pod \"3b704396-3411-40af-95fe-4b67a41ae69f\" (UID: \"3b704396-3411-40af-95fe-4b67a41ae69f\") " Nov 01 16:10:06.048301 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.047538 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74344814-e7cb-4b6f-b295-bb441f192023-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74344814-e7cb-4b6f-b295-bb441f192023-workload-certs\") pod \"74344814-e7cb-4b6f-b295-bb441f192023\" (UID: \"74344814-e7cb-4b6f-b295-bb441f192023\") " Nov 01 16:10:06.048301 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.047716 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408aa757-573f-4cd4-b81d-c26c9af252f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408aa757-573f-4cd4-b81d-c26c9af252f2-istio-data\") pod \"408aa757-573f-4cd4-b81d-c26c9af252f2\" (UID: \"408aa757-573f-4cd4-b81d-c26c9af252f2\") " Nov 01 16:10:06.048301 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.047782 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07c0b57d-ed0c-4366-9716-256714694541-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07c0b57d-ed0c-4366-9716-256714694541-istiod-ca-cert\") pod \"07c0b57d-ed0c-4366-9716-256714694541\" (UID: \"07c0b57d-ed0c-4366-9716-256714694541\") " Nov 01 16:10:06.048301 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.047824 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-workload-socket\") pod \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\" (UID: \"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0\") " Nov 01 16:10:06.048301 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.047939 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-credential-socket" pod "0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0" (UID: "0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:06.048301 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.048158 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" Nov 01 16:10:06.050934 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.048458 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-workload-socket" pod "0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0" (UID: "0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:06.050934 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.048719 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b704396-3411-40af-95fe-4b67a41ae69f-workload-certs" pod "3b704396-3411-40af-95fe-4b67a41ae69f" (UID: "3b704396-3411-40af-95fe-4b67a41ae69f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:06.050934 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.048880 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b704396-3411-40af-95fe-4b67a41ae69f-credential-socket" pod "3b704396-3411-40af-95fe-4b67a41ae69f" (UID: "3b704396-3411-40af-95fe-4b67a41ae69f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:06.050934 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.049523 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408aa757-573f-4cd4-b81d-c26c9af252f2-workload-certs" pod "408aa757-573f-4cd4-b81d-c26c9af252f2" (UID: "408aa757-573f-4cd4-b81d-c26c9af252f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:06.050934 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.049661 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07c0b57d-ed0c-4366-9716-256714694541-workload-certs" pod "07c0b57d-ed0c-4366-9716-256714694541" (UID: "07c0b57d-ed0c-4366-9716-256714694541"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:06.050934 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.049707 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-istio-data" pod "0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0" (UID: "0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:06.050934 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.049744 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b704396-3411-40af-95fe-4b67a41ae69f-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.050934 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.049769 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.050934 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.049786 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408aa757-573f-4cd4-b81d-c26c9af252f2-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.050934 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.049803 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b704396-3411-40af-95fe-4b67a41ae69f-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.050934 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.049818 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408aa757-573f-4cd4-b81d-c26c9af252f2-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.050934 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.049832 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b704396-3411-40af-95fe-4b67a41ae69f-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.050934 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.049848 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.051689 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.051088 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b704396-3411-40af-95fe-4b67a41ae69f-workload-socket" pod "3b704396-3411-40af-95fe-4b67a41ae69f" (UID: "3b704396-3411-40af-95fe-4b67a41ae69f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:06.051689 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.051146 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74344814-e7cb-4b6f-b295-bb441f192023-istio-data" pod "74344814-e7cb-4b6f-b295-bb441f192023" (UID: "74344814-e7cb-4b6f-b295-bb441f192023"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:06.051689 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.051369 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408aa757-573f-4cd4-b81d-c26c9af252f2-workload-socket" pod "408aa757-573f-4cd4-b81d-c26c9af252f2" (UID: "408aa757-573f-4cd4-b81d-c26c9af252f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:06.052966 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.052777 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07c0b57d-ed0c-4366-9716-256714694541-credential-socket" pod "07c0b57d-ed0c-4366-9716-256714694541" (UID: "07c0b57d-ed0c-4366-9716-256714694541"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:06.056761 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.053319 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-istiod-ca-cert" pod "0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0" (UID: "0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:10:06.056761 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.053579 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-workload-certs" pod "0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0" (UID: "0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:06.056761 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.053791 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74344814-e7cb-4b6f-b295-bb441f192023-workload-socket" pod "74344814-e7cb-4b6f-b295-bb441f192023" (UID: "74344814-e7cb-4b6f-b295-bb441f192023"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:06.056761 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.054825 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74344814-e7cb-4b6f-b295-bb441f192023-credential-socket" pod "74344814-e7cb-4b6f-b295-bb441f192023" (UID: "74344814-e7cb-4b6f-b295-bb441f192023"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:06.056761 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.055954 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74344814-e7cb-4b6f-b295-bb441f192023-workload-certs" pod "74344814-e7cb-4b6f-b295-bb441f192023" (UID: "74344814-e7cb-4b6f-b295-bb441f192023"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:06.056761 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.055999 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74344814-e7cb-4b6f-b295-bb441f192023-istiod-ca-cert" pod "74344814-e7cb-4b6f-b295-bb441f192023" (UID: "74344814-e7cb-4b6f-b295-bb441f192023"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:10:06.056761 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.056242 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408aa757-573f-4cd4-b81d-c26c9af252f2-istio-data" pod "408aa757-573f-4cd4-b81d-c26c9af252f2" (UID: "408aa757-573f-4cd4-b81d-c26c9af252f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:06.056761 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.056298 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b704396-3411-40af-95fe-4b67a41ae69f-istiod-ca-cert" pod "3b704396-3411-40af-95fe-4b67a41ae69f" (UID: "3b704396-3411-40af-95fe-4b67a41ae69f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:10:06.056761 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.056390 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07c0b57d-ed0c-4366-9716-256714694541-workload-socket" pod "07c0b57d-ed0c-4366-9716-256714694541" (UID: "07c0b57d-ed0c-4366-9716-256714694541"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:06.056761 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.056405 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-istio-podinfo" pod "0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0" (UID: "0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:10:06.056761 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.056623 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07c0b57d-ed0c-4366-9716-256714694541-istio-data" pod "07c0b57d-ed0c-4366-9716-256714694541" (UID: "07c0b57d-ed0c-4366-9716-256714694541"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:06.056761 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.056660 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07c0b57d-ed0c-4366-9716-256714694541-istiod-ca-cert" pod "07c0b57d-ed0c-4366-9716-256714694541" (UID: "07c0b57d-ed0c-4366-9716-256714694541"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:10:06.061628 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.061604 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07c0b57d-ed0c-4366-9716-256714694541-kube-api-access-rp5lm" pod "07c0b57d-ed0c-4366-9716-256714694541" (UID: "07c0b57d-ed0c-4366-9716-256714694541"). InnerVolumeSpecName "kube-api-access-rp5lm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:06.065227 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.065188 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-istio-token" pod "0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0" (UID: "0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:06.069239 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.060789 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74344814-e7cb-4b6f-b295-bb441f192023-kube-api-access-xv45k" pod "74344814-e7cb-4b6f-b295-bb441f192023" (UID: "74344814-e7cb-4b6f-b295-bb441f192023"). InnerVolumeSpecName "kube-api-access-xv45k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:06.072290 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.071742 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/408aa757-573f-4cd4-b81d-c26c9af252f2-kube-api-access-q8rzn" pod "408aa757-573f-4cd4-b81d-c26c9af252f2" (UID: "408aa757-573f-4cd4-b81d-c26c9af252f2"). InnerVolumeSpecName "kube-api-access-q8rzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:06.072290 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.072256 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-kube-api-access-t8qbv" pod "0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0" (UID: "0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0"). InnerVolumeSpecName "kube-api-access-t8qbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:06.073599 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.073574 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07c0b57d-ed0c-4366-9716-256714694541-istio-podinfo" pod "07c0b57d-ed0c-4366-9716-256714694541" (UID: "07c0b57d-ed0c-4366-9716-256714694541"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:10:06.075316 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.075281 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b704396-3411-40af-95fe-4b67a41ae69f-istio-envoy" pod "3b704396-3411-40af-95fe-4b67a41ae69f" (UID: "3b704396-3411-40af-95fe-4b67a41ae69f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:06.075316 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.075291 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74344814-e7cb-4b6f-b295-bb441f192023-istio-token" pod "74344814-e7cb-4b6f-b295-bb441f192023" (UID: "74344814-e7cb-4b6f-b295-bb441f192023"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:06.075472 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.075327 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b704396-3411-40af-95fe-4b67a41ae69f-kube-api-access-mm6ln" pod "3b704396-3411-40af-95fe-4b67a41ae69f" (UID: "3b704396-3411-40af-95fe-4b67a41ae69f"). InnerVolumeSpecName "kube-api-access-mm6ln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:06.075472 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.075361 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07c0b57d-ed0c-4366-9716-256714694541-istio-token" pod "07c0b57d-ed0c-4366-9716-256714694541" (UID: "07c0b57d-ed0c-4366-9716-256714694541"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:06.075472 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.075377 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b704396-3411-40af-95fe-4b67a41ae69f-istio-podinfo" pod "3b704396-3411-40af-95fe-4b67a41ae69f" (UID: "3b704396-3411-40af-95fe-4b67a41ae69f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:10:06.075472 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.075386 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b704396-3411-40af-95fe-4b67a41ae69f-istio-token" pod "3b704396-3411-40af-95fe-4b67a41ae69f" (UID: "3b704396-3411-40af-95fe-4b67a41ae69f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:06.075472 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.075406 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/408aa757-573f-4cd4-b81d-c26c9af252f2-istio-token" pod "408aa757-573f-4cd4-b81d-c26c9af252f2" (UID: "408aa757-573f-4cd4-b81d-c26c9af252f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:06.075472 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.075433 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74344814-e7cb-4b6f-b295-bb441f192023-istio-envoy" pod "74344814-e7cb-4b6f-b295-bb441f192023" (UID: "74344814-e7cb-4b6f-b295-bb441f192023"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:06.075472 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.075470 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74344814-e7cb-4b6f-b295-bb441f192023-istio-podinfo" pod "74344814-e7cb-4b6f-b295-bb441f192023" (UID: "74344814-e7cb-4b6f-b295-bb441f192023"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:10:06.075779 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.075560 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/408aa757-573f-4cd4-b81d-c26c9af252f2-istio-podinfo" pod "408aa757-573f-4cd4-b81d-c26c9af252f2" (UID: "408aa757-573f-4cd4-b81d-c26c9af252f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:10:06.075779 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.075700 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07c0b57d-ed0c-4366-9716-256714694541-istio-envoy" pod "07c0b57d-ed0c-4366-9716-256714694541" (UID: "07c0b57d-ed0c-4366-9716-256714694541"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:06.075779 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.075764 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408aa757-573f-4cd4-b81d-c26c9af252f2-istio-envoy" pod "408aa757-573f-4cd4-b81d-c26c9af252f2" (UID: "408aa757-573f-4cd4-b81d-c26c9af252f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:06.077143 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.077118 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-istio-envoy" pod "0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0" (UID: "0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:06.152656 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.152628 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edcab30a-6e4e-4019-9fd7-96fe7b085711-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edcab30a-6e4e-4019-9fd7-96fe7b085711-istiod-ca-cert\") pod \"edcab30a-6e4e-4019-9fd7-96fe7b085711\" (UID: \"edcab30a-6e4e-4019-9fd7-96fe7b085711\") " Nov 01 16:10:06.152988 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.152965 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edcab30a-6e4e-4019-9fd7-96fe7b085711-istio-token\" (UniqueName: \"kubernetes.io/projected/edcab30a-6e4e-4019-9fd7-96fe7b085711-istio-token\") pod \"edcab30a-6e4e-4019-9fd7-96fe7b085711\" (UID: \"edcab30a-6e4e-4019-9fd7-96fe7b085711\") " Nov 01 16:10:06.153362 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.153319 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edcab30a-6e4e-4019-9fd7-96fe7b085711-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edcab30a-6e4e-4019-9fd7-96fe7b085711-workload-certs\") pod \"edcab30a-6e4e-4019-9fd7-96fe7b085711\" (UID: \"edcab30a-6e4e-4019-9fd7-96fe7b085711\") " Nov 01 16:10:06.153585 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.153566 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edcab30a-6e4e-4019-9fd7-96fe7b085711-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edcab30a-6e4e-4019-9fd7-96fe7b085711-istio-envoy\") pod \"edcab30a-6e4e-4019-9fd7-96fe7b085711\" (UID: \"edcab30a-6e4e-4019-9fd7-96fe7b085711\") " Nov 01 16:10:06.153761 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.153727 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/edcab30a-6e4e-4019-9fd7-96fe7b085711-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edcab30a-6e4e-4019-9fd7-96fe7b085711-istio-podinfo\") pod \"edcab30a-6e4e-4019-9fd7-96fe7b085711\" (UID: \"edcab30a-6e4e-4019-9fd7-96fe7b085711\") " Nov 01 16:10:06.153929 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.153099 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edcab30a-6e4e-4019-9fd7-96fe7b085711-istiod-ca-cert" pod "edcab30a-6e4e-4019-9fd7-96fe7b085711" (UID: "edcab30a-6e4e-4019-9fd7-96fe7b085711"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:10:06.153929 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.153637 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edcab30a-6e4e-4019-9fd7-96fe7b085711-workload-certs" pod "edcab30a-6e4e-4019-9fd7-96fe7b085711" (UID: "edcab30a-6e4e-4019-9fd7-96fe7b085711"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:06.154123 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.154034 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edcab30a-6e4e-4019-9fd7-96fe7b085711-kube-api-access-cwrd6\" (UniqueName: \"kubernetes.io/projected/edcab30a-6e4e-4019-9fd7-96fe7b085711-kube-api-access-cwrd6\") pod \"edcab30a-6e4e-4019-9fd7-96fe7b085711\" (UID: \"edcab30a-6e4e-4019-9fd7-96fe7b085711\") " Nov 01 16:10:06.154179 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.154159 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edcab30a-6e4e-4019-9fd7-96fe7b085711-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcab30a-6e4e-4019-9fd7-96fe7b085711-credential-socket\") pod \"edcab30a-6e4e-4019-9fd7-96fe7b085711\" (UID: \"edcab30a-6e4e-4019-9fd7-96fe7b085711\") " Nov 01 16:10:06.154243 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.154215 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edcab30a-6e4e-4019-9fd7-96fe7b085711-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcab30a-6e4e-4019-9fd7-96fe7b085711-workload-socket\") pod \"edcab30a-6e4e-4019-9fd7-96fe7b085711\" (UID: \"edcab30a-6e4e-4019-9fd7-96fe7b085711\") " Nov 01 16:10:06.154307 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.154263 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edcab30a-6e4e-4019-9fd7-96fe7b085711-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edcab30a-6e4e-4019-9fd7-96fe7b085711-istio-data\") pod \"edcab30a-6e4e-4019-9fd7-96fe7b085711\" (UID: \"edcab30a-6e4e-4019-9fd7-96fe7b085711\") " Nov 01 16:10:06.156314 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.155398 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edcab30a-6e4e-4019-9fd7-96fe7b085711-workload-socket" pod "edcab30a-6e4e-4019-9fd7-96fe7b085711" (UID: "edcab30a-6e4e-4019-9fd7-96fe7b085711"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:06.156314 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.155462 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edcab30a-6e4e-4019-9fd7-96fe7b085711-credential-socket" pod "edcab30a-6e4e-4019-9fd7-96fe7b085711" (UID: "edcab30a-6e4e-4019-9fd7-96fe7b085711"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:06.156314 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.156179 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07c0b57d-ed0c-4366-9716-256714694541-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.156314 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.156205 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.156314 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.156221 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv45k\" (UniqueName: \"kubernetes.io/projected/74344814-e7cb-4b6f-b295-bb441f192023-kube-api-access-xv45k\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.156314 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.156236 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74344814-e7cb-4b6f-b295-bb441f192023-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.156314 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.156250 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b704396-3411-40af-95fe-4b67a41ae69f-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.156314 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.156265 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74344814-e7cb-4b6f-b295-bb441f192023-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.156314 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.156279 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408aa757-573f-4cd4-b81d-c26c9af252f2-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.156314 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.156274 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edcab30a-6e4e-4019-9fd7-96fe7b085711-istio-data" pod "edcab30a-6e4e-4019-9fd7-96fe7b085711" (UID: "edcab30a-6e4e-4019-9fd7-96fe7b085711"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:06.156314 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.156293 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07c0b57d-ed0c-4366-9716-256714694541-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.156314 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.156308 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.156314 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.156322 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edcab30a-6e4e-4019-9fd7-96fe7b085711-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.157043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.156336 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408aa757-573f-4cd4-b81d-c26c9af252f2-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.157043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.156350 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408aa757-573f-4cd4-b81d-c26c9af252f2-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.157043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.156364 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408aa757-573f-4cd4-b81d-c26c9af252f2-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.157043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.156379 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b704396-3411-40af-95fe-4b67a41ae69f-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.157043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.156393 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.157043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.156408 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408aa757-573f-4cd4-b81d-c26c9af252f2-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.157043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.156422 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07c0b57d-ed0c-4366-9716-256714694541-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.157043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.156436 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07c0b57d-ed0c-4366-9716-256714694541-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.157043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.156449 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07c0b57d-ed0c-4366-9716-256714694541-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.157043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.156464 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74344814-e7cb-4b6f-b295-bb441f192023-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.157043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.156487 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp5lm\" (UniqueName: \"kubernetes.io/projected/07c0b57d-ed0c-4366-9716-256714694541-kube-api-access-rp5lm\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.157043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.156502 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07c0b57d-ed0c-4366-9716-256714694541-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.157043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.156516 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74344814-e7cb-4b6f-b295-bb441f192023-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.157043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.156530 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b704396-3411-40af-95fe-4b67a41ae69f-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.157043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.156543 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74344814-e7cb-4b6f-b295-bb441f192023-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.157043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.156557 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm6ln\" (UniqueName: \"kubernetes.io/projected/3b704396-3411-40af-95fe-4b67a41ae69f-kube-api-access-mm6ln\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.157043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.156587 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408aa757-573f-4cd4-b81d-c26c9af252f2-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.157043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.156602 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.157043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.156615 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b704396-3411-40af-95fe-4b67a41ae69f-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.157043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.156629 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74344814-e7cb-4b6f-b295-bb441f192023-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.157043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.156644 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8rzn\" (UniqueName: \"kubernetes.io/projected/408aa757-573f-4cd4-b81d-c26c9af252f2-kube-api-access-q8rzn\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.157043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.156658 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07c0b57d-ed0c-4366-9716-256714694541-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.157043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.156673 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b704396-3411-40af-95fe-4b67a41ae69f-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.157043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.156686 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74344814-e7cb-4b6f-b295-bb441f192023-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.157043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.156700 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.157043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.156715 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74344814-e7cb-4b6f-b295-bb441f192023-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.157043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.156729 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8qbv\" (UniqueName: \"kubernetes.io/projected/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-kube-api-access-t8qbv\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.157043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.156743 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edcab30a-6e4e-4019-9fd7-96fe7b085711-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.157043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.156757 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.157043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.156771 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07c0b57d-ed0c-4366-9716-256714694541-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.159158 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.158527 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edcab30a-6e4e-4019-9fd7-96fe7b085711-istio-envoy" pod "edcab30a-6e4e-4019-9fd7-96fe7b085711" (UID: "edcab30a-6e4e-4019-9fd7-96fe7b085711"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:06.163477 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.163448 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edcab30a-6e4e-4019-9fd7-96fe7b085711-kube-api-access-cwrd6" pod "edcab30a-6e4e-4019-9fd7-96fe7b085711" (UID: "edcab30a-6e4e-4019-9fd7-96fe7b085711"). InnerVolumeSpecName "kube-api-access-cwrd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:06.164281 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.164244 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edcab30a-6e4e-4019-9fd7-96fe7b085711-istio-token" pod "edcab30a-6e4e-4019-9fd7-96fe7b085711" (UID: "edcab30a-6e4e-4019-9fd7-96fe7b085711"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:06.165406 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.165381 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/edcab30a-6e4e-4019-9fd7-96fe7b085711-istio-podinfo" pod "edcab30a-6e4e-4019-9fd7-96fe7b085711" (UID: "edcab30a-6e4e-4019-9fd7-96fe7b085711"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:10:06.211096 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.210130 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc"] Nov 01 16:10:06.215222 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.215195 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-xhbvc"] Nov 01 16:10:06.257940 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.257902 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwrd6\" (UniqueName: \"kubernetes.io/projected/edcab30a-6e4e-4019-9fd7-96fe7b085711-kube-api-access-cwrd6\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.257940 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.257940 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcab30a-6e4e-4019-9fd7-96fe7b085711-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.258159 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.257960 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcab30a-6e4e-4019-9fd7-96fe7b085711-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.258159 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.257970 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edcab30a-6e4e-4019-9fd7-96fe7b085711-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.258159 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.257979 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edcab30a-6e4e-4019-9fd7-96fe7b085711-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.258159 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.257988 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edcab30a-6e4e-4019-9fd7-96fe7b085711-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.258159 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.258001 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edcab30a-6e4e-4019-9fd7-96fe7b085711-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.611547 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.611089 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:10:06.611547 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.611153 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" podUID="3b704396-3411-40af-95fe-4b67a41ae69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:10:06.811892 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.811854 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4070ca0f-1230-4eed-992c-b3f6bce6a4bb" path="/var/lib/kubelet/pods/4070ca0f-1230-4eed-992c-b3f6bce6a4bb/volumes" Nov 01 16:10:06.812493 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.812469 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459" path="/var/lib/kubelet/pods/4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459/volumes" Nov 01 16:10:06.812935 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.812920 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfe59fad-a9c8-4073-81c6-11f4eda63086" path="/var/lib/kubelet/pods/bfe59fad-a9c8-4073-81c6-11f4eda63086/volumes" Nov 01 16:10:06.839059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.839035 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" Nov 01 16:10:06.839162 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.839063 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws" event={"ID":"07c0b57d-ed0c-4366-9716-256714694541","Type":"ContainerDied","Data":"ae9c7a9b0d5754646b8b8a22d90b32535dc3b2d0fe287dfe48136a603c066038"} Nov 01 16:10:06.839162 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.839117 4363 scope.go:122] "RemoveContainer" containerID="af74ce07a3b12fa506a685bf3df5118f906d5acf180c7cfbf31991680d62a347" Nov 01 16:10:06.840703 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.840685 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" Nov 01 16:10:06.840813 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.840684 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz" event={"ID":"408aa757-573f-4cd4-b81d-c26c9af252f2","Type":"ContainerDied","Data":"0e472d5787e85111b689389b62c19eba08f683f4a4dc969356616ead202f0437"} Nov 01 16:10:06.842543 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.842518 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" Nov 01 16:10:06.842863 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.842758 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8" event={"ID":"74344814-e7cb-4b6f-b295-bb441f192023","Type":"ContainerDied","Data":"cc4bf8c178d37d3485925fa878326d665140db7fa55e153dbae7f93761b06231"} Nov 01 16:10:06.844533 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.844512 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" Nov 01 16:10:06.844621 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.844542 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8" event={"ID":"3b704396-3411-40af-95fe-4b67a41ae69f","Type":"ContainerDied","Data":"a6c54014bf0b4bc8ddcc1664fc98471ff051093f426b675fe65d7d81dd51655b"} Nov 01 16:10:06.847455 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.847417 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" event={"ID":"0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0","Type":"ContainerDied","Data":"90c9a3dea0985cade70087ca7126c196f71d27988737162f1d17dc10f4652cc3"} Nov 01 16:10:06.847579 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.847479 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl" Nov 01 16:10:06.849223 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.848999 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" Nov 01 16:10:06.849223 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.848996 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79" event={"ID":"edcab30a-6e4e-4019-9fd7-96fe7b085711","Type":"ContainerDied","Data":"bee7486032ea1fa6bfaa1e4dc638577ac6e968081f97c17456978514d1989d0d"} Nov 01 16:10:06.854732 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.854694 4363 scope.go:122] "RemoveContainer" containerID="becc68c69d5083a9e078509dcb7c90251f33191fb317ea61e3a59037fa7cee65" Nov 01 16:10:06.865650 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.865613 4363 scope.go:122] "RemoveContainer" containerID="00a54faa1bcc3176a4d7bb7c40007969ddd95f826cee12a7c991e871bfdf9a2e" Nov 01 16:10:06.866145 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.866121 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8"] Nov 01 16:10:06.896338 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.891037 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-jqvc8"] Nov 01 16:10:06.911101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.904128 4363 scope.go:122] "RemoveContainer" containerID="6d837305a83abbd54c37311b0b33ff057014a6ad06193106ea3fe7f41ed5e19f" Nov 01 16:10:06.939202 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.939174 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8"] Nov 01 16:10:06.945790 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.945751 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-26tv8"] Nov 01 16:10:06.950444 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.950406 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws"] Nov 01 16:10:06.950528 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.950479 4363 scope.go:122] "RemoveContainer" containerID="5caefde19962b8bdfbe8aa21d27864e87a211f1a46aec0201e9fed1c1d85b500" Nov 01 16:10:06.956730 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.956709 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-l8xws"] Nov 01 16:10:06.962081 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.962062 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz"] Nov 01 16:10:06.963901 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.963883 4363 scope.go:122] "RemoveContainer" containerID="15bfb16861b677eac440a82beb7cd7f2a9964e25b9351c73f69ca970eeb0ebed" Nov 01 16:10:06.966324 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.966296 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-bjdzz"] Nov 01 16:10:06.973528 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:06.973505 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl"] Nov 01 16:10:07.011220 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.011180 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-8fqcl"] Nov 01 16:10:07.016815 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.016780 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79"] Nov 01 16:10:07.023173 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.023146 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-kcw79"] Nov 01 16:10:07.309857 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.309760 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-166778-77d4bcfc6f-88bhr"] Nov 01 16:10:07.311196 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.311127 4363 cpu_manager.go:414] "containerMap: removing container" podUID="07c0b57d-ed0c-4366-9716-256714694541" containerName="istio-proxy" Nov 01 16:10:07.311196 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.311192 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07c0b57d-ed0c-4366-9716-256714694541" containerName="istio-proxy" Nov 01 16:10:07.311445 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.311235 4363 cpu_manager.go:414] "containerMap: removing container" podUID="edcab30a-6e4e-4019-9fd7-96fe7b085711" containerName="istio-proxy" Nov 01 16:10:07.311445 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.311245 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edcab30a-6e4e-4019-9fd7-96fe7b085711" containerName="istio-proxy" Nov 01 16:10:07.311445 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.311272 4363 cpu_manager.go:414] "containerMap: removing container" podUID="4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459" containerName="istio-proxy" Nov 01 16:10:07.311445 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.311279 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459" containerName="istio-proxy" Nov 01 16:10:07.311445 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.311306 4363 cpu_manager.go:414] "containerMap: removing container" podUID="74344814-e7cb-4b6f-b295-bb441f192023" containerName="istio-proxy" Nov 01 16:10:07.311445 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.311313 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74344814-e7cb-4b6f-b295-bb441f192023" containerName="istio-proxy" Nov 01 16:10:07.311445 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.311326 4363 cpu_manager.go:414] "containerMap: removing container" podUID="4070ca0f-1230-4eed-992c-b3f6bce6a4bb" containerName="istio-proxy" Nov 01 16:10:07.311445 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.311333 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4070ca0f-1230-4eed-992c-b3f6bce6a4bb" containerName="istio-proxy" Nov 01 16:10:07.311445 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.311365 4363 cpu_manager.go:414] "containerMap: removing container" podUID="0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0" containerName="istio-proxy" Nov 01 16:10:07.311445 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.311379 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0" containerName="istio-proxy" Nov 01 16:10:07.311445 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.311392 4363 cpu_manager.go:414] "containerMap: removing container" podUID="bfe59fad-a9c8-4073-81c6-11f4eda63086" containerName="istio-proxy" Nov 01 16:10:07.311774 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.311582 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfe59fad-a9c8-4073-81c6-11f4eda63086" containerName="istio-proxy" Nov 01 16:10:07.311774 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.311603 4363 cpu_manager.go:414] "containerMap: removing container" podUID="3b704396-3411-40af-95fe-4b67a41ae69f" containerName="istio-proxy" Nov 01 16:10:07.311774 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.311607 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b704396-3411-40af-95fe-4b67a41ae69f" containerName="istio-proxy" Nov 01 16:10:07.311774 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.311616 4363 cpu_manager.go:414] "containerMap: removing container" podUID="408aa757-573f-4cd4-b81d-c26c9af252f2" containerName="istio-proxy" Nov 01 16:10:07.311774 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.311623 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="408aa757-573f-4cd4-b81d-c26c9af252f2" containerName="istio-proxy" Nov 01 16:10:07.311774 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.311761 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="74344814-e7cb-4b6f-b295-bb441f192023" containerName="istio-proxy" Nov 01 16:10:07.311774 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.311768 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="4070ca0f-1230-4eed-992c-b3f6bce6a4bb" containerName="istio-proxy" Nov 01 16:10:07.311774 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.311777 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0" containerName="istio-proxy" Nov 01 16:10:07.312004 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.311785 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfe59fad-a9c8-4073-81c6-11f4eda63086" containerName="istio-proxy" Nov 01 16:10:07.312004 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.311792 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b704396-3411-40af-95fe-4b67a41ae69f" containerName="istio-proxy" Nov 01 16:10:07.312004 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.311799 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="408aa757-573f-4cd4-b81d-c26c9af252f2" containerName="istio-proxy" Nov 01 16:10:07.312004 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.311805 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="07c0b57d-ed0c-4366-9716-256714694541" containerName="istio-proxy" Nov 01 16:10:07.312004 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.311813 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="edcab30a-6e4e-4019-9fd7-96fe7b085711" containerName="istio-proxy" Nov 01 16:10:07.312004 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.311819 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aaf0c7e-19c6-4da9-8f12-0cd7ed7dc459" containerName="istio-proxy" Nov 01 16:10:07.318097 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.318074 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-166778-77d4bcfc6f-88bhr" Nov 01 16:10:07.320867 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.320844 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-166778-77d4bcfc6f-88bhr"] Nov 01 16:10:07.325580 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.325559 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-166778-7d8fcb5659-jq7tt"] Nov 01 16:10:07.337329 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.337308 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-166778-7d8fcb5659-jq7tt" Nov 01 16:10:07.345652 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.345614 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-166778-7d8fcb5659-jq7tt"] Nov 01 16:10:07.345957 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.345930 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-166778\"" Nov 01 16:10:07.351761 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.351741 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-166778-dockercfg-6cdqf\"" Nov 01 16:10:07.367988 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.367628 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9"] Nov 01 16:10:07.413153 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.413122 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9" Nov 01 16:10:07.414561 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.414533 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9"] Nov 01 16:10:07.451569 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.451540 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-022f62-serving-cert\"" Nov 01 16:10:07.514933 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.514896 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hbq2\" (UniqueName: \"kubernetes.io/projected/5c1a1dba-28cb-489a-be20-1ae4bdc0ad85-kube-api-access-5hbq2\") pod \"payload-processing-e2e-api-a-166778-77d4bcfc6f-88bhr\" (UID: \"5c1a1dba-28cb-489a-be20-1ae4bdc0ad85\") " pod="openshift-ingress/payload-processing-e2e-api-a-166778-77d4bcfc6f-88bhr" Nov 01 16:10:07.515476 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.515439 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51609bfa-e393-49c1-a3c3-f93a6097183e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-166778-7d8fcb5659-jq7tt\" (UID: \"51609bfa-e393-49c1-a3c3-f93a6097183e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-166778-7d8fcb5659-jq7tt" Nov 01 16:10:07.516626 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.516585 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c1a1dba-28cb-489a-be20-1ae4bdc0ad85-plugins-config-volume\") pod \"payload-processing-e2e-api-a-166778-77d4bcfc6f-88bhr\" (UID: \"5c1a1dba-28cb-489a-be20-1ae4bdc0ad85\") " pod="openshift-ingress/payload-processing-e2e-api-a-166778-77d4bcfc6f-88bhr" Nov 01 16:10:07.516974 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.516954 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkmrh\" (UniqueName: \"kubernetes.io/projected/51609bfa-e393-49c1-a3c3-f93a6097183e-kube-api-access-hkmrh\") pod \"payload-pre-processing-e2e-api-a-166778-7d8fcb5659-jq7tt\" (UID: \"51609bfa-e393-49c1-a3c3-f93a6097183e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-166778-7d8fcb5659-jq7tt" Nov 01 16:10:07.627497 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.627461 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51609bfa-e393-49c1-a3c3-f93a6097183e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-166778-7d8fcb5659-jq7tt\" (UID: \"51609bfa-e393-49c1-a3c3-f93a6097183e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-166778-7d8fcb5659-jq7tt" Nov 01 16:10:07.628192 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.628166 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/375990c6-6bd9-4c31-9a12-74732ca46a47-metrics-certs\") pod \"maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9\" (UID: \"375990c6-6bd9-4c31-9a12-74732ca46a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9" Nov 01 16:10:07.628350 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.628333 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtf9f\" (UniqueName: \"kubernetes.io/projected/375990c6-6bd9-4c31-9a12-74732ca46a47-kube-api-access-dtf9f\") pod \"maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9\" (UID: \"375990c6-6bd9-4c31-9a12-74732ca46a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9" Nov 01 16:10:07.628460 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.628432 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51609bfa-e393-49c1-a3c3-f93a6097183e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-166778-7d8fcb5659-jq7tt\" (UID: \"51609bfa-e393-49c1-a3c3-f93a6097183e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-166778-7d8fcb5659-jq7tt" Nov 01 16:10:07.628580 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.628562 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c1a1dba-28cb-489a-be20-1ae4bdc0ad85-plugins-config-volume\") pod \"payload-processing-e2e-api-a-166778-77d4bcfc6f-88bhr\" (UID: \"5c1a1dba-28cb-489a-be20-1ae4bdc0ad85\") " pod="openshift-ingress/payload-processing-e2e-api-a-166778-77d4bcfc6f-88bhr" Nov 01 16:10:07.628838 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.628823 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/375990c6-6bd9-4c31-9a12-74732ca46a47-openshift-service-ca\") pod \"maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9\" (UID: \"375990c6-6bd9-4c31-9a12-74732ca46a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9" Nov 01 16:10:07.628969 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.628954 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkmrh\" (UniqueName: \"kubernetes.io/projected/51609bfa-e393-49c1-a3c3-f93a6097183e-kube-api-access-hkmrh\") pod \"payload-pre-processing-e2e-api-a-166778-7d8fcb5659-jq7tt\" (UID: \"51609bfa-e393-49c1-a3c3-f93a6097183e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-166778-7d8fcb5659-jq7tt" Nov 01 16:10:07.637369 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.637329 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/375990c6-6bd9-4c31-9a12-74732ca46a47-maas-api-tls\") pod \"maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9\" (UID: \"375990c6-6bd9-4c31-9a12-74732ca46a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9" Nov 01 16:10:07.637840 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.637799 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hbq2\" (UniqueName: \"kubernetes.io/projected/5c1a1dba-28cb-489a-be20-1ae4bdc0ad85-kube-api-access-5hbq2\") pod \"payload-processing-e2e-api-a-166778-77d4bcfc6f-88bhr\" (UID: \"5c1a1dba-28cb-489a-be20-1ae4bdc0ad85\") " pod="openshift-ingress/payload-processing-e2e-api-a-166778-77d4bcfc6f-88bhr" Nov 01 16:10:07.643593 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.643570 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c1a1dba-28cb-489a-be20-1ae4bdc0ad85-plugins-config-volume\") pod \"payload-processing-e2e-api-a-166778-77d4bcfc6f-88bhr\" (UID: \"5c1a1dba-28cb-489a-be20-1ae4bdc0ad85\") " pod="openshift-ingress/payload-processing-e2e-api-a-166778-77d4bcfc6f-88bhr" Nov 01 16:10:07.764041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.736150 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hbq2\" (UniqueName: \"kubernetes.io/projected/5c1a1dba-28cb-489a-be20-1ae4bdc0ad85-kube-api-access-5hbq2\") pod \"payload-processing-e2e-api-a-166778-77d4bcfc6f-88bhr\" (UID: \"5c1a1dba-28cb-489a-be20-1ae4bdc0ad85\") " pod="openshift-ingress/payload-processing-e2e-api-a-166778-77d4bcfc6f-88bhr" Nov 01 16:10:07.801047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.777758 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkmrh\" (UniqueName: \"kubernetes.io/projected/51609bfa-e393-49c1-a3c3-f93a6097183e-kube-api-access-hkmrh\") pod \"payload-pre-processing-e2e-api-a-166778-7d8fcb5659-jq7tt\" (UID: \"51609bfa-e393-49c1-a3c3-f93a6097183e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-166778-7d8fcb5659-jq7tt" Nov 01 16:10:07.801343 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.788088 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/375990c6-6bd9-4c31-9a12-74732ca46a47-maas-api-tls\") pod \"maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9\" (UID: \"375990c6-6bd9-4c31-9a12-74732ca46a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9" Nov 01 16:10:07.823149 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:10:07.788242 4363 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-022f62-serving-cert: secret "maas-api-e2e-api-a-022f62-serving-cert" not found Nov 01 16:10:07.823410 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.823378 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/375990c6-6bd9-4c31-9a12-74732ca46a47-metrics-certs\") pod \"maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9\" (UID: \"375990c6-6bd9-4c31-9a12-74732ca46a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9" Nov 01 16:10:07.823504 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.823437 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtf9f\" (UniqueName: \"kubernetes.io/projected/375990c6-6bd9-4c31-9a12-74732ca46a47-kube-api-access-dtf9f\") pod \"maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9\" (UID: \"375990c6-6bd9-4c31-9a12-74732ca46a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9" Nov 01 16:10:07.824564 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.823616 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/375990c6-6bd9-4c31-9a12-74732ca46a47-openshift-service-ca\") pod \"maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9\" (UID: \"375990c6-6bd9-4c31-9a12-74732ca46a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9" Nov 01 16:10:07.824687 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.824591 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/375990c6-6bd9-4c31-9a12-74732ca46a47-openshift-service-ca\") pod \"maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9\" (UID: \"375990c6-6bd9-4c31-9a12-74732ca46a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9" Nov 01 16:10:07.824866 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:10:07.824849 4363 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/375990c6-6bd9-4c31-9a12-74732ca46a47-maas-api-tls podName:375990c6-6bd9-4c31-9a12-74732ca46a47 nodeName:}" failed. No retries permitted until 2026-11-01 16:10:08.324821561 +0000 UTC m=+1341.656559202 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/375990c6-6bd9-4c31-9a12-74732ca46a47-maas-api-tls") pod "maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9" (UID: "375990c6-6bd9-4c31-9a12-74732ca46a47") : secret "maas-api-e2e-api-a-022f62-serving-cert" not found Nov 01 16:10:07.954976 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.954884 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-166778-77d4bcfc6f-88bhr" Nov 01 16:10:07.974857 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:07.974725 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-166778-7d8fcb5659-jq7tt" Nov 01 16:10:08.030891 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:08.030857 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtf9f\" (UniqueName: \"kubernetes.io/projected/375990c6-6bd9-4c31-9a12-74732ca46a47-kube-api-access-dtf9f\") pod \"maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9\" (UID: \"375990c6-6bd9-4c31-9a12-74732ca46a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9" Nov 01 16:10:08.031605 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:08.031575 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/375990c6-6bd9-4c31-9a12-74732ca46a47-metrics-certs\") pod \"maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9\" (UID: \"375990c6-6bd9-4c31-9a12-74732ca46a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9" Nov 01 16:10:08.435889 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:08.435858 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/375990c6-6bd9-4c31-9a12-74732ca46a47-maas-api-tls\") pod \"maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9\" (UID: \"375990c6-6bd9-4c31-9a12-74732ca46a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9" Nov 01 16:10:08.439573 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:08.439514 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/375990c6-6bd9-4c31-9a12-74732ca46a47-maas-api-tls\") pod \"maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9\" (UID: \"375990c6-6bd9-4c31-9a12-74732ca46a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9" Nov 01 16:10:08.639344 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:08.639309 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9" Nov 01 16:10:08.842290 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:08.842260 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07c0b57d-ed0c-4366-9716-256714694541" path="/var/lib/kubelet/pods/07c0b57d-ed0c-4366-9716-256714694541/volumes" Nov 01 16:10:08.843149 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:08.843127 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0" path="/var/lib/kubelet/pods/0b28b754-a4f9-44a1-8b8c-bcb7854ac1b0/volumes" Nov 01 16:10:08.843948 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:08.843929 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b704396-3411-40af-95fe-4b67a41ae69f" path="/var/lib/kubelet/pods/3b704396-3411-40af-95fe-4b67a41ae69f/volumes" Nov 01 16:10:08.845144 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:08.845126 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="408aa757-573f-4cd4-b81d-c26c9af252f2" path="/var/lib/kubelet/pods/408aa757-573f-4cd4-b81d-c26c9af252f2/volumes" Nov 01 16:10:08.845907 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:08.845889 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74344814-e7cb-4b6f-b295-bb441f192023" path="/var/lib/kubelet/pods/74344814-e7cb-4b6f-b295-bb441f192023/volumes" Nov 01 16:10:08.846817 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:08.846800 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edcab30a-6e4e-4019-9fd7-96fe7b085711" path="/var/lib/kubelet/pods/edcab30a-6e4e-4019-9fd7-96fe7b085711/volumes" Nov 01 16:10:08.847842 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:08.847822 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-166778-77d4bcfc6f-88bhr"] Nov 01 16:10:08.963044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:08.962987 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-166778-7d8fcb5659-jq7tt"] Nov 01 16:10:09.054496 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:09.054460 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-166778-77d4bcfc6f-88bhr" event={"ID":"5c1a1dba-28cb-489a-be20-1ae4bdc0ad85","Type":"ContainerStarted","Data":"88eb0febc5b8f8df7440402b2c47891b9da369f21a363a9f3a6441710b497d28"} Nov 01 16:10:09.054644 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:09.054519 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-166778-77d4bcfc6f-88bhr" event={"ID":"5c1a1dba-28cb-489a-be20-1ae4bdc0ad85","Type":"ContainerStarted","Data":"ba52d713bcc70feacc9ab9f7e9259c95a312ac91517a902770380a607964b938"} Nov 01 16:10:09.055148 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:09.055101 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-166778-77d4bcfc6f-88bhr" Nov 01 16:10:09.058526 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:09.058473 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-166778-7d8fcb5659-jq7tt" event={"ID":"51609bfa-e393-49c1-a3c3-f93a6097183e","Type":"ContainerStarted","Data":"30e31d6a46d9d8b8d94123005c74858dc1616d849ab3f3477662fb3967ec1d4d"} Nov 01 16:10:09.075178 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:09.075133 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-166778-77d4bcfc6f-88bhr" podStartSLOduration=2.075119791 podStartE2EDuration="2.075119791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:10:09.072754279 +0000 UTC m=+1342.404491930" watchObservedRunningTime="2026-11-01 16:10:09.075119791 +0000 UTC m=+1342.406857440" Nov 01 16:10:09.287508 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:09.275040 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9"] Nov 01 16:10:10.068381 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:10.067617 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-166778-7d8fcb5659-jq7tt" event={"ID":"51609bfa-e393-49c1-a3c3-f93a6097183e","Type":"ContainerStarted","Data":"f50d05c3db4ac9a934b0e526891577d4f671756187de2b0fab7f4c23acca7f1a"} Nov 01 16:10:10.068381 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:10.067787 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-166778-7d8fcb5659-jq7tt" Nov 01 16:10:10.069836 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:10.069810 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9" event={"ID":"375990c6-6bd9-4c31-9a12-74732ca46a47","Type":"ContainerStarted","Data":"10797d2c688855ad8b840577990403ba4b74814ae8ce61d85d349771ff2e2301"} Nov 01 16:10:10.069954 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:10.069858 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9" event={"ID":"375990c6-6bd9-4c31-9a12-74732ca46a47","Type":"ContainerStarted","Data":"35f7b1db7e9acc8dbd5bcc052c880a070010cfba068b37be4c87399a019fcefe"} Nov 01 16:10:10.070039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:10.069965 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9" Nov 01 16:10:10.118040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:10.117892 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-166778-7d8fcb5659-jq7tt" podStartSLOduration=3.117872481 podStartE2EDuration="3.117872481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:10:10.099468956 +0000 UTC m=+1343.431206606" watchObservedRunningTime="2026-11-01 16:10:10.117872481 +0000 UTC m=+1343.449610133" Nov 01 16:10:10.130110 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:10.130064 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9" podStartSLOduration=2.527456553 podStartE2EDuration="3.130049531s" totalImagesPullingTime="602.592978ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:10:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:10:10.129629579 +0000 UTC m=+1343.461367229" watchObservedRunningTime="2026-11-01 16:10:10.130049531 +0000 UTC m=+1343.461787180" Nov 01 16:10:12.352424 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:12.352389 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5c6b75-58cdb8f699-tzrc2"] Nov 01 16:10:12.372955 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:12.372053 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5c6b75-9f88c8787-vm79j"] Nov 01 16:10:16.107067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:16.106995 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9" Nov 01 16:10:16.733988 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:16.733954 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6963-0ae593"] Nov 01 16:10:16.740594 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:16.740562 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6963-0ae593" Nov 01 16:10:16.749270 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:16.745555 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6963-0ae593"] Nov 01 16:10:16.774214 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:16.774181 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hqwjk\"" Nov 01 16:10:16.911030 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:16.910980 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw9vx\" (UniqueName: \"kubernetes.io/projected/0d723602-2b68-4ab0-8c4b-10f9962180f7-kube-api-access-bw9vx\") pod \"test-curl-6963-0ae593\" (UID: \"0d723602-2b68-4ab0-8c4b-10f9962180f7\") " pod="opendatahub/test-curl-6963-0ae593" Nov 01 16:10:17.012866 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:17.012774 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw9vx\" (UniqueName: \"kubernetes.io/projected/0d723602-2b68-4ab0-8c4b-10f9962180f7-kube-api-access-bw9vx\") pod \"test-curl-6963-0ae593\" (UID: \"0d723602-2b68-4ab0-8c4b-10f9962180f7\") " pod="opendatahub/test-curl-6963-0ae593" Nov 01 16:10:17.021407 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:17.021376 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw9vx\" (UniqueName: \"kubernetes.io/projected/0d723602-2b68-4ab0-8c4b-10f9962180f7-kube-api-access-bw9vx\") pod \"test-curl-6963-0ae593\" (UID: \"0d723602-2b68-4ab0-8c4b-10f9962180f7\") " pod="opendatahub/test-curl-6963-0ae593" Nov 01 16:10:17.062541 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:17.062508 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6963-0ae593" Nov 01 16:10:17.353067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:17.352993 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-5c6b75-58cdb8f699-tzrc2" podUID="729e8251-2e65-4826-a535-a19b551e04c7" containerName="payload-processing" containerID="cri-o://307bbc000ccbf043f74a3fd2618d3fd25241b1b413b8ec0cb7562d6621d7689a" gracePeriod=25 Nov 01 16:10:17.367175 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:17.367148 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6963-0ae593"] Nov 01 16:10:17.373654 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:17.373608 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c6b75-9f88c8787-vm79j" podUID="df7c1364-8484-4cb7-9010-b7acbd09ccba" containerName="payload-pre-processing" containerID="cri-o://c8f18e85231ccdb24839f583a9dc8bba3207c70dc65e9e8c75594f9f4004a707" gracePeriod=25 Nov 01 16:10:17.670115 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:17.670080 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5c6b75-58cdb8f699-tzrc2" Nov 01 16:10:17.722191 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:17.721825 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c6b75-9f88c8787-vm79j" Nov 01 16:10:17.748280 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:17.748246 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df7c1364-8484-4cb7-9010-b7acbd09ccba-kube-api-access-lgn95\" (UniqueName: \"kubernetes.io/projected/df7c1364-8484-4cb7-9010-b7acbd09ccba-kube-api-access-lgn95\") pod \"df7c1364-8484-4cb7-9010-b7acbd09ccba\" (UID: \"df7c1364-8484-4cb7-9010-b7acbd09ccba\") " Nov 01 16:10:17.748464 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:17.748342 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df7c1364-8484-4cb7-9010-b7acbd09ccba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df7c1364-8484-4cb7-9010-b7acbd09ccba-plugins-config-volume\") pod \"df7c1364-8484-4cb7-9010-b7acbd09ccba\" (UID: \"df7c1364-8484-4cb7-9010-b7acbd09ccba\") " Nov 01 16:10:17.748592 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:17.748572 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/729e8251-2e65-4826-a535-a19b551e04c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/729e8251-2e65-4826-a535-a19b551e04c7-plugins-config-volume\") pod \"729e8251-2e65-4826-a535-a19b551e04c7\" (UID: \"729e8251-2e65-4826-a535-a19b551e04c7\") " Nov 01 16:10:17.748697 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:17.748680 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/729e8251-2e65-4826-a535-a19b551e04c7-kube-api-access-fdtww\" (UniqueName: \"kubernetes.io/projected/729e8251-2e65-4826-a535-a19b551e04c7-kube-api-access-fdtww\") pod \"729e8251-2e65-4826-a535-a19b551e04c7\" (UID: \"729e8251-2e65-4826-a535-a19b551e04c7\") " Nov 01 16:10:17.750805 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:17.750751 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/729e8251-2e65-4826-a535-a19b551e04c7-plugins-config-volume" pod "729e8251-2e65-4826-a535-a19b551e04c7" (UID: "729e8251-2e65-4826-a535-a19b551e04c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:10:17.750939 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:17.750901 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df7c1364-8484-4cb7-9010-b7acbd09ccba-plugins-config-volume" pod "df7c1364-8484-4cb7-9010-b7acbd09ccba" (UID: "df7c1364-8484-4cb7-9010-b7acbd09ccba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:10:17.753614 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:17.753575 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/729e8251-2e65-4826-a535-a19b551e04c7-kube-api-access-fdtww" pod "729e8251-2e65-4826-a535-a19b551e04c7" (UID: "729e8251-2e65-4826-a535-a19b551e04c7"). InnerVolumeSpecName "kube-api-access-fdtww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:17.754479 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:17.754414 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df7c1364-8484-4cb7-9010-b7acbd09ccba-kube-api-access-lgn95" pod "df7c1364-8484-4cb7-9010-b7acbd09ccba" (UID: "df7c1364-8484-4cb7-9010-b7acbd09ccba"). InnerVolumeSpecName "kube-api-access-lgn95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:17.902080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:17.901564 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/729e8251-2e65-4826-a535-a19b551e04c7-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:17.902080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:17.901598 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdtww\" (UniqueName: \"kubernetes.io/projected/729e8251-2e65-4826-a535-a19b551e04c7-kube-api-access-fdtww\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:17.902080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:17.901614 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgn95\" (UniqueName: \"kubernetes.io/projected/df7c1364-8484-4cb7-9010-b7acbd09ccba-kube-api-access-lgn95\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:17.902080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:17.901626 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df7c1364-8484-4cb7-9010-b7acbd09ccba-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:18.233095 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:18.233066 4363 generic.go:356] "Generic (PLEG): container finished" podID="a77a07cf-0f15-4724-8170-da59b2a1e5f7" containerID="0285c96e956edbf8d64bb79c684597d677be6dfff08e15028b990bad54fcdbd6" exitCode=0 Nov 01 16:10:18.233295 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:18.233146 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" event={"ID":"a77a07cf-0f15-4724-8170-da59b2a1e5f7","Type":"ContainerDied","Data":"0285c96e956edbf8d64bb79c684597d677be6dfff08e15028b990bad54fcdbd6"} Nov 01 16:10:18.233662 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:18.233642 4363 scope.go:122] "RemoveContainer" containerID="0285c96e956edbf8d64bb79c684597d677be6dfff08e15028b990bad54fcdbd6" Nov 01 16:10:18.234766 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:18.234663 4363 generic.go:356] "Generic (PLEG): container finished" podID="df7c1364-8484-4cb7-9010-b7acbd09ccba" containerID="c8f18e85231ccdb24839f583a9dc8bba3207c70dc65e9e8c75594f9f4004a707" exitCode=0 Nov 01 16:10:18.234766 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:18.234689 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c6b75-9f88c8787-vm79j" event={"ID":"df7c1364-8484-4cb7-9010-b7acbd09ccba","Type":"ContainerDied","Data":"c8f18e85231ccdb24839f583a9dc8bba3207c70dc65e9e8c75594f9f4004a707"} Nov 01 16:10:18.234766 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:18.234712 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c6b75-9f88c8787-vm79j" Nov 01 16:10:18.234766 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:18.234735 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c6b75-9f88c8787-vm79j" event={"ID":"df7c1364-8484-4cb7-9010-b7acbd09ccba","Type":"ContainerDied","Data":"9b8e8adeba0ac81eab170e3844d7e9907cb144712ec4eae77d6b9214fb54ce86"} Nov 01 16:10:18.234766 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:18.234755 4363 scope.go:122] "RemoveContainer" containerID="c8f18e85231ccdb24839f583a9dc8bba3207c70dc65e9e8c75594f9f4004a707" Nov 01 16:10:18.235946 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:18.235786 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:10:18.235946 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:18.235873 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:10:18.235946 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:18.235922 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:10:18.236374 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:18.236353 4363 generic.go:356] "Generic (PLEG): container finished" podID="729e8251-2e65-4826-a535-a19b551e04c7" containerID="307bbc000ccbf043f74a3fd2618d3fd25241b1b413b8ec0cb7562d6621d7689a" exitCode=0 Nov 01 16:10:18.236483 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:18.236402 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5c6b75-58cdb8f699-tzrc2" Nov 01 16:10:18.236553 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:18.236471 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5c6b75-58cdb8f699-tzrc2" event={"ID":"729e8251-2e65-4826-a535-a19b551e04c7","Type":"ContainerDied","Data":"307bbc000ccbf043f74a3fd2618d3fd25241b1b413b8ec0cb7562d6621d7689a"} Nov 01 16:10:18.236553 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:18.236529 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5c6b75-58cdb8f699-tzrc2" event={"ID":"729e8251-2e65-4826-a535-a19b551e04c7","Type":"ContainerDied","Data":"ba2bc9bc8bb701171068cae37e7622b505884a40b34087c529027d100157478c"} Nov 01 16:10:18.239177 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:18.239118 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6963-0ae593" event={"ID":"0d723602-2b68-4ab0-8c4b-10f9962180f7","Type":"ContainerStarted","Data":"4b334d9427f260217b53901877ffd9c03fab146e3baf2433e3f5c1b41a86df61"} Nov 01 16:10:18.239177 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:18.239157 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6963-0ae593" event={"ID":"0d723602-2b68-4ab0-8c4b-10f9962180f7","Type":"ContainerStarted","Data":"8dac060df2c9df5337ffb90a3c32b3497f1445eade704bde36dc3e5fee1754f1"} Nov 01 16:10:18.251990 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:18.251967 4363 scope.go:122] "RemoveContainer" containerID="c8f18e85231ccdb24839f583a9dc8bba3207c70dc65e9e8c75594f9f4004a707" Nov 01 16:10:18.252330 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:10:18.252305 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8f18e85231ccdb24839f583a9dc8bba3207c70dc65e9e8c75594f9f4004a707\": container with ID starting with c8f18e85231ccdb24839f583a9dc8bba3207c70dc65e9e8c75594f9f4004a707 not found: ID does not exist" containerID="c8f18e85231ccdb24839f583a9dc8bba3207c70dc65e9e8c75594f9f4004a707" Nov 01 16:10:18.252448 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:18.252387 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8f18e85231ccdb24839f583a9dc8bba3207c70dc65e9e8c75594f9f4004a707"} err="failed to get container status \"c8f18e85231ccdb24839f583a9dc8bba3207c70dc65e9e8c75594f9f4004a707\": rpc error: code = NotFound desc = could not find container \"c8f18e85231ccdb24839f583a9dc8bba3207c70dc65e9e8c75594f9f4004a707\": container with ID starting with c8f18e85231ccdb24839f583a9dc8bba3207c70dc65e9e8c75594f9f4004a707 not found: ID does not exist" Nov 01 16:10:18.252448 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:18.252412 4363 scope.go:122] "RemoveContainer" containerID="307bbc000ccbf043f74a3fd2618d3fd25241b1b413b8ec0cb7562d6621d7689a" Nov 01 16:10:18.264129 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:18.264096 4363 scope.go:122] "RemoveContainer" containerID="307bbc000ccbf043f74a3fd2618d3fd25241b1b413b8ec0cb7562d6621d7689a" Nov 01 16:10:18.264446 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:10:18.264423 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"307bbc000ccbf043f74a3fd2618d3fd25241b1b413b8ec0cb7562d6621d7689a\": container with ID starting with 307bbc000ccbf043f74a3fd2618d3fd25241b1b413b8ec0cb7562d6621d7689a not found: ID does not exist" containerID="307bbc000ccbf043f74a3fd2618d3fd25241b1b413b8ec0cb7562d6621d7689a" Nov 01 16:10:18.264553 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:18.264452 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"307bbc000ccbf043f74a3fd2618d3fd25241b1b413b8ec0cb7562d6621d7689a"} err="failed to get container status \"307bbc000ccbf043f74a3fd2618d3fd25241b1b413b8ec0cb7562d6621d7689a\": rpc error: code = NotFound desc = could not find container \"307bbc000ccbf043f74a3fd2618d3fd25241b1b413b8ec0cb7562d6621d7689a\": container with ID starting with 307bbc000ccbf043f74a3fd2618d3fd25241b1b413b8ec0cb7562d6621d7689a not found: ID does not exist" Nov 01 16:10:18.293082 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:18.292944 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6963-0ae593" podStartSLOduration=2.292925432 podStartE2EDuration="2.292925432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:10:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:10:18.270157621 +0000 UTC m=+1351.601895272" watchObservedRunningTime="2026-11-01 16:10:18.292925432 +0000 UTC m=+1351.624663083" Nov 01 16:10:18.300109 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:18.300073 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5c6b75-58cdb8f699-tzrc2"] Nov 01 16:10:18.313356 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:18.313329 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5c6b75-58cdb8f699-tzrc2"] Nov 01 16:10:18.330251 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:18.326052 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5c6b75-9f88c8787-vm79j"] Nov 01 16:10:18.333159 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:18.332986 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5c6b75-9f88c8787-vm79j"] Nov 01 16:10:18.340409 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:10:18.340382 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:10:18.802146 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:18.802112 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="729e8251-2e65-4826-a535-a19b551e04c7" path="/var/lib/kubelet/pods/729e8251-2e65-4826-a535-a19b551e04c7/volumes" Nov 01 16:10:18.802698 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:18.802682 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df7c1364-8484-4cb7-9010-b7acbd09ccba" path="/var/lib/kubelet/pods/df7c1364-8484-4cb7-9010-b7acbd09ccba/volumes" Nov 01 16:10:18.932106 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:18.932075 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6963-0ae593"] Nov 01 16:10:18.938643 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:18.938612 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6963-0ae593"] Nov 01 16:10:19.439836 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:19.439793 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" event={"ID":"a77a07cf-0f15-4724-8170-da59b2a1e5f7","Type":"ContainerStarted","Data":"87c70fd1295e0cc86bea738904f47c071d47b9f01313a765303d9b3e691c0613"} Nov 01 16:10:19.839796 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:19.839707 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" Nov 01 16:10:19.840209 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:19.839890 4363 patch_prober.go:29] interesting pod/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 01 16:10:19.840209 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:19.839933 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" podUID="a77a07cf-0f15-4724-8170-da59b2a1e5f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 01 16:10:20.092523 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:20.092496 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-166778-77d4bcfc6f-88bhr" Nov 01 16:10:20.497094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:20.495085 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6963-0ae593" podUID="0d723602-2b68-4ab0-8c4b-10f9962180f7" containerName="test-curl-6963-0ae593" containerID="cri-o://4b334d9427f260217b53901877ffd9c03fab146e3baf2433e3f5c1b41a86df61" gracePeriod=2 Nov 01 16:10:20.517901 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:10:20.517858 4363 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6963-0ae593\" is forbidden: User \"system:node:ip-10-0-128-238.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-238.us-east-2.compute.internal' and this object" podUID="0d723602-2b68-4ab0-8c4b-10f9962180f7" pod="opendatahub/test-curl-6963-0ae593" Nov 01 16:10:20.740979 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:20.740942 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6"] Nov 01 16:10:20.742479 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:20.742453 4363 cpu_manager.go:414] "containerMap: removing container" podUID="0d723602-2b68-4ab0-8c4b-10f9962180f7" containerName="test-curl-6963-0ae593" Nov 01 16:10:20.742479 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:20.742477 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d723602-2b68-4ab0-8c4b-10f9962180f7" containerName="test-curl-6963-0ae593" Nov 01 16:10:20.742669 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:20.742496 4363 cpu_manager.go:414] "containerMap: removing container" podUID="df7c1364-8484-4cb7-9010-b7acbd09ccba" containerName="payload-pre-processing" Nov 01 16:10:20.742669 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:20.742504 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df7c1364-8484-4cb7-9010-b7acbd09ccba" containerName="payload-pre-processing" Nov 01 16:10:20.742669 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:20.742567 4363 cpu_manager.go:414] "containerMap: removing container" podUID="729e8251-2e65-4826-a535-a19b551e04c7" containerName="payload-processing" Nov 01 16:10:20.742669 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:20.742576 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="729e8251-2e65-4826-a535-a19b551e04c7" containerName="payload-processing" Nov 01 16:10:20.742868 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:20.742776 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="729e8251-2e65-4826-a535-a19b551e04c7" containerName="payload-processing" Nov 01 16:10:20.742868 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:20.742801 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d723602-2b68-4ab0-8c4b-10f9962180f7" containerName="test-curl-6963-0ae593" Nov 01 16:10:20.742868 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:20.742812 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="df7c1364-8484-4cb7-9010-b7acbd09ccba" containerName="payload-pre-processing" Nov 01 16:10:20.748695 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:20.748634 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" Nov 01 16:10:20.766680 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:20.765849 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6"] Nov 01 16:10:20.811217 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:10:20.811180 4363 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6963-0ae593\" is forbidden: User \"system:node:ip-10-0-128-238.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-238.us-east-2.compute.internal' and this object" podUID="0d723602-2b68-4ab0-8c4b-10f9962180f7" pod="opendatahub/test-curl-6963-0ae593" Nov 01 16:10:20.823384 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:20.823354 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-375350-openshift-default-dockercfg-wzh7t\"" Nov 01 16:10:20.839397 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:20.839267 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba075b49-a3d6-4813-bbcf-37f11e5550a1-credential-socket\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6\" (UID: \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" Nov 01 16:10:20.839397 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:20.839353 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba075b49-a3d6-4813-bbcf-37f11e5550a1-istiod-ca-cert\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6\" (UID: \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" Nov 01 16:10:20.839642 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:20.839470 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba075b49-a3d6-4813-bbcf-37f11e5550a1-istio-envoy\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6\" (UID: \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" Nov 01 16:10:20.839642 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:20.839581 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba075b49-a3d6-4813-bbcf-37f11e5550a1-workload-socket\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6\" (UID: \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" Nov 01 16:10:20.839642 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:20.839629 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba075b49-a3d6-4813-bbcf-37f11e5550a1-istio-data\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6\" (UID: \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" Nov 01 16:10:20.840285 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:20.839710 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba075b49-a3d6-4813-bbcf-37f11e5550a1-workload-certs\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6\" (UID: \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" Nov 01 16:10:20.840285 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:20.839738 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba075b49-a3d6-4813-bbcf-37f11e5550a1-istio-token\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6\" (UID: \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" Nov 01 16:10:20.840285 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:20.839867 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68j26\" (UniqueName: \"kubernetes.io/projected/ba075b49-a3d6-4813-bbcf-37f11e5550a1-kube-api-access-68j26\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6\" (UID: \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" Nov 01 16:10:20.840285 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:20.839948 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba075b49-a3d6-4813-bbcf-37f11e5550a1-istio-podinfo\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6\" (UID: \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" Nov 01 16:10:20.840657 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:20.840631 4363 patch_prober.go:29] interesting pod/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 01 16:10:20.840750 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:20.840685 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" podUID="a77a07cf-0f15-4724-8170-da59b2a1e5f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 01 16:10:20.991149 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:20.991114 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba075b49-a3d6-4813-bbcf-37f11e5550a1-workload-socket\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6\" (UID: \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" Nov 01 16:10:21.007638 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:20.991988 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba075b49-a3d6-4813-bbcf-37f11e5550a1-workload-socket\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6\" (UID: \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" Nov 01 16:10:21.008354 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:21.008317 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba075b49-a3d6-4813-bbcf-37f11e5550a1-istio-data\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6\" (UID: \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" Nov 01 16:10:21.008548 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:21.007536 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba075b49-a3d6-4813-bbcf-37f11e5550a1-istio-data\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6\" (UID: \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" Nov 01 16:10:21.009132 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:21.009095 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba075b49-a3d6-4813-bbcf-37f11e5550a1-workload-certs\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6\" (UID: \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" Nov 01 16:10:21.009648 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:21.009570 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba075b49-a3d6-4813-bbcf-37f11e5550a1-workload-certs\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6\" (UID: \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" Nov 01 16:10:21.009797 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:21.009619 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba075b49-a3d6-4813-bbcf-37f11e5550a1-istio-token\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6\" (UID: \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" Nov 01 16:10:21.010953 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:21.010927 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68j26\" (UniqueName: \"kubernetes.io/projected/ba075b49-a3d6-4813-bbcf-37f11e5550a1-kube-api-access-68j26\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6\" (UID: \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" Nov 01 16:10:21.028382 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:21.011590 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba075b49-a3d6-4813-bbcf-37f11e5550a1-istio-podinfo\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6\" (UID: \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" Nov 01 16:10:21.028691 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:21.028669 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba075b49-a3d6-4813-bbcf-37f11e5550a1-credential-socket\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6\" (UID: \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" Nov 01 16:10:21.034403 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:21.029309 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba075b49-a3d6-4813-bbcf-37f11e5550a1-istiod-ca-cert\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6\" (UID: \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" Nov 01 16:10:21.034803 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:21.029174 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba075b49-a3d6-4813-bbcf-37f11e5550a1-credential-socket\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6\" (UID: \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" Nov 01 16:10:21.042527 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:21.030090 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba075b49-a3d6-4813-bbcf-37f11e5550a1-istiod-ca-cert\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6\" (UID: \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" Nov 01 16:10:21.042838 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:21.042818 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba075b49-a3d6-4813-bbcf-37f11e5550a1-istio-envoy\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6\" (UID: \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" Nov 01 16:10:21.083116 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:21.066929 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68j26\" (UniqueName: \"kubernetes.io/projected/ba075b49-a3d6-4813-bbcf-37f11e5550a1-kube-api-access-68j26\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6\" (UID: \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" Nov 01 16:10:21.119879 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:21.119849 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-166778-7d8fcb5659-jq7tt" Nov 01 16:10:21.126038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:21.125387 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba075b49-a3d6-4813-bbcf-37f11e5550a1-istio-envoy\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6\" (UID: \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" Nov 01 16:10:21.126849 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:21.126730 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba075b49-a3d6-4813-bbcf-37f11e5550a1-istio-token\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6\" (UID: \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" Nov 01 16:10:21.164069 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:21.153632 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba075b49-a3d6-4813-bbcf-37f11e5550a1-istio-podinfo\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6\" (UID: \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" Nov 01 16:10:21.399864 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:21.399165 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" Nov 01 16:10:21.840266 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:21.840234 4363 patch_prober.go:29] interesting pod/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 01 16:10:21.840749 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:21.840296 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" podUID="a77a07cf-0f15-4724-8170-da59b2a1e5f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 01 16:10:21.864051 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:10:21.864006 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba075b49_a3d6_4813_bbcf_37f11e5550a1.slice/crio-49e844f5036b6ee011b7e64272efc058b14de10edfc0c3d2c558194ea382e764 WatchSource:0}: Error finding container 49e844f5036b6ee011b7e64272efc058b14de10edfc0c3d2c558194ea382e764: Status 404 returned error can't find the container with id 49e844f5036b6ee011b7e64272efc058b14de10edfc0c3d2c558194ea382e764 Nov 01 16:10:21.864738 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:21.864679 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6"] Nov 01 16:10:21.866072 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:21.866040 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:10:21.866164 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:21.866103 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:10:21.866164 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:21.866133 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:10:22.740753 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:22.738286 4363 generic.go:356] "Generic (PLEG): container finished" podID="0d723602-2b68-4ab0-8c4b-10f9962180f7" containerID="4b334d9427f260217b53901877ffd9c03fab146e3baf2433e3f5c1b41a86df61" exitCode=137 Nov 01 16:10:22.740753 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:22.740637 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" event={"ID":"ba075b49-a3d6-4813-bbcf-37f11e5550a1","Type":"ContainerStarted","Data":"57d261958ee14cf1fb89882b6a7e49c3f10510781bfa7b4b0ce85668171dae1f"} Nov 01 16:10:22.740753 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:22.740699 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" event={"ID":"ba075b49-a3d6-4813-bbcf-37f11e5550a1","Type":"ContainerStarted","Data":"49e844f5036b6ee011b7e64272efc058b14de10edfc0c3d2c558194ea382e764"} Nov 01 16:10:22.766316 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:22.766258 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" podStartSLOduration=2.766239156 podStartE2EDuration="2.766239156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:10:22.764314642 +0000 UTC m=+1356.096052294" watchObservedRunningTime="2026-11-01 16:10:22.766239156 +0000 UTC m=+1356.097976808" Nov 01 16:10:22.839705 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:22.839676 4363 patch_prober.go:29] interesting pod/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 01 16:10:22.839829 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:22.839726 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" podUID="a77a07cf-0f15-4724-8170-da59b2a1e5f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 01 16:10:22.858467 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:22.858439 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6963-0ae593" Nov 01 16:10:23.074255 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:23.071116 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-166778-cd665bd8b-xhwj6"] Nov 01 16:10:23.077980 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:23.077954 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-166778-cd665bd8b-xhwj6"] Nov 01 16:10:23.078313 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:23.078295 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-166778-cd665bd8b-xhwj6" Nov 01 16:10:23.120043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:23.113983 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d723602-2b68-4ab0-8c4b-10f9962180f7-kube-api-access-bw9vx\" (UniqueName: \"kubernetes.io/projected/0d723602-2b68-4ab0-8c4b-10f9962180f7-kube-api-access-bw9vx\") pod \"0d723602-2b68-4ab0-8c4b-10f9962180f7\" (UID: \"0d723602-2b68-4ab0-8c4b-10f9962180f7\") " Nov 01 16:10:23.125708 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:23.125670 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-166778-serving-cert\"" Nov 01 16:10:23.148342 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:23.147987 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d723602-2b68-4ab0-8c4b-10f9962180f7-kube-api-access-bw9vx" pod "0d723602-2b68-4ab0-8c4b-10f9962180f7" (UID: "0d723602-2b68-4ab0-8c4b-10f9962180f7"). InnerVolumeSpecName "kube-api-access-bw9vx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:23.265087 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:23.247386 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcf26f07-340c-4852-8213-e5ab594f33b0-maas-api-tls\") pod \"maas-api-e2e-api-a-166778-cd665bd8b-xhwj6\" (UID: \"dcf26f07-340c-4852-8213-e5ab594f33b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-166778-cd665bd8b-xhwj6" Nov 01 16:10:23.265087 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:23.247488 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcf26f07-340c-4852-8213-e5ab594f33b0-openshift-service-ca\") pod \"maas-api-e2e-api-a-166778-cd665bd8b-xhwj6\" (UID: \"dcf26f07-340c-4852-8213-e5ab594f33b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-166778-cd665bd8b-xhwj6" Nov 01 16:10:23.265087 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:23.247833 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbdt6\" (UniqueName: \"kubernetes.io/projected/dcf26f07-340c-4852-8213-e5ab594f33b0-kube-api-access-fbdt6\") pod \"maas-api-e2e-api-a-166778-cd665bd8b-xhwj6\" (UID: \"dcf26f07-340c-4852-8213-e5ab594f33b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-166778-cd665bd8b-xhwj6" Nov 01 16:10:23.265087 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:23.247932 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcf26f07-340c-4852-8213-e5ab594f33b0-metrics-certs\") pod \"maas-api-e2e-api-a-166778-cd665bd8b-xhwj6\" (UID: \"dcf26f07-340c-4852-8213-e5ab594f33b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-166778-cd665bd8b-xhwj6" Nov 01 16:10:23.265087 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:23.248160 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw9vx\" (UniqueName: \"kubernetes.io/projected/0d723602-2b68-4ab0-8c4b-10f9962180f7-kube-api-access-bw9vx\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:23.366045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:23.364760 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbdt6\" (UniqueName: \"kubernetes.io/projected/dcf26f07-340c-4852-8213-e5ab594f33b0-kube-api-access-fbdt6\") pod \"maas-api-e2e-api-a-166778-cd665bd8b-xhwj6\" (UID: \"dcf26f07-340c-4852-8213-e5ab594f33b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-166778-cd665bd8b-xhwj6" Nov 01 16:10:23.366045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:23.364839 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcf26f07-340c-4852-8213-e5ab594f33b0-metrics-certs\") pod \"maas-api-e2e-api-a-166778-cd665bd8b-xhwj6\" (UID: \"dcf26f07-340c-4852-8213-e5ab594f33b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-166778-cd665bd8b-xhwj6" Nov 01 16:10:23.366045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:23.364999 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcf26f07-340c-4852-8213-e5ab594f33b0-maas-api-tls\") pod \"maas-api-e2e-api-a-166778-cd665bd8b-xhwj6\" (UID: \"dcf26f07-340c-4852-8213-e5ab594f33b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-166778-cd665bd8b-xhwj6" Nov 01 16:10:23.424039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:23.422481 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" Nov 01 16:10:23.447795 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:23.446609 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcf26f07-340c-4852-8213-e5ab594f33b0-maas-api-tls\") pod \"maas-api-e2e-api-a-166778-cd665bd8b-xhwj6\" (UID: \"dcf26f07-340c-4852-8213-e5ab594f33b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-166778-cd665bd8b-xhwj6" Nov 01 16:10:23.478158 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:23.464798 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" Nov 01 16:10:23.478158 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:23.475244 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcf26f07-340c-4852-8213-e5ab594f33b0-openshift-service-ca\") pod \"maas-api-e2e-api-a-166778-cd665bd8b-xhwj6\" (UID: \"dcf26f07-340c-4852-8213-e5ab594f33b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-166778-cd665bd8b-xhwj6" Nov 01 16:10:23.488079 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:23.481646 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcf26f07-340c-4852-8213-e5ab594f33b0-openshift-service-ca\") pod \"maas-api-e2e-api-a-166778-cd665bd8b-xhwj6\" (UID: \"dcf26f07-340c-4852-8213-e5ab594f33b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-166778-cd665bd8b-xhwj6" Nov 01 16:10:23.488079 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:23.485927 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbdt6\" (UniqueName: \"kubernetes.io/projected/dcf26f07-340c-4852-8213-e5ab594f33b0-kube-api-access-fbdt6\") pod \"maas-api-e2e-api-a-166778-cd665bd8b-xhwj6\" (UID: \"dcf26f07-340c-4852-8213-e5ab594f33b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-166778-cd665bd8b-xhwj6" Nov 01 16:10:23.532748 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:23.532700 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcf26f07-340c-4852-8213-e5ab594f33b0-metrics-certs\") pod \"maas-api-e2e-api-a-166778-cd665bd8b-xhwj6\" (UID: \"dcf26f07-340c-4852-8213-e5ab594f33b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-166778-cd665bd8b-xhwj6" Nov 01 16:10:23.712177 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:23.708394 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-166778-cd665bd8b-xhwj6" Nov 01 16:10:23.815966 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:23.815937 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6963-0ae593" Nov 01 16:10:23.830732 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:23.827186 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" Nov 01 16:10:23.830732 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:23.827256 4363 scope.go:122] "RemoveContainer" containerID="4b334d9427f260217b53901877ffd9c03fab146e3baf2433e3f5c1b41a86df61" Nov 01 16:10:23.844247 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:23.843109 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" Nov 01 16:10:23.845320 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:23.845295 4363 patch_prober.go:29] interesting pod/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 01 16:10:23.848049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:23.845515 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" podUID="a77a07cf-0f15-4724-8170-da59b2a1e5f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 01 16:10:23.978158 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:10:23.977469 4363 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6963-0ae593\" is forbidden: User \"system:node:ip-10-0-128-238.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-238.us-east-2.compute.internal' and this object" podUID="0d723602-2b68-4ab0-8c4b-10f9962180f7" pod="opendatahub/test-curl-6963-0ae593" Nov 01 16:10:24.366554 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:10:24.366517 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcf26f07_340c_4852_8213_e5ab594f33b0.slice/crio-07eefb8e53c48488e1fdde0f07f4198a72f3ad7a892647275012b1a52da46b73 WatchSource:0}: Error finding container 07eefb8e53c48488e1fdde0f07f4198a72f3ad7a892647275012b1a52da46b73: Status 404 returned error can't find the container with id 07eefb8e53c48488e1fdde0f07f4198a72f3ad7a892647275012b1a52da46b73 Nov 01 16:10:24.367808 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:24.367775 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-166778-cd665bd8b-xhwj6"] Nov 01 16:10:24.814522 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:24.814491 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d723602-2b68-4ab0-8c4b-10f9962180f7" path="/var/lib/kubelet/pods/0d723602-2b68-4ab0-8c4b-10f9962180f7/volumes" Nov 01 16:10:24.825954 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:24.825920 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-166778-cd665bd8b-xhwj6" event={"ID":"dcf26f07-340c-4852-8213-e5ab594f33b0","Type":"ContainerStarted","Data":"07eefb8e53c48488e1fdde0f07f4198a72f3ad7a892647275012b1a52da46b73"} Nov 01 16:10:24.839387 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:24.839357 4363 patch_prober.go:29] interesting pod/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 01 16:10:24.839494 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:24.839412 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" podUID="a77a07cf-0f15-4724-8170-da59b2a1e5f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 01 16:10:25.661904 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:25.661276 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" Nov 01 16:10:25.881054 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:25.876648 4363 patch_prober.go:29] interesting pod/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 01 16:10:25.881054 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:25.876708 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" podUID="a77a07cf-0f15-4724-8170-da59b2a1e5f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 01 16:10:26.037089 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:26.034891 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-166778-cd665bd8b-xhwj6" event={"ID":"dcf26f07-340c-4852-8213-e5ab594f33b0","Type":"ContainerStarted","Data":"bdb2f5f39501aad360fb48d8ba8c9471a51599c9a354138238e2688251f85f32"} Nov 01 16:10:26.037089 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:26.034984 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-166778-cd665bd8b-xhwj6" Nov 01 16:10:26.636139 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:26.636085 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-166778-cd665bd8b-xhwj6" podStartSLOduration=4.196636545 podStartE2EDuration="4.636070495s" totalImagesPullingTime="439.43395ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:10:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:10:26.076079297 +0000 UTC m=+1359.407816948" watchObservedRunningTime="2026-11-01 16:10:26.636070495 +0000 UTC m=+1359.967808144" Nov 01 16:10:26.637226 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:26.637206 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-375350-b98cbd657-76rtq"] Nov 01 16:10:26.643969 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:26.643940 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-375350-b98cbd657-76rtq" Nov 01 16:10:26.653863 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:26.653826 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-375350-b98cbd657-76rtq"] Nov 01 16:10:26.655258 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:26.655234 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-375350\"" Nov 01 16:10:26.669064 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:26.668995 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-375350-68b888b95-4mx6d"] Nov 01 16:10:26.697081 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:26.687519 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-375350-68b888b95-4mx6d" Nov 01 16:10:26.699735 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:26.699707 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-375350-dockercfg-srdtm\"" Nov 01 16:10:26.703529 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:26.703462 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-375350-68b888b95-4mx6d"] Nov 01 16:10:26.808803 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:26.808765 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6g7z\" (UniqueName: \"kubernetes.io/projected/7d0c63a3-8c1f-4c54-9373-afed7b61795c-kube-api-access-d6g7z\") pod \"payload-processing-e2e-inf-b-375350-b98cbd657-76rtq\" (UID: \"7d0c63a3-8c1f-4c54-9373-afed7b61795c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-375350-b98cbd657-76rtq" Nov 01 16:10:26.822267 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:26.822224 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb084141-b900-462d-ac93-b1ce933d6f2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-375350-68b888b95-4mx6d\" (UID: \"eb084141-b900-462d-ac93-b1ce933d6f2e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-375350-68b888b95-4mx6d" Nov 01 16:10:26.823228 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:26.823198 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvs54\" (UniqueName: \"kubernetes.io/projected/eb084141-b900-462d-ac93-b1ce933d6f2e-kube-api-access-hvs54\") pod \"payload-pre-processing-e2e-inf-b-375350-68b888b95-4mx6d\" (UID: \"eb084141-b900-462d-ac93-b1ce933d6f2e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-375350-68b888b95-4mx6d" Nov 01 16:10:26.823436 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:26.823418 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d0c63a3-8c1f-4c54-9373-afed7b61795c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-375350-b98cbd657-76rtq\" (UID: \"7d0c63a3-8c1f-4c54-9373-afed7b61795c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-375350-b98cbd657-76rtq" Nov 01 16:10:26.937768 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:26.934368 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6g7z\" (UniqueName: \"kubernetes.io/projected/7d0c63a3-8c1f-4c54-9373-afed7b61795c-kube-api-access-d6g7z\") pod \"payload-processing-e2e-inf-b-375350-b98cbd657-76rtq\" (UID: \"7d0c63a3-8c1f-4c54-9373-afed7b61795c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-375350-b98cbd657-76rtq" Nov 01 16:10:26.937768 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:26.934528 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb084141-b900-462d-ac93-b1ce933d6f2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-375350-68b888b95-4mx6d\" (UID: \"eb084141-b900-462d-ac93-b1ce933d6f2e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-375350-68b888b95-4mx6d" Nov 01 16:10:26.937768 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:26.934662 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvs54\" (UniqueName: \"kubernetes.io/projected/eb084141-b900-462d-ac93-b1ce933d6f2e-kube-api-access-hvs54\") pod \"payload-pre-processing-e2e-inf-b-375350-68b888b95-4mx6d\" (UID: \"eb084141-b900-462d-ac93-b1ce933d6f2e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-375350-68b888b95-4mx6d" Nov 01 16:10:26.937768 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:26.934700 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d0c63a3-8c1f-4c54-9373-afed7b61795c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-375350-b98cbd657-76rtq\" (UID: \"7d0c63a3-8c1f-4c54-9373-afed7b61795c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-375350-b98cbd657-76rtq" Nov 01 16:10:26.937768 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:26.935661 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d0c63a3-8c1f-4c54-9373-afed7b61795c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-375350-b98cbd657-76rtq\" (UID: \"7d0c63a3-8c1f-4c54-9373-afed7b61795c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-375350-b98cbd657-76rtq" Nov 01 16:10:26.941009 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:26.940982 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb084141-b900-462d-ac93-b1ce933d6f2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-375350-68b888b95-4mx6d\" (UID: \"eb084141-b900-462d-ac93-b1ce933d6f2e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-375350-68b888b95-4mx6d" Nov 01 16:10:26.975814 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:26.975777 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6g7z\" (UniqueName: \"kubernetes.io/projected/7d0c63a3-8c1f-4c54-9373-afed7b61795c-kube-api-access-d6g7z\") pod \"payload-processing-e2e-inf-b-375350-b98cbd657-76rtq\" (UID: \"7d0c63a3-8c1f-4c54-9373-afed7b61795c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-375350-b98cbd657-76rtq" Nov 01 16:10:26.976359 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:26.976331 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-375350-b98cbd657-76rtq" Nov 01 16:10:27.008568 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:27.005687 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvs54\" (UniqueName: \"kubernetes.io/projected/eb084141-b900-462d-ac93-b1ce933d6f2e-kube-api-access-hvs54\") pod \"payload-pre-processing-e2e-inf-b-375350-68b888b95-4mx6d\" (UID: \"eb084141-b900-462d-ac93-b1ce933d6f2e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-375350-68b888b95-4mx6d" Nov 01 16:10:27.043890 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:27.034575 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-375350-68b888b95-4mx6d" Nov 01 16:10:27.168126 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:27.161506 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" Nov 01 16:10:27.168704 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:27.168604 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" Nov 01 16:10:27.560633 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:10:27.560602 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb084141_b900_462d_ac93_b1ce933d6f2e.slice/crio-1f01067561241c49907a3d32702e867139f8f926410ed989c06bb4e255680ac9 WatchSource:0}: Error finding container 1f01067561241c49907a3d32702e867139f8f926410ed989c06bb4e255680ac9: Status 404 returned error can't find the container with id 1f01067561241c49907a3d32702e867139f8f926410ed989c06bb4e255680ac9 Nov 01 16:10:27.561154 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:10:27.561130 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d0c63a3_8c1f_4c54_9373_afed7b61795c.slice/crio-697b4e4255ad1855f8aea8ec3a19a1819201ac59311c119915864c2a96bd5b72 WatchSource:0}: Error finding container 697b4e4255ad1855f8aea8ec3a19a1819201ac59311c119915864c2a96bd5b72: Status 404 returned error can't find the container with id 697b4e4255ad1855f8aea8ec3a19a1819201ac59311c119915864c2a96bd5b72 Nov 01 16:10:27.563699 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:27.563678 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-375350-68b888b95-4mx6d"] Nov 01 16:10:27.568498 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:27.568473 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-375350-b98cbd657-76rtq"] Nov 01 16:10:28.083462 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:28.083421 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-375350-b98cbd657-76rtq" event={"ID":"7d0c63a3-8c1f-4c54-9373-afed7b61795c","Type":"ContainerStarted","Data":"bc94780c2d6af818ef77a848f09edb05f03b31df70a630ae9e05d5b59175d469"} Nov 01 16:10:28.083913 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:28.083486 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-375350-b98cbd657-76rtq" Nov 01 16:10:28.083913 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:28.083508 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-375350-b98cbd657-76rtq" event={"ID":"7d0c63a3-8c1f-4c54-9373-afed7b61795c","Type":"ContainerStarted","Data":"697b4e4255ad1855f8aea8ec3a19a1819201ac59311c119915864c2a96bd5b72"} Nov 01 16:10:28.085799 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:28.085770 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-375350-68b888b95-4mx6d" event={"ID":"eb084141-b900-462d-ac93-b1ce933d6f2e","Type":"ContainerStarted","Data":"6df9f030f44edd6d80b248963bafe3935cb735feb6ac14a5b00411804b593283"} Nov 01 16:10:28.085938 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:28.085825 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-375350-68b888b95-4mx6d" event={"ID":"eb084141-b900-462d-ac93-b1ce933d6f2e","Type":"ContainerStarted","Data":"1f01067561241c49907a3d32702e867139f8f926410ed989c06bb4e255680ac9"} Nov 01 16:10:28.085938 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:28.085853 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-375350-68b888b95-4mx6d" Nov 01 16:10:28.100622 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:28.100567 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-375350-b98cbd657-76rtq" podStartSLOduration=2.100553716 podStartE2EDuration="2.100553716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:10:28.098265928 +0000 UTC m=+1361.430003579" watchObservedRunningTime="2026-11-01 16:10:28.100553716 +0000 UTC m=+1361.432291366" Nov 01 16:10:28.114476 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:28.114428 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-375350-68b888b95-4mx6d" podStartSLOduration=2.114414464 podStartE2EDuration="2.114414464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:10:28.112663554 +0000 UTC m=+1361.444401216" watchObservedRunningTime="2026-11-01 16:10:28.114414464 +0000 UTC m=+1361.446152113" Nov 01 16:10:29.307989 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:29.307901 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-vrmjz"] Nov 01 16:10:29.322382 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:29.322349 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-lqtdt"] Nov 01 16:10:31.236594 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:31.236558 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n"] Nov 01 16:10:31.248535 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:31.248501 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" Nov 01 16:10:31.253187 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:31.253161 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-58b0d1-openshift-default-dockercfg-8plw7\"" Nov 01 16:10:31.255283 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:31.255251 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n"] Nov 01 16:10:31.264718 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:31.264684 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62gfr\" (UniqueName: \"kubernetes.io/projected/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-kube-api-access-62gfr\") pod \"e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n\" (UID: \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\") " pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" Nov 01 16:10:31.264718 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:31.264760 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-istio-envoy\") pod \"e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n\" (UID: \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\") " pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" Nov 01 16:10:31.265214 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:31.264805 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-istio-podinfo\") pod \"e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n\" (UID: \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\") " pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" Nov 01 16:10:31.265214 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:31.264838 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-credential-socket\") pod \"e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n\" (UID: \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\") " pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" Nov 01 16:10:31.265214 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:31.265047 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-workload-socket\") pod \"e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n\" (UID: \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\") " pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" Nov 01 16:10:31.267409 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:31.266097 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-workload-certs\") pod \"e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n\" (UID: \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\") " pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" Nov 01 16:10:31.267409 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:31.266167 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-istio-data\") pod \"e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n\" (UID: \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\") " pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" Nov 01 16:10:31.267409 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:31.266230 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-istio-token\") pod \"e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n\" (UID: \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\") " pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" Nov 01 16:10:31.267409 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:31.266265 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-istiod-ca-cert\") pod \"e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n\" (UID: \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\") " pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" Nov 01 16:10:31.412045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:31.405951 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-workload-certs\") pod \"e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n\" (UID: \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\") " pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" Nov 01 16:10:31.412045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:31.406072 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-istio-data\") pod \"e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n\" (UID: \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\") " pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" Nov 01 16:10:31.412045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:31.406309 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-istio-token\") pod \"e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n\" (UID: \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\") " pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" Nov 01 16:10:31.412045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:31.406348 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-istiod-ca-cert\") pod \"e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n\" (UID: \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\") " pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" Nov 01 16:10:31.412045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:31.406400 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62gfr\" (UniqueName: \"kubernetes.io/projected/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-kube-api-access-62gfr\") pod \"e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n\" (UID: \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\") " pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" Nov 01 16:10:31.412045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:31.406550 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-istio-envoy\") pod \"e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n\" (UID: \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\") " pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" Nov 01 16:10:31.412045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:31.406627 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-istio-podinfo\") pod \"e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n\" (UID: \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\") " pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" Nov 01 16:10:31.412045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:31.406698 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-credential-socket\") pod \"e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n\" (UID: \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\") " pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" Nov 01 16:10:31.412045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:31.406931 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-workload-socket\") pod \"e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n\" (UID: \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\") " pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" Nov 01 16:10:31.412045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:31.407527 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-workload-socket\") pod \"e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n\" (UID: \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\") " pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" Nov 01 16:10:31.412045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:31.407776 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-workload-certs\") pod \"e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n\" (UID: \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\") " pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" Nov 01 16:10:31.412045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:31.408080 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-istio-data\") pod \"e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n\" (UID: \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\") " pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" Nov 01 16:10:31.412045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:31.409980 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-credential-socket\") pod \"e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n\" (UID: \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\") " pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" Nov 01 16:10:31.412045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:31.410396 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-istiod-ca-cert\") pod \"e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n\" (UID: \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\") " pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" Nov 01 16:10:31.437055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:31.415259 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-istio-envoy\") pod \"e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n\" (UID: \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\") " pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" Nov 01 16:10:31.488635 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:31.469618 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-istio-podinfo\") pod \"e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n\" (UID: \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\") " pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" Nov 01 16:10:31.488635 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:31.476838 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-istio-token\") pod \"e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n\" (UID: \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\") " pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" Nov 01 16:10:31.568042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:31.552843 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62gfr\" (UniqueName: \"kubernetes.io/projected/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-kube-api-access-62gfr\") pod \"e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n\" (UID: \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\") " pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" Nov 01 16:10:31.604152 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:31.596197 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" Nov 01 16:10:32.030966 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:10:32.030930 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod905f9285_a9c5_4a2a_abbf_8114e5ac9f26.slice/crio-c5f51fe52ad144614d5f8a6c68b796c349aad0fb7aabce6e15503507f4b23f8c WatchSource:0}: Error finding container c5f51fe52ad144614d5f8a6c68b796c349aad0fb7aabce6e15503507f4b23f8c: Status 404 returned error can't find the container with id c5f51fe52ad144614d5f8a6c68b796c349aad0fb7aabce6e15503507f4b23f8c Nov 01 16:10:32.033061 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:32.033036 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n"] Nov 01 16:10:32.034375 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:32.034341 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:10:32.034481 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:32.034409 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:10:32.034481 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:32.034442 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:10:32.081747 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:32.081721 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-166778-cd665bd8b-xhwj6" Nov 01 16:10:32.249676 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:32.249566 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" event={"ID":"905f9285-a9c5-4a2a-abbf-8114e5ac9f26","Type":"ContainerStarted","Data":"c5f51fe52ad144614d5f8a6c68b796c349aad0fb7aabce6e15503507f4b23f8c"} Nov 01 16:10:33.266790 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:33.266748 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" event={"ID":"905f9285-a9c5-4a2a-abbf-8114e5ac9f26","Type":"ContainerStarted","Data":"d4ed8c081c6ccf18dbbda3c56e45e17bc2f2c8535f043a60ffef029233d40292"} Nov 01 16:10:33.288666 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:33.288609 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" podStartSLOduration=2.28859513 podStartE2EDuration="2.28859513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:10:33.286245164 +0000 UTC m=+1366.617982815" watchObservedRunningTime="2026-11-01 16:10:33.28859513 +0000 UTC m=+1366.620332833" Nov 01 16:10:33.597356 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:33.597327 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" Nov 01 16:10:33.603982 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:33.603954 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" Nov 01 16:10:34.275489 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:34.275455 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" Nov 01 16:10:34.276713 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:34.276684 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" Nov 01 16:10:34.308910 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:34.308856 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-vrmjz" podUID="11a30818-8812-4f4d-a5c3-764d879deb52" containerName="payload-processing" containerID="cri-o://fdbf01d14088d583fcd34c8caa42a8299e9f25c8ac450e1ee22d0606f7330c09" gracePeriod=25 Nov 01 16:10:34.323822 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:34.323748 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-lqtdt" podUID="37d0ec61-3743-48db-bed1-7ba33650158a" containerName="payload-pre-processing" containerID="cri-o://49bbd26bead5c5a52af3a09249aaea79894eb6e3b02aae6521f0dfd9cc440fa0" gracePeriod=25 Nov 01 16:10:34.754734 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:34.754706 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-vrmjz" Nov 01 16:10:34.789977 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:34.789942 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-lqtdt" Nov 01 16:10:34.873586 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:34.873498 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11a30818-8812-4f4d-a5c3-764d879deb52-kube-api-access-wc5kk\" (UniqueName: \"kubernetes.io/projected/11a30818-8812-4f4d-a5c3-764d879deb52-kube-api-access-wc5kk\") pod \"11a30818-8812-4f4d-a5c3-764d879deb52\" (UID: \"11a30818-8812-4f4d-a5c3-764d879deb52\") " Nov 01 16:10:34.873870 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:34.873590 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11a30818-8812-4f4d-a5c3-764d879deb52-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a30818-8812-4f4d-a5c3-764d879deb52-plugins-config-volume\") pod \"11a30818-8812-4f4d-a5c3-764d879deb52\" (UID: \"11a30818-8812-4f4d-a5c3-764d879deb52\") " Nov 01 16:10:34.873870 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:34.873648 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37d0ec61-3743-48db-bed1-7ba33650158a-kube-api-access-tzlvr\" (UniqueName: \"kubernetes.io/projected/37d0ec61-3743-48db-bed1-7ba33650158a-kube-api-access-tzlvr\") pod \"37d0ec61-3743-48db-bed1-7ba33650158a\" (UID: \"37d0ec61-3743-48db-bed1-7ba33650158a\") " Nov 01 16:10:34.873870 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:34.873725 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37d0ec61-3743-48db-bed1-7ba33650158a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37d0ec61-3743-48db-bed1-7ba33650158a-plugins-config-volume\") pod \"37d0ec61-3743-48db-bed1-7ba33650158a\" (UID: \"37d0ec61-3743-48db-bed1-7ba33650158a\") " Nov 01 16:10:34.877064 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:34.874091 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11a30818-8812-4f4d-a5c3-764d879deb52-plugins-config-volume" pod "11a30818-8812-4f4d-a5c3-764d879deb52" (UID: "11a30818-8812-4f4d-a5c3-764d879deb52"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:10:34.877064 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:34.874138 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37d0ec61-3743-48db-bed1-7ba33650158a-plugins-config-volume" pod "37d0ec61-3743-48db-bed1-7ba33650158a" (UID: "37d0ec61-3743-48db-bed1-7ba33650158a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:10:34.877064 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:34.874660 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a30818-8812-4f4d-a5c3-764d879deb52-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:34.877064 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:34.874686 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37d0ec61-3743-48db-bed1-7ba33650158a-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:34.877064 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:34.876038 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11a30818-8812-4f4d-a5c3-764d879deb52-kube-api-access-wc5kk" pod "11a30818-8812-4f4d-a5c3-764d879deb52" (UID: "11a30818-8812-4f4d-a5c3-764d879deb52"). InnerVolumeSpecName "kube-api-access-wc5kk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:34.877064 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:34.876440 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37d0ec61-3743-48db-bed1-7ba33650158a-kube-api-access-tzlvr" pod "37d0ec61-3743-48db-bed1-7ba33650158a" (UID: "37d0ec61-3743-48db-bed1-7ba33650158a"). InnerVolumeSpecName "kube-api-access-tzlvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:34.976838 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:34.976803 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc5kk\" (UniqueName: \"kubernetes.io/projected/11a30818-8812-4f4d-a5c3-764d879deb52-kube-api-access-wc5kk\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:34.976838 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:34.976834 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzlvr\" (UniqueName: \"kubernetes.io/projected/37d0ec61-3743-48db-bed1-7ba33650158a-kube-api-access-tzlvr\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:35.285863 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:35.285773 4363 generic.go:356] "Generic (PLEG): container finished" podID="37d0ec61-3743-48db-bed1-7ba33650158a" containerID="49bbd26bead5c5a52af3a09249aaea79894eb6e3b02aae6521f0dfd9cc440fa0" exitCode=0 Nov 01 16:10:35.285863 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:35.285818 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-lqtdt" Nov 01 16:10:35.285863 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:35.285852 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-lqtdt" event={"ID":"37d0ec61-3743-48db-bed1-7ba33650158a","Type":"ContainerDied","Data":"49bbd26bead5c5a52af3a09249aaea79894eb6e3b02aae6521f0dfd9cc440fa0"} Nov 01 16:10:35.286452 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:35.285906 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-lqtdt" event={"ID":"37d0ec61-3743-48db-bed1-7ba33650158a","Type":"ContainerDied","Data":"e792f86f3278e6ada6af18578ed125e169599c889cd4d71f9219493193bceba2"} Nov 01 16:10:35.286452 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:35.285931 4363 scope.go:122] "RemoveContainer" containerID="49bbd26bead5c5a52af3a09249aaea79894eb6e3b02aae6521f0dfd9cc440fa0" Nov 01 16:10:35.289278 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:35.289245 4363 generic.go:356] "Generic (PLEG): container finished" podID="11a30818-8812-4f4d-a5c3-764d879deb52" containerID="fdbf01d14088d583fcd34c8caa42a8299e9f25c8ac450e1ee22d0606f7330c09" exitCode=0 Nov 01 16:10:35.289410 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:35.289303 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-vrmjz" Nov 01 16:10:35.289548 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:35.289525 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-vrmjz" event={"ID":"11a30818-8812-4f4d-a5c3-764d879deb52","Type":"ContainerDied","Data":"fdbf01d14088d583fcd34c8caa42a8299e9f25c8ac450e1ee22d0606f7330c09"} Nov 01 16:10:35.289724 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:35.289694 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-vrmjz" event={"ID":"11a30818-8812-4f4d-a5c3-764d879deb52","Type":"ContainerDied","Data":"f8503eafa00333fd98ea805636d838b0807ae00386b075a263ab1d954823b9f9"} Nov 01 16:10:35.301145 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:35.301121 4363 scope.go:122] "RemoveContainer" containerID="49bbd26bead5c5a52af3a09249aaea79894eb6e3b02aae6521f0dfd9cc440fa0" Nov 01 16:10:35.301443 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:10:35.301423 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49bbd26bead5c5a52af3a09249aaea79894eb6e3b02aae6521f0dfd9cc440fa0\": container with ID starting with 49bbd26bead5c5a52af3a09249aaea79894eb6e3b02aae6521f0dfd9cc440fa0 not found: ID does not exist" containerID="49bbd26bead5c5a52af3a09249aaea79894eb6e3b02aae6521f0dfd9cc440fa0" Nov 01 16:10:35.301497 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:35.301452 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49bbd26bead5c5a52af3a09249aaea79894eb6e3b02aae6521f0dfd9cc440fa0"} err="failed to get container status \"49bbd26bead5c5a52af3a09249aaea79894eb6e3b02aae6521f0dfd9cc440fa0\": rpc error: code = NotFound desc = could not find container \"49bbd26bead5c5a52af3a09249aaea79894eb6e3b02aae6521f0dfd9cc440fa0\": container with ID starting with 49bbd26bead5c5a52af3a09249aaea79894eb6e3b02aae6521f0dfd9cc440fa0 not found: ID does not exist" Nov 01 16:10:35.301497 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:35.301476 4363 scope.go:122] "RemoveContainer" containerID="fdbf01d14088d583fcd34c8caa42a8299e9f25c8ac450e1ee22d0606f7330c09" Nov 01 16:10:35.311577 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:35.311415 4363 scope.go:122] "RemoveContainer" containerID="fdbf01d14088d583fcd34c8caa42a8299e9f25c8ac450e1ee22d0606f7330c09" Nov 01 16:10:35.311737 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:10:35.311717 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdbf01d14088d583fcd34c8caa42a8299e9f25c8ac450e1ee22d0606f7330c09\": container with ID starting with fdbf01d14088d583fcd34c8caa42a8299e9f25c8ac450e1ee22d0606f7330c09 not found: ID does not exist" containerID="fdbf01d14088d583fcd34c8caa42a8299e9f25c8ac450e1ee22d0606f7330c09" Nov 01 16:10:35.311786 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:35.311749 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdbf01d14088d583fcd34c8caa42a8299e9f25c8ac450e1ee22d0606f7330c09"} err="failed to get container status \"fdbf01d14088d583fcd34c8caa42a8299e9f25c8ac450e1ee22d0606f7330c09\": rpc error: code = NotFound desc = could not find container \"fdbf01d14088d583fcd34c8caa42a8299e9f25c8ac450e1ee22d0606f7330c09\": container with ID starting with fdbf01d14088d583fcd34c8caa42a8299e9f25c8ac450e1ee22d0606f7330c09 not found: ID does not exist" Nov 01 16:10:35.319849 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:35.319824 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-lqtdt"] Nov 01 16:10:35.324420 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:35.324397 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-76063d10-78f95fdf8c-lqtdt"] Nov 01 16:10:35.332938 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:35.332911 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-vrmjz"] Nov 01 16:10:35.337114 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:35.337090 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-76063d10-f66f6f4bf-vrmjz"] Nov 01 16:10:36.821890 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:36.818186 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11a30818-8812-4f4d-a5c3-764d879deb52" path="/var/lib/kubelet/pods/11a30818-8812-4f4d-a5c3-764d879deb52/volumes" Nov 01 16:10:36.821890 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:36.818695 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37d0ec61-3743-48db-bed1-7ba33650158a" path="/var/lib/kubelet/pods/37d0ec61-3743-48db-bed1-7ba33650158a/volumes" Nov 01 16:10:36.821890 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:36.821153 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m"] Nov 01 16:10:36.822507 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:36.822407 4363 cpu_manager.go:414] "containerMap: removing container" podUID="37d0ec61-3743-48db-bed1-7ba33650158a" containerName="payload-pre-processing" Nov 01 16:10:36.822507 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:36.822427 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37d0ec61-3743-48db-bed1-7ba33650158a" containerName="payload-pre-processing" Nov 01 16:10:36.822507 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:36.822460 4363 cpu_manager.go:414] "containerMap: removing container" podUID="11a30818-8812-4f4d-a5c3-764d879deb52" containerName="payload-processing" Nov 01 16:10:36.822507 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:36.822469 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11a30818-8812-4f4d-a5c3-764d879deb52" containerName="payload-processing" Nov 01 16:10:36.822713 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:36.822642 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="11a30818-8812-4f4d-a5c3-764d879deb52" containerName="payload-processing" Nov 01 16:10:36.822713 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:36.822676 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="37d0ec61-3743-48db-bed1-7ba33650158a" containerName="payload-pre-processing" Nov 01 16:10:36.830554 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:36.830528 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m"] Nov 01 16:10:36.830554 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:36.830640 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" Nov 01 16:10:36.839584 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:36.838085 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1c65e3-openshift-default-dockercfg-dccv4\"" Nov 01 16:10:36.924737 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:36.924699 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/145ad297-6cf1-4c70-ba0e-db5047a25f20-istiod-ca-cert\") pod \"e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m\" (UID: \"145ad297-6cf1-4c70-ba0e-db5047a25f20\") " pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" Nov 01 16:10:36.924906 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:36.924768 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7frg\" (UniqueName: \"kubernetes.io/projected/145ad297-6cf1-4c70-ba0e-db5047a25f20-kube-api-access-v7frg\") pod \"e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m\" (UID: \"145ad297-6cf1-4c70-ba0e-db5047a25f20\") " pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" Nov 01 16:10:36.924953 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:36.924915 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/145ad297-6cf1-4c70-ba0e-db5047a25f20-istio-podinfo\") pod \"e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m\" (UID: \"145ad297-6cf1-4c70-ba0e-db5047a25f20\") " pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" Nov 01 16:10:36.924953 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:36.924943 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/145ad297-6cf1-4c70-ba0e-db5047a25f20-istio-data\") pod \"e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m\" (UID: \"145ad297-6cf1-4c70-ba0e-db5047a25f20\") " pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" Nov 01 16:10:36.925049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:36.924973 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/145ad297-6cf1-4c70-ba0e-db5047a25f20-workload-socket\") pod \"e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m\" (UID: \"145ad297-6cf1-4c70-ba0e-db5047a25f20\") " pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" Nov 01 16:10:36.925049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:36.925030 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/145ad297-6cf1-4c70-ba0e-db5047a25f20-istio-token\") pod \"e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m\" (UID: \"145ad297-6cf1-4c70-ba0e-db5047a25f20\") " pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" Nov 01 16:10:36.925118 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:36.925055 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/145ad297-6cf1-4c70-ba0e-db5047a25f20-workload-certs\") pod \"e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m\" (UID: \"145ad297-6cf1-4c70-ba0e-db5047a25f20\") " pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" Nov 01 16:10:36.925118 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:36.925099 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/145ad297-6cf1-4c70-ba0e-db5047a25f20-credential-socket\") pod \"e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m\" (UID: \"145ad297-6cf1-4c70-ba0e-db5047a25f20\") " pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" Nov 01 16:10:36.925179 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:36.925136 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/145ad297-6cf1-4c70-ba0e-db5047a25f20-istio-envoy\") pod \"e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m\" (UID: \"145ad297-6cf1-4c70-ba0e-db5047a25f20\") " pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" Nov 01 16:10:36.932821 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:36.932790 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-58b0d1-5455d7568c-jh7m4"] Nov 01 16:10:36.940262 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:36.940228 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-58b0d1-786cf8cdb9-cbftv"] Nov 01 16:10:36.940626 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:36.940603 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-58b0d1-5455d7568c-jh7m4" Nov 01 16:10:36.946234 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:36.946210 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-58b0d1-5455d7568c-jh7m4"] Nov 01 16:10:36.946234 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:36.946237 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-58b0d1-786cf8cdb9-cbftv" Nov 01 16:10:36.948795 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:36.948546 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-58b0d1\"" Nov 01 16:10:36.953162 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:36.953119 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-58b0d1-dockercfg-dcdv9\"" Nov 01 16:10:37.033900 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:37.014484 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-58b0d1-786cf8cdb9-cbftv"] Nov 01 16:10:37.034450 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:37.034408 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7frg\" (UniqueName: \"kubernetes.io/projected/145ad297-6cf1-4c70-ba0e-db5047a25f20-kube-api-access-v7frg\") pod \"e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m\" (UID: \"145ad297-6cf1-4c70-ba0e-db5047a25f20\") " pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" Nov 01 16:10:37.034639 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:37.034617 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qltc\" (UniqueName: \"kubernetes.io/projected/4e4adc20-5cbc-485c-aaee-acca31664813-kube-api-access-9qltc\") pod \"payload-processing-e2e-api-b-58b0d1-786cf8cdb9-cbftv\" (UID: \"4e4adc20-5cbc-485c-aaee-acca31664813\") " pod="openshift-ingress/payload-processing-e2e-api-b-58b0d1-786cf8cdb9-cbftv" Nov 01 16:10:37.034731 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:37.034712 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/145ad297-6cf1-4c70-ba0e-db5047a25f20-istio-podinfo\") pod \"e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m\" (UID: \"145ad297-6cf1-4c70-ba0e-db5047a25f20\") " pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" Nov 01 16:10:37.034799 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:37.034764 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/145ad297-6cf1-4c70-ba0e-db5047a25f20-istio-data\") pod \"e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m\" (UID: \"145ad297-6cf1-4c70-ba0e-db5047a25f20\") " pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" Nov 01 16:10:37.034858 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:37.034804 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4adc20-5cbc-485c-aaee-acca31664813-plugins-config-volume\") pod \"payload-processing-e2e-api-b-58b0d1-786cf8cdb9-cbftv\" (UID: \"4e4adc20-5cbc-485c-aaee-acca31664813\") " pod="openshift-ingress/payload-processing-e2e-api-b-58b0d1-786cf8cdb9-cbftv" Nov 01 16:10:37.034858 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:37.034835 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/145ad297-6cf1-4c70-ba0e-db5047a25f20-workload-socket\") pod \"e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m\" (UID: \"145ad297-6cf1-4c70-ba0e-db5047a25f20\") " pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" Nov 01 16:10:37.034952 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:37.034879 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/145ad297-6cf1-4c70-ba0e-db5047a25f20-istio-token\") pod \"e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m\" (UID: \"145ad297-6cf1-4c70-ba0e-db5047a25f20\") " pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" Nov 01 16:10:37.034952 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:37.034906 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/145ad297-6cf1-4c70-ba0e-db5047a25f20-workload-certs\") pod \"e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m\" (UID: \"145ad297-6cf1-4c70-ba0e-db5047a25f20\") " pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" Nov 01 16:10:37.035064 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:37.034971 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/145ad297-6cf1-4c70-ba0e-db5047a25f20-credential-socket\") pod \"e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m\" (UID: \"145ad297-6cf1-4c70-ba0e-db5047a25f20\") " pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" Nov 01 16:10:37.035064 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:37.035036 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/145ad297-6cf1-4c70-ba0e-db5047a25f20-istio-envoy\") pod \"e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m\" (UID: \"145ad297-6cf1-4c70-ba0e-db5047a25f20\") " pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" Nov 01 16:10:37.084007 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:37.078259 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/145ad297-6cf1-4c70-ba0e-db5047a25f20-istiod-ca-cert\") pod \"e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m\" (UID: \"145ad297-6cf1-4c70-ba0e-db5047a25f20\") " pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" Nov 01 16:10:37.084007 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:37.078316 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4ccl\" (UniqueName: \"kubernetes.io/projected/c20f4fe4-124a-446f-89b5-e526085a8cea-kube-api-access-b4ccl\") pod \"payload-pre-processing-e2e-api-b-58b0d1-5455d7568c-jh7m4\" (UID: \"c20f4fe4-124a-446f-89b5-e526085a8cea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-58b0d1-5455d7568c-jh7m4" Nov 01 16:10:37.084007 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:37.078357 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c20f4fe4-124a-446f-89b5-e526085a8cea-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-58b0d1-5455d7568c-jh7m4\" (UID: \"c20f4fe4-124a-446f-89b5-e526085a8cea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-58b0d1-5455d7568c-jh7m4" Nov 01 16:10:37.084007 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:37.079378 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/145ad297-6cf1-4c70-ba0e-db5047a25f20-istiod-ca-cert\") pod \"e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m\" (UID: \"145ad297-6cf1-4c70-ba0e-db5047a25f20\") " pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" Nov 01 16:10:37.220041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:37.180429 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/145ad297-6cf1-4c70-ba0e-db5047a25f20-istio-data\") pod \"e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m\" (UID: \"145ad297-6cf1-4c70-ba0e-db5047a25f20\") " pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" Nov 01 16:10:37.220041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:37.216042 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/145ad297-6cf1-4c70-ba0e-db5047a25f20-workload-socket\") pod \"e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m\" (UID: \"145ad297-6cf1-4c70-ba0e-db5047a25f20\") " pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" Nov 01 16:10:37.220041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:37.216313 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/145ad297-6cf1-4c70-ba0e-db5047a25f20-workload-certs\") pod \"e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m\" (UID: \"145ad297-6cf1-4c70-ba0e-db5047a25f20\") " pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" Nov 01 16:10:37.220041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:37.216550 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/145ad297-6cf1-4c70-ba0e-db5047a25f20-credential-socket\") pod \"e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m\" (UID: \"145ad297-6cf1-4c70-ba0e-db5047a25f20\") " pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" Nov 01 16:10:37.260086 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:37.234341 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qltc\" (UniqueName: \"kubernetes.io/projected/4e4adc20-5cbc-485c-aaee-acca31664813-kube-api-access-9qltc\") pod \"payload-processing-e2e-api-b-58b0d1-786cf8cdb9-cbftv\" (UID: \"4e4adc20-5cbc-485c-aaee-acca31664813\") " pod="openshift-ingress/payload-processing-e2e-api-b-58b0d1-786cf8cdb9-cbftv" Nov 01 16:10:37.260086 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:37.234492 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4adc20-5cbc-485c-aaee-acca31664813-plugins-config-volume\") pod \"payload-processing-e2e-api-b-58b0d1-786cf8cdb9-cbftv\" (UID: \"4e4adc20-5cbc-485c-aaee-acca31664813\") " pod="openshift-ingress/payload-processing-e2e-api-b-58b0d1-786cf8cdb9-cbftv" Nov 01 16:10:37.260086 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:37.234715 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4ccl\" (UniqueName: \"kubernetes.io/projected/c20f4fe4-124a-446f-89b5-e526085a8cea-kube-api-access-b4ccl\") pod \"payload-pre-processing-e2e-api-b-58b0d1-5455d7568c-jh7m4\" (UID: \"c20f4fe4-124a-446f-89b5-e526085a8cea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-58b0d1-5455d7568c-jh7m4" Nov 01 16:10:37.260086 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:37.234748 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c20f4fe4-124a-446f-89b5-e526085a8cea-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-58b0d1-5455d7568c-jh7m4\" (UID: \"c20f4fe4-124a-446f-89b5-e526085a8cea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-58b0d1-5455d7568c-jh7m4" Nov 01 16:10:37.260086 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:37.235728 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c20f4fe4-124a-446f-89b5-e526085a8cea-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-58b0d1-5455d7568c-jh7m4\" (UID: \"c20f4fe4-124a-446f-89b5-e526085a8cea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-58b0d1-5455d7568c-jh7m4" Nov 01 16:10:37.260086 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:37.235878 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4adc20-5cbc-485c-aaee-acca31664813-plugins-config-volume\") pod \"payload-processing-e2e-api-b-58b0d1-786cf8cdb9-cbftv\" (UID: \"4e4adc20-5cbc-485c-aaee-acca31664813\") " pod="openshift-ingress/payload-processing-e2e-api-b-58b0d1-786cf8cdb9-cbftv" Nov 01 16:10:37.475029 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:37.466507 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7frg\" (UniqueName: \"kubernetes.io/projected/145ad297-6cf1-4c70-ba0e-db5047a25f20-kube-api-access-v7frg\") pod \"e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m\" (UID: \"145ad297-6cf1-4c70-ba0e-db5047a25f20\") " pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" Nov 01 16:10:37.475029 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:37.472566 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/145ad297-6cf1-4c70-ba0e-db5047a25f20-istio-envoy\") pod \"e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m\" (UID: \"145ad297-6cf1-4c70-ba0e-db5047a25f20\") " pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" Nov 01 16:10:37.478081 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:37.478033 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/145ad297-6cf1-4c70-ba0e-db5047a25f20-istio-token\") pod \"e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m\" (UID: \"145ad297-6cf1-4c70-ba0e-db5047a25f20\") " pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" Nov 01 16:10:37.478401 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:37.478379 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/145ad297-6cf1-4c70-ba0e-db5047a25f20-istio-podinfo\") pod \"e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m\" (UID: \"145ad297-6cf1-4c70-ba0e-db5047a25f20\") " pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" Nov 01 16:10:37.478890 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:37.478870 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qltc\" (UniqueName: \"kubernetes.io/projected/4e4adc20-5cbc-485c-aaee-acca31664813-kube-api-access-9qltc\") pod \"payload-processing-e2e-api-b-58b0d1-786cf8cdb9-cbftv\" (UID: \"4e4adc20-5cbc-485c-aaee-acca31664813\") " pod="openshift-ingress/payload-processing-e2e-api-b-58b0d1-786cf8cdb9-cbftv" Nov 01 16:10:37.482564 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:37.482541 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4ccl\" (UniqueName: \"kubernetes.io/projected/c20f4fe4-124a-446f-89b5-e526085a8cea-kube-api-access-b4ccl\") pod \"payload-pre-processing-e2e-api-b-58b0d1-5455d7568c-jh7m4\" (UID: \"c20f4fe4-124a-446f-89b5-e526085a8cea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-58b0d1-5455d7568c-jh7m4" Nov 01 16:10:37.585049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:37.584989 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-58b0d1-5455d7568c-jh7m4" Nov 01 16:10:37.754382 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:37.754287 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" Nov 01 16:10:37.768933 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:37.768897 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-58b0d1-786cf8cdb9-cbftv" Nov 01 16:10:37.893235 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:10:37.893139 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc20f4fe4_124a_446f_89b5_e526085a8cea.slice/crio-5753e2d01b4fe0f5718a2fcb32c303a78eec7886685742bda122ad2b37bd8f00 WatchSource:0}: Error finding container 5753e2d01b4fe0f5718a2fcb32c303a78eec7886685742bda122ad2b37bd8f00: Status 404 returned error can't find the container with id 5753e2d01b4fe0f5718a2fcb32c303a78eec7886685742bda122ad2b37bd8f00 Nov 01 16:10:37.894470 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:37.894333 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-58b0d1-5455d7568c-jh7m4"] Nov 01 16:10:38.128345 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:10:38.128308 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod145ad297_6cf1_4c70_ba0e_db5047a25f20.slice/crio-23446185d65106a65413c3b9102f936ba20845727e78b56a655e6d733e5794b2 WatchSource:0}: Error finding container 23446185d65106a65413c3b9102f936ba20845727e78b56a655e6d733e5794b2: Status 404 returned error can't find the container with id 23446185d65106a65413c3b9102f936ba20845727e78b56a655e6d733e5794b2 Nov 01 16:10:38.129421 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:38.129270 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m"] Nov 01 16:10:38.138218 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:38.138186 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:10:38.138313 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:38.138255 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:10:38.138313 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:38.138287 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:10:38.307478 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:10:38.307444 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e4adc20_5cbc_485c_aaee_acca31664813.slice/crio-996228e4a28f6c1b94ad13193f86d949c9e49a8211d06739e846ac21e2aa4841 WatchSource:0}: Error finding container 996228e4a28f6c1b94ad13193f86d949c9e49a8211d06739e846ac21e2aa4841: Status 404 returned error can't find the container with id 996228e4a28f6c1b94ad13193f86d949c9e49a8211d06739e846ac21e2aa4841 Nov 01 16:10:38.313055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:38.309400 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-58b0d1-786cf8cdb9-cbftv"] Nov 01 16:10:38.488539 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:38.488483 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-58b0d1-786cf8cdb9-cbftv" event={"ID":"4e4adc20-5cbc-485c-aaee-acca31664813","Type":"ContainerStarted","Data":"9eef98366200810482cd73cc8152600f0fb7e132205dfcbe5c37a2413a45623e"} Nov 01 16:10:38.488736 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:38.488596 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-58b0d1-786cf8cdb9-cbftv" event={"ID":"4e4adc20-5cbc-485c-aaee-acca31664813","Type":"ContainerStarted","Data":"996228e4a28f6c1b94ad13193f86d949c9e49a8211d06739e846ac21e2aa4841"} Nov 01 16:10:38.488865 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:38.488831 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-58b0d1-786cf8cdb9-cbftv" Nov 01 16:10:38.490676 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:38.490647 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" event={"ID":"145ad297-6cf1-4c70-ba0e-db5047a25f20","Type":"ContainerStarted","Data":"2afc294017df9e90a1e8837a1cb74a7ff430675a829c41007f669d7b80469874"} Nov 01 16:10:38.490777 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:38.490697 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" event={"ID":"145ad297-6cf1-4c70-ba0e-db5047a25f20","Type":"ContainerStarted","Data":"23446185d65106a65413c3b9102f936ba20845727e78b56a655e6d733e5794b2"} Nov 01 16:10:38.492833 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:38.492801 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-58b0d1-5455d7568c-jh7m4" event={"ID":"c20f4fe4-124a-446f-89b5-e526085a8cea","Type":"ContainerStarted","Data":"4c12d3fb4ad3ff5de5e328b3345b40fe55c177923b08f016191b276813ab6de3"} Nov 01 16:10:38.492963 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:38.492850 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-58b0d1-5455d7568c-jh7m4" event={"ID":"c20f4fe4-124a-446f-89b5-e526085a8cea","Type":"ContainerStarted","Data":"5753e2d01b4fe0f5718a2fcb32c303a78eec7886685742bda122ad2b37bd8f00"} Nov 01 16:10:38.493229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:38.493110 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-58b0d1-5455d7568c-jh7m4" Nov 01 16:10:38.509712 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:38.509665 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-58b0d1-786cf8cdb9-cbftv" podStartSLOduration=2.509652309 podStartE2EDuration="2.509652309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:10:38.505614403 +0000 UTC m=+1371.837352054" watchObservedRunningTime="2026-11-01 16:10:38.509652309 +0000 UTC m=+1371.841389960" Nov 01 16:10:38.527233 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:38.527179 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-58b0d1-5455d7568c-jh7m4" podStartSLOduration=2.527164689 podStartE2EDuration="2.527164689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:10:38.522994669 +0000 UTC m=+1371.854732322" watchObservedRunningTime="2026-11-01 16:10:38.527164689 +0000 UTC m=+1371.858902340" Nov 01 16:10:38.556796 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:38.556744 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" podStartSLOduration=2.556728804 podStartE2EDuration="2.556728804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:10:38.54932354 +0000 UTC m=+1371.881061202" watchObservedRunningTime="2026-11-01 16:10:38.556728804 +0000 UTC m=+1371.888466454" Nov 01 16:10:38.756327 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:38.756103 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" Nov 01 16:10:39.111448 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:39.111419 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-375350-68b888b95-4mx6d" Nov 01 16:10:39.111883 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:39.111496 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-375350-b98cbd657-76rtq" Nov 01 16:10:39.760916 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:39.760885 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" Nov 01 16:10:40.559761 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:40.559723 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" Nov 01 16:10:40.561523 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:40.561500 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" Nov 01 16:10:42.019873 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:42.019837 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1c65e3-54b4ddd7d7-lbrcc"] Nov 01 16:10:42.024976 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:42.024951 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1c65e3-54b4ddd7d7-lbrcc" Nov 01 16:10:42.031208 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:42.031180 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1c65e3\"" Nov 01 16:10:42.031859 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:42.031841 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1c65e3-dockercfg-mgqtx\"" Nov 01 16:10:42.034677 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:42.034649 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1c65e3-54b4ddd7d7-lbrcc"] Nov 01 16:10:42.061791 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:42.061741 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1c65e3-77686b98cf-wzm9h"] Nov 01 16:10:42.069289 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:42.069256 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c65e3-77686b98cf-wzm9h" Nov 01 16:10:42.071129 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:42.071101 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1c65e3-77686b98cf-wzm9h"] Nov 01 16:10:42.132580 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:42.132346 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5c6b75-7478fffbb9-st6hq"] Nov 01 16:10:42.134602 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:42.134413 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6b75-7478fffbb9-st6hq" podUID="a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25" containerName="maas-api" containerID="cri-o://1c04ac4afba3883bed375df88bc96613170c4300de681f166cac80accfc048d6" gracePeriod=30 Nov 01 16:10:42.155527 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:42.154428 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85d2d15c-3c9a-4b88-89cb-5f5c06979aa9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1c65e3-54b4ddd7d7-lbrcc\" (UID: \"85d2d15c-3c9a-4b88-89cb-5f5c06979aa9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1c65e3-54b4ddd7d7-lbrcc" Nov 01 16:10:42.155527 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:42.154681 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kgn2\" (UniqueName: \"kubernetes.io/projected/85d2d15c-3c9a-4b88-89cb-5f5c06979aa9-kube-api-access-7kgn2\") pod \"payload-processing-e2e-inf-a-1c65e3-54b4ddd7d7-lbrcc\" (UID: \"85d2d15c-3c9a-4b88-89cb-5f5c06979aa9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1c65e3-54b4ddd7d7-lbrcc" Nov 01 16:10:42.155527 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:42.154737 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06fcb8d1-bff1-4182-92d2-98a57314f6fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1c65e3-77686b98cf-wzm9h\" (UID: \"06fcb8d1-bff1-4182-92d2-98a57314f6fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c65e3-77686b98cf-wzm9h" Nov 01 16:10:42.155527 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:42.154837 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjplw\" (UniqueName: \"kubernetes.io/projected/06fcb8d1-bff1-4182-92d2-98a57314f6fd-kube-api-access-hjplw\") pod \"payload-pre-processing-e2e-inf-a-1c65e3-77686b98cf-wzm9h\" (UID: \"06fcb8d1-bff1-4182-92d2-98a57314f6fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c65e3-77686b98cf-wzm9h" Nov 01 16:10:42.257908 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:42.257878 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06fcb8d1-bff1-4182-92d2-98a57314f6fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1c65e3-77686b98cf-wzm9h\" (UID: \"06fcb8d1-bff1-4182-92d2-98a57314f6fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c65e3-77686b98cf-wzm9h" Nov 01 16:10:42.258045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:42.257966 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjplw\" (UniqueName: \"kubernetes.io/projected/06fcb8d1-bff1-4182-92d2-98a57314f6fd-kube-api-access-hjplw\") pod \"payload-pre-processing-e2e-inf-a-1c65e3-77686b98cf-wzm9h\" (UID: \"06fcb8d1-bff1-4182-92d2-98a57314f6fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c65e3-77686b98cf-wzm9h" Nov 01 16:10:42.258121 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:42.258108 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85d2d15c-3c9a-4b88-89cb-5f5c06979aa9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1c65e3-54b4ddd7d7-lbrcc\" (UID: \"85d2d15c-3c9a-4b88-89cb-5f5c06979aa9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1c65e3-54b4ddd7d7-lbrcc" Nov 01 16:10:42.258259 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:42.258248 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kgn2\" (UniqueName: \"kubernetes.io/projected/85d2d15c-3c9a-4b88-89cb-5f5c06979aa9-kube-api-access-7kgn2\") pod \"payload-processing-e2e-inf-a-1c65e3-54b4ddd7d7-lbrcc\" (UID: \"85d2d15c-3c9a-4b88-89cb-5f5c06979aa9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1c65e3-54b4ddd7d7-lbrcc" Nov 01 16:10:42.258584 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:42.258565 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06fcb8d1-bff1-4182-92d2-98a57314f6fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1c65e3-77686b98cf-wzm9h\" (UID: \"06fcb8d1-bff1-4182-92d2-98a57314f6fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c65e3-77686b98cf-wzm9h" Nov 01 16:10:42.258951 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:42.258927 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85d2d15c-3c9a-4b88-89cb-5f5c06979aa9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1c65e3-54b4ddd7d7-lbrcc\" (UID: \"85d2d15c-3c9a-4b88-89cb-5f5c06979aa9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1c65e3-54b4ddd7d7-lbrcc" Nov 01 16:10:42.265366 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:42.265337 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjplw\" (UniqueName: \"kubernetes.io/projected/06fcb8d1-bff1-4182-92d2-98a57314f6fd-kube-api-access-hjplw\") pod \"payload-pre-processing-e2e-inf-a-1c65e3-77686b98cf-wzm9h\" (UID: \"06fcb8d1-bff1-4182-92d2-98a57314f6fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c65e3-77686b98cf-wzm9h" Nov 01 16:10:42.265604 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:42.265583 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kgn2\" (UniqueName: \"kubernetes.io/projected/85d2d15c-3c9a-4b88-89cb-5f5c06979aa9-kube-api-access-7kgn2\") pod \"payload-processing-e2e-inf-a-1c65e3-54b4ddd7d7-lbrcc\" (UID: \"85d2d15c-3c9a-4b88-89cb-5f5c06979aa9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1c65e3-54b4ddd7d7-lbrcc" Nov 01 16:10:42.414044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:42.406528 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c65e3-77686b98cf-wzm9h" Nov 01 16:10:42.428063 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:42.420691 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1c65e3-54b4ddd7d7-lbrcc" Nov 01 16:10:42.937039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:42.934790 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-375350-76f869b4d8-ljvvj"] Nov 01 16:10:42.938873 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:42.937973 4363 generic.go:356] "Generic (PLEG): container finished" podID="a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25" containerID="1c04ac4afba3883bed375df88bc96613170c4300de681f166cac80accfc048d6" exitCode=0 Nov 01 16:10:42.949888 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:42.949856 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6b75-7478fffbb9-st6hq" Nov 01 16:10:42.964133 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:42.953371 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6b75-7478fffbb9-st6hq" event={"ID":"a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25","Type":"ContainerDied","Data":"1c04ac4afba3883bed375df88bc96613170c4300de681f166cac80accfc048d6"} Nov 01 16:10:42.964133 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:42.953448 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-375350-76f869b4d8-ljvvj"] Nov 01 16:10:42.964133 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:42.953490 4363 scope.go:122] "RemoveContainer" containerID="1c04ac4afba3883bed375df88bc96613170c4300de681f166cac80accfc048d6" Nov 01 16:10:42.964133 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:42.953722 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-375350-76f869b4d8-ljvvj" Nov 01 16:10:42.964642 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:42.964363 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-375350-serving-cert\"" Nov 01 16:10:43.113636 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:43.111618 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25-maas-api-tls\") pod \"a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25\" (UID: \"a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25\") " Nov 01 16:10:43.113636 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:43.111726 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25-kube-api-access-ccxhl\" (UniqueName: \"kubernetes.io/projected/a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25-kube-api-access-ccxhl\") pod \"a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25\" (UID: \"a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25\") " Nov 01 16:10:43.113636 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:43.111900 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25-openshift-service-ca\") pod \"a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25\" (UID: \"a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25\") " Nov 01 16:10:43.259063 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:43.225773 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25-openshift-service-ca" pod "a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25" (UID: "a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:10:43.259063 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:43.258289 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25-metrics-certs\") pod \"a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25\" (UID: \"a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25\") " Nov 01 16:10:43.259063 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:43.258810 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gns29\" (UniqueName: \"kubernetes.io/projected/cf5d34ea-8ba0-44e6-86f8-85e1dfc59412-kube-api-access-gns29\") pod \"maas-api-e2e-inf-b-375350-76f869b4d8-ljvvj\" (UID: \"cf5d34ea-8ba0-44e6-86f8-85e1dfc59412\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-375350-76f869b4d8-ljvvj" Nov 01 16:10:43.259063 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:43.259005 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf5d34ea-8ba0-44e6-86f8-85e1dfc59412-metrics-certs\") pod \"maas-api-e2e-inf-b-375350-76f869b4d8-ljvvj\" (UID: \"cf5d34ea-8ba0-44e6-86f8-85e1dfc59412\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-375350-76f869b4d8-ljvvj" Nov 01 16:10:43.277219 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:43.276241 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf5d34ea-8ba0-44e6-86f8-85e1dfc59412-openshift-service-ca\") pod \"maas-api-e2e-inf-b-375350-76f869b4d8-ljvvj\" (UID: \"cf5d34ea-8ba0-44e6-86f8-85e1dfc59412\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-375350-76f869b4d8-ljvvj" Nov 01 16:10:43.277219 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:43.276403 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf5d34ea-8ba0-44e6-86f8-85e1dfc59412-maas-api-tls\") pod \"maas-api-e2e-inf-b-375350-76f869b4d8-ljvvj\" (UID: \"cf5d34ea-8ba0-44e6-86f8-85e1dfc59412\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-375350-76f869b4d8-ljvvj" Nov 01 16:10:43.277219 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:43.276586 4363 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25-openshift-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:43.287787 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:43.285763 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25-metrics-certs" pod "a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25" (UID: "a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:10:43.290733 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:43.289321 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25-maas-api-tls" pod "a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25" (UID: "a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:10:43.296998 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:43.296237 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1c65e3-54b4ddd7d7-lbrcc"] Nov 01 16:10:43.305294 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:43.305205 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25-kube-api-access-ccxhl" pod "a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25" (UID: "a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25"). InnerVolumeSpecName "kube-api-access-ccxhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:43.378277 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:43.378239 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gns29\" (UniqueName: \"kubernetes.io/projected/cf5d34ea-8ba0-44e6-86f8-85e1dfc59412-kube-api-access-gns29\") pod \"maas-api-e2e-inf-b-375350-76f869b4d8-ljvvj\" (UID: \"cf5d34ea-8ba0-44e6-86f8-85e1dfc59412\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-375350-76f869b4d8-ljvvj" Nov 01 16:10:43.378591 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:43.378426 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf5d34ea-8ba0-44e6-86f8-85e1dfc59412-metrics-certs\") pod \"maas-api-e2e-inf-b-375350-76f869b4d8-ljvvj\" (UID: \"cf5d34ea-8ba0-44e6-86f8-85e1dfc59412\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-375350-76f869b4d8-ljvvj" Nov 01 16:10:43.378591 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:43.378545 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf5d34ea-8ba0-44e6-86f8-85e1dfc59412-openshift-service-ca\") pod \"maas-api-e2e-inf-b-375350-76f869b4d8-ljvvj\" (UID: \"cf5d34ea-8ba0-44e6-86f8-85e1dfc59412\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-375350-76f869b4d8-ljvvj" Nov 01 16:10:43.378746 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:43.378665 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf5d34ea-8ba0-44e6-86f8-85e1dfc59412-maas-api-tls\") pod \"maas-api-e2e-inf-b-375350-76f869b4d8-ljvvj\" (UID: \"cf5d34ea-8ba0-44e6-86f8-85e1dfc59412\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-375350-76f869b4d8-ljvvj" Nov 01 16:10:43.378857 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:43.378840 4363 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25-metrics-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:43.378938 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:43.378863 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25-maas-api-tls\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:43.378938 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:43.378882 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccxhl\" (UniqueName: \"kubernetes.io/projected/a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25-kube-api-access-ccxhl\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:43.380372 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:43.380347 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf5d34ea-8ba0-44e6-86f8-85e1dfc59412-openshift-service-ca\") pod \"maas-api-e2e-inf-b-375350-76f869b4d8-ljvvj\" (UID: \"cf5d34ea-8ba0-44e6-86f8-85e1dfc59412\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-375350-76f869b4d8-ljvvj" Nov 01 16:10:43.382441 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:43.382416 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf5d34ea-8ba0-44e6-86f8-85e1dfc59412-maas-api-tls\") pod \"maas-api-e2e-inf-b-375350-76f869b4d8-ljvvj\" (UID: \"cf5d34ea-8ba0-44e6-86f8-85e1dfc59412\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-375350-76f869b4d8-ljvvj" Nov 01 16:10:43.382582 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:43.382561 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf5d34ea-8ba0-44e6-86f8-85e1dfc59412-metrics-certs\") pod \"maas-api-e2e-inf-b-375350-76f869b4d8-ljvvj\" (UID: \"cf5d34ea-8ba0-44e6-86f8-85e1dfc59412\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-375350-76f869b4d8-ljvvj" Nov 01 16:10:43.388399 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:43.388377 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gns29\" (UniqueName: \"kubernetes.io/projected/cf5d34ea-8ba0-44e6-86f8-85e1dfc59412-kube-api-access-gns29\") pod \"maas-api-e2e-inf-b-375350-76f869b4d8-ljvvj\" (UID: \"cf5d34ea-8ba0-44e6-86f8-85e1dfc59412\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-375350-76f869b4d8-ljvvj" Nov 01 16:10:43.446764 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:43.446727 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1c65e3-77686b98cf-wzm9h"] Nov 01 16:10:43.446899 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:10:43.446737 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06fcb8d1_bff1_4182_92d2_98a57314f6fd.slice/crio-c52624cd13d965b8ea0121410aa8e16bc01dc724392f3b2fcb7a772fbb62d745 WatchSource:0}: Error finding container c52624cd13d965b8ea0121410aa8e16bc01dc724392f3b2fcb7a772fbb62d745: Status 404 returned error can't find the container with id c52624cd13d965b8ea0121410aa8e16bc01dc724392f3b2fcb7a772fbb62d745 Nov 01 16:10:43.584947 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:43.582360 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-375350-76f869b4d8-ljvvj" Nov 01 16:10:44.079556 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:10:44.079518 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf5d34ea_8ba0_44e6_86f8_85e1dfc59412.slice/crio-3dad26764e931b9153e9a6b4cf212181b83c5b5055d04272db0de8d086ab6d4c WatchSource:0}: Error finding container 3dad26764e931b9153e9a6b4cf212181b83c5b5055d04272db0de8d086ab6d4c: Status 404 returned error can't find the container with id 3dad26764e931b9153e9a6b4cf212181b83c5b5055d04272db0de8d086ab6d4c Nov 01 16:10:44.089785 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:44.089755 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6b75-7478fffbb9-st6hq" event={"ID":"a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25","Type":"ContainerDied","Data":"6352fdf4fd6b691a239668d1d0cf1b31f381c89d3b791dc57009e386879535de"} Nov 01 16:10:44.095156 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:44.095119 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6b75-7478fffbb9-st6hq" Nov 01 16:10:44.101101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:44.100095 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1c65e3-54b4ddd7d7-lbrcc" event={"ID":"85d2d15c-3c9a-4b88-89cb-5f5c06979aa9","Type":"ContainerStarted","Data":"ebca0458ff258072c27d4077a4ca73acb1ad6360cd26932e70e502456fd38405"} Nov 01 16:10:44.101101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:44.100160 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1c65e3-54b4ddd7d7-lbrcc" event={"ID":"85d2d15c-3c9a-4b88-89cb-5f5c06979aa9","Type":"ContainerStarted","Data":"271185efbb8a67a17bedc2623f53be76ada913267b58c0fdb746c7221d5ffe02"} Nov 01 16:10:44.101101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:44.101054 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1c65e3-54b4ddd7d7-lbrcc" Nov 01 16:10:44.104348 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:44.103435 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c65e3-77686b98cf-wzm9h" event={"ID":"06fcb8d1-bff1-4182-92d2-98a57314f6fd","Type":"ContainerStarted","Data":"e2f2704741032bd934dedbe00593467488a3e6e04e845624b1d3ee7d22b461f3"} Nov 01 16:10:44.104348 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:44.103484 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c65e3-77686b98cf-wzm9h" event={"ID":"06fcb8d1-bff1-4182-92d2-98a57314f6fd","Type":"ContainerStarted","Data":"c52624cd13d965b8ea0121410aa8e16bc01dc724392f3b2fcb7a772fbb62d745"} Nov 01 16:10:44.104348 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:44.104295 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c65e3-77686b98cf-wzm9h" Nov 01 16:10:44.121713 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:44.121675 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-375350-76f869b4d8-ljvvj"] Nov 01 16:10:44.135135 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:44.135090 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c65e3-77686b98cf-wzm9h" podStartSLOduration=2.135070393 podStartE2EDuration="2.135070393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:10:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:10:44.130876809 +0000 UTC m=+1377.462614459" watchObservedRunningTime="2026-11-01 16:10:44.135070393 +0000 UTC m=+1377.466808044" Nov 01 16:10:44.152871 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:44.152822 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-1c65e3-54b4ddd7d7-lbrcc" podStartSLOduration=3.152809316 podStartE2EDuration="3.152809316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:10:44.146541143 +0000 UTC m=+1377.478278817" watchObservedRunningTime="2026-11-01 16:10:44.152809316 +0000 UTC m=+1377.484546965" Nov 01 16:10:44.162453 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:44.162424 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5c6b75-7478fffbb9-st6hq"] Nov 01 16:10:44.167785 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:44.167764 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5c6b75-7478fffbb9-st6hq"] Nov 01 16:10:44.806140 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:44.806062 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25" path="/var/lib/kubelet/pods/a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25/volumes" Nov 01 16:10:45.116314 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:45.116275 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-375350-76f869b4d8-ljvvj" event={"ID":"cf5d34ea-8ba0-44e6-86f8-85e1dfc59412","Type":"ContainerStarted","Data":"369ab74a0e91a1d649356ffb030da0fdef8efde8cbb6a8f906c77e39c988f2ac"} Nov 01 16:10:45.116517 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:45.116355 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-375350-76f869b4d8-ljvvj" event={"ID":"cf5d34ea-8ba0-44e6-86f8-85e1dfc59412","Type":"ContainerStarted","Data":"3dad26764e931b9153e9a6b4cf212181b83c5b5055d04272db0de8d086ab6d4c"} Nov 01 16:10:45.116517 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:45.116388 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-375350-76f869b4d8-ljvvj" Nov 01 16:10:45.134087 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:45.134005 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-375350-76f869b4d8-ljvvj" podStartSLOduration=2.703166274 podStartE2EDuration="3.133990468s" totalImagesPullingTime="430.824194ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:10:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:10:45.131533482 +0000 UTC m=+1378.463271133" watchObservedRunningTime="2026-11-01 16:10:45.133990468 +0000 UTC m=+1378.465728118" Nov 01 16:10:45.830611 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:45.830573 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-2zh4r"] Nov 01 16:10:45.830854 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:45.830815 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-2zh4r" podUID="21e3bc0d-84f1-4ae1-baaa-97509d961b89" containerName="maas-api" containerID="cri-o://b85a4b4298ecdae0a78c4771a0e57de23e4a8040b699d19fc80295f83ce9823a" gracePeriod=30 Nov 01 16:10:46.557166 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:46.556821 4363 generic.go:356] "Generic (PLEG): container finished" podID="21e3bc0d-84f1-4ae1-baaa-97509d961b89" containerID="b85a4b4298ecdae0a78c4771a0e57de23e4a8040b699d19fc80295f83ce9823a" exitCode=0 Nov 01 16:10:46.558111 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:46.557130 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-2zh4r" event={"ID":"21e3bc0d-84f1-4ae1-baaa-97509d961b89","Type":"ContainerDied","Data":"b85a4b4298ecdae0a78c4771a0e57de23e4a8040b699d19fc80295f83ce9823a"} Nov 01 16:10:46.570087 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:46.569622 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-2zh4r" Nov 01 16:10:46.723041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:46.709727 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21e3bc0d-84f1-4ae1-baaa-97509d961b89-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e3bc0d-84f1-4ae1-baaa-97509d961b89-openshift-service-ca\") pod \"21e3bc0d-84f1-4ae1-baaa-97509d961b89\" (UID: \"21e3bc0d-84f1-4ae1-baaa-97509d961b89\") " Nov 01 16:10:46.723041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:46.709844 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21e3bc0d-84f1-4ae1-baaa-97509d961b89-metrics-certs\" (UniqueName: \"kubernetes.io/secret/21e3bc0d-84f1-4ae1-baaa-97509d961b89-metrics-certs\") pod \"21e3bc0d-84f1-4ae1-baaa-97509d961b89\" (UID: \"21e3bc0d-84f1-4ae1-baaa-97509d961b89\") " Nov 01 16:10:46.723041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:46.709899 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21e3bc0d-84f1-4ae1-baaa-97509d961b89-kube-api-access-8h9md\" (UniqueName: \"kubernetes.io/projected/21e3bc0d-84f1-4ae1-baaa-97509d961b89-kube-api-access-8h9md\") pod \"21e3bc0d-84f1-4ae1-baaa-97509d961b89\" (UID: \"21e3bc0d-84f1-4ae1-baaa-97509d961b89\") " Nov 01 16:10:46.723041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:46.710008 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21e3bc0d-84f1-4ae1-baaa-97509d961b89-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21e3bc0d-84f1-4ae1-baaa-97509d961b89-maas-api-tls\") pod \"21e3bc0d-84f1-4ae1-baaa-97509d961b89\" (UID: \"21e3bc0d-84f1-4ae1-baaa-97509d961b89\") " Nov 01 16:10:46.745978 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:46.745901 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21e3bc0d-84f1-4ae1-baaa-97509d961b89-openshift-service-ca" pod "21e3bc0d-84f1-4ae1-baaa-97509d961b89" (UID: "21e3bc0d-84f1-4ae1-baaa-97509d961b89"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:10:46.767771 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:46.757344 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21e3bc0d-84f1-4ae1-baaa-97509d961b89-metrics-certs" pod "21e3bc0d-84f1-4ae1-baaa-97509d961b89" (UID: "21e3bc0d-84f1-4ae1-baaa-97509d961b89"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:10:46.773455 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:46.773381 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21e3bc0d-84f1-4ae1-baaa-97509d961b89-maas-api-tls" pod "21e3bc0d-84f1-4ae1-baaa-97509d961b89" (UID: "21e3bc0d-84f1-4ae1-baaa-97509d961b89"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:10:46.777061 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:46.776999 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21e3bc0d-84f1-4ae1-baaa-97509d961b89-kube-api-access-8h9md" pod "21e3bc0d-84f1-4ae1-baaa-97509d961b89" (UID: "21e3bc0d-84f1-4ae1-baaa-97509d961b89"). InnerVolumeSpecName "kube-api-access-8h9md". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:46.832818 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:46.832790 4363 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e3bc0d-84f1-4ae1-baaa-97509d961b89-openshift-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:46.832818 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:46.832815 4363 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21e3bc0d-84f1-4ae1-baaa-97509d961b89-metrics-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:46.832973 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:46.832826 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h9md\" (UniqueName: \"kubernetes.io/projected/21e3bc0d-84f1-4ae1-baaa-97509d961b89-kube-api-access-8h9md\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:46.832973 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:46.832836 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21e3bc0d-84f1-4ae1-baaa-97509d961b89-maas-api-tls\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:47.411982 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:47.411809 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5"] Nov 01 16:10:47.413055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:47.413003 4363 cpu_manager.go:414] "containerMap: removing container" podUID="a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25" containerName="maas-api" Nov 01 16:10:47.413055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:47.413056 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25" containerName="maas-api" Nov 01 16:10:47.413233 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:47.413090 4363 cpu_manager.go:414] "containerMap: removing container" podUID="21e3bc0d-84f1-4ae1-baaa-97509d961b89" containerName="maas-api" Nov 01 16:10:47.413233 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:47.413095 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21e3bc0d-84f1-4ae1-baaa-97509d961b89" containerName="maas-api" Nov 01 16:10:47.413233 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:47.413225 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6b8a2f8-f6a0-4faa-8cc2-8b86b70fdd25" containerName="maas-api" Nov 01 16:10:47.413383 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:47.413239 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="21e3bc0d-84f1-4ae1-baaa-97509d961b89" containerName="maas-api" Nov 01 16:10:47.421528 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:47.417931 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" Nov 01 16:10:47.423806 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:47.423784 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-26310f-openshift-default-dockercfg-tvrb4\"" Nov 01 16:10:47.431152 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:47.431106 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5"] Nov 01 16:10:47.440299 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:47.440270 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1193ea-2099-4ce6-b417-82d1e8e051b8-workload-socket\") pod \"e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5\" (UID: \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\") " pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" Nov 01 16:10:47.440630 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:47.440606 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d1193ea-2099-4ce6-b417-82d1e8e051b8-istio-data\") pod \"e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5\" (UID: \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\") " pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" Nov 01 16:10:47.440738 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:47.440648 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d1193ea-2099-4ce6-b417-82d1e8e051b8-istiod-ca-cert\") pod \"e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5\" (UID: \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\") " pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" Nov 01 16:10:47.440738 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:47.440720 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv5l6\" (UniqueName: \"kubernetes.io/projected/2d1193ea-2099-4ce6-b417-82d1e8e051b8-kube-api-access-fv5l6\") pod \"e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5\" (UID: \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\") " pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" Nov 01 16:10:47.440857 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:47.440778 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d1193ea-2099-4ce6-b417-82d1e8e051b8-istio-token\") pod \"e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5\" (UID: \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\") " pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" Nov 01 16:10:47.440857 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:47.440834 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d1193ea-2099-4ce6-b417-82d1e8e051b8-istio-podinfo\") pod \"e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5\" (UID: \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\") " pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" Nov 01 16:10:47.440977 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:47.440867 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1193ea-2099-4ce6-b417-82d1e8e051b8-credential-socket\") pod \"e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5\" (UID: \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\") " pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" Nov 01 16:10:47.440977 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:47.440893 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d1193ea-2099-4ce6-b417-82d1e8e051b8-workload-certs\") pod \"e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5\" (UID: \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\") " pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" Nov 01 16:10:47.441157 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:47.441139 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d1193ea-2099-4ce6-b417-82d1e8e051b8-istio-envoy\") pod \"e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5\" (UID: \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\") " pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" Nov 01 16:10:47.673558 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:47.664309 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d1193ea-2099-4ce6-b417-82d1e8e051b8-istio-podinfo\") pod \"e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5\" (UID: \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\") " pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" Nov 01 16:10:47.674188 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:47.674154 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1193ea-2099-4ce6-b417-82d1e8e051b8-credential-socket\") pod \"e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5\" (UID: \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\") " pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" Nov 01 16:10:47.674366 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:47.674349 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d1193ea-2099-4ce6-b417-82d1e8e051b8-workload-certs\") pod \"e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5\" (UID: \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\") " pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" Nov 01 16:10:47.682116 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:47.674637 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d1193ea-2099-4ce6-b417-82d1e8e051b8-istio-envoy\") pod \"e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5\" (UID: \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\") " pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" Nov 01 16:10:47.682429 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:47.682404 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1193ea-2099-4ce6-b417-82d1e8e051b8-workload-socket\") pod \"e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5\" (UID: \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\") " pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" Nov 01 16:10:47.682708 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:47.682691 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d1193ea-2099-4ce6-b417-82d1e8e051b8-istio-data\") pod \"e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5\" (UID: \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\") " pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" Nov 01 16:10:47.711290 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:47.709072 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d1193ea-2099-4ce6-b417-82d1e8e051b8-istiod-ca-cert\") pod \"e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5\" (UID: \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\") " pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" Nov 01 16:10:47.711290 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:47.709184 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv5l6\" (UniqueName: \"kubernetes.io/projected/2d1193ea-2099-4ce6-b417-82d1e8e051b8-kube-api-access-fv5l6\") pod \"e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5\" (UID: \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\") " pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" Nov 01 16:10:47.711290 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:47.709279 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d1193ea-2099-4ce6-b417-82d1e8e051b8-istio-token\") pod \"e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5\" (UID: \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\") " pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" Nov 01 16:10:47.711290 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:47.708959 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d1193ea-2099-4ce6-b417-82d1e8e051b8-istio-podinfo\") pod \"e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5\" (UID: \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\") " pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" Nov 01 16:10:47.711290 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:47.683402 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1193ea-2099-4ce6-b417-82d1e8e051b8-credential-socket\") pod \"e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5\" (UID: \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\") " pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" Nov 01 16:10:47.711717 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:47.711313 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d1193ea-2099-4ce6-b417-82d1e8e051b8-istiod-ca-cert\") pod \"e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5\" (UID: \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\") " pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" Nov 01 16:10:47.753038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:47.691272 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1193ea-2099-4ce6-b417-82d1e8e051b8-workload-socket\") pod \"e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5\" (UID: \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\") " pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" Nov 01 16:10:47.753038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:47.691989 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d1193ea-2099-4ce6-b417-82d1e8e051b8-workload-certs\") pod \"e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5\" (UID: \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\") " pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" Nov 01 16:10:47.753038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:47.716753 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d1193ea-2099-4ce6-b417-82d1e8e051b8-istio-data\") pod \"e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5\" (UID: \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\") " pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" Nov 01 16:10:47.762661 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:47.762627 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d1193ea-2099-4ce6-b417-82d1e8e051b8-istio-envoy\") pod \"e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5\" (UID: \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\") " pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" Nov 01 16:10:47.877922 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:47.848518 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-2zh4r" event={"ID":"21e3bc0d-84f1-4ae1-baaa-97509d961b89","Type":"ContainerDied","Data":"ff6f2ce3352e0a8602571bb0ee3bcb50fe1d7314fd8670211d4b9a5282611571"} Nov 01 16:10:47.878527 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:47.878490 4363 scope.go:122] "RemoveContainer" containerID="b85a4b4298ecdae0a78c4771a0e57de23e4a8040b699d19fc80295f83ce9823a" Nov 01 16:10:47.879449 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:47.848822 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-2zh4r" Nov 01 16:10:47.902288 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:47.902254 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv5l6\" (UniqueName: \"kubernetes.io/projected/2d1193ea-2099-4ce6-b417-82d1e8e051b8-kube-api-access-fv5l6\") pod \"e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5\" (UID: \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\") " pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" Nov 01 16:10:47.904217 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:47.904154 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d1193ea-2099-4ce6-b417-82d1e8e051b8-istio-token\") pod \"e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5\" (UID: \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\") " pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" Nov 01 16:10:47.967943 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:47.966810 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-2zh4r"] Nov 01 16:10:48.016396 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:47.985716 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-2zh4r"] Nov 01 16:10:48.040039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:48.039957 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" Nov 01 16:10:48.412731 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:48.412702 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5"] Nov 01 16:10:48.419643 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:48.419027 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:10:48.419643 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:48.419106 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:10:48.419643 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:10:48.419133 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:10:48.419643 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:48.419142 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:10:48.804088 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:48.803969 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21e3bc0d-84f1-4ae1-baaa-97509d961b89" path="/var/lib/kubelet/pods/21e3bc0d-84f1-4ae1-baaa-97509d961b89/volumes" Nov 01 16:10:48.863329 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:48.863300 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" event={"ID":"2d1193ea-2099-4ce6-b417-82d1e8e051b8","Type":"ContainerStarted","Data":"f2ce23c7cc7f14058bb2d2a272324caa25278b384841729f423191a077b7e365"} Nov 01 16:10:48.863521 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:48.863494 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" event={"ID":"2d1193ea-2099-4ce6-b417-82d1e8e051b8","Type":"ContainerStarted","Data":"cf4e824c29457c671795dd89448c932cb48bad86e88df6d523d62f1b4067c472"} Nov 01 16:10:48.894277 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:48.893670 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" podStartSLOduration=1.893652248 podStartE2EDuration="1.893652248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:10:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:10:48.880662511 +0000 UTC m=+1382.212400161" watchObservedRunningTime="2026-11-01 16:10:48.893652248 +0000 UTC m=+1382.225389898" Nov 01 16:10:49.058255 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:49.058156 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" Nov 01 16:10:49.375044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:49.370258 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" Nov 01 16:10:49.507522 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:49.507493 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-58b0d1-5455d7568c-jh7m4" Nov 01 16:10:49.507705 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:49.507560 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-58b0d1-786cf8cdb9-cbftv" Nov 01 16:10:49.988065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:49.983179 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" Nov 01 16:10:49.988065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:49.984887 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" Nov 01 16:10:51.564062 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:51.564000 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-375350-76f869b4d8-ljvvj" Nov 01 16:10:51.600575 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:51.600545 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm"] Nov 01 16:10:52.502744 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:52.502713 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-26310f-778676cc8c-ppxzh"] Nov 01 16:10:52.510580 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:52.510548 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-26310f-58f7f46c57-hqjhw"] Nov 01 16:10:52.510752 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:52.510707 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-26310f-778676cc8c-ppxzh" Nov 01 16:10:52.519398 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:52.519374 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-26310f\"" Nov 01 16:10:52.519531 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:52.519489 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-26310f-dockercfg-nqspb\"" Nov 01 16:10:52.520593 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:52.520571 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-26310f-778676cc8c-ppxzh"] Nov 01 16:10:52.520701 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:52.520663 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-26310f-58f7f46c57-hqjhw" Nov 01 16:10:52.522834 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:52.522809 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-26310f-58f7f46c57-hqjhw"] Nov 01 16:10:52.680878 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:52.679796 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8krqj\" (UniqueName: \"kubernetes.io/projected/8227cf4e-1b88-4ba8-914d-30c14e972787-kube-api-access-8krqj\") pod \"payload-processing-e2e-api-b-26310f-58f7f46c57-hqjhw\" (UID: \"8227cf4e-1b88-4ba8-914d-30c14e972787\") " pod="openshift-ingress/payload-processing-e2e-api-b-26310f-58f7f46c57-hqjhw" Nov 01 16:10:52.680878 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:52.679963 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fwdt\" (UniqueName: \"kubernetes.io/projected/b398ad44-84ce-42af-98e9-4f097895b3fd-kube-api-access-9fwdt\") pod \"payload-pre-processing-e2e-api-b-26310f-778676cc8c-ppxzh\" (UID: \"b398ad44-84ce-42af-98e9-4f097895b3fd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-26310f-778676cc8c-ppxzh" Nov 01 16:10:52.680878 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:52.680094 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8227cf4e-1b88-4ba8-914d-30c14e972787-plugins-config-volume\") pod \"payload-processing-e2e-api-b-26310f-58f7f46c57-hqjhw\" (UID: \"8227cf4e-1b88-4ba8-914d-30c14e972787\") " pod="openshift-ingress/payload-processing-e2e-api-b-26310f-58f7f46c57-hqjhw" Nov 01 16:10:52.680878 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:52.680178 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b398ad44-84ce-42af-98e9-4f097895b3fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-26310f-778676cc8c-ppxzh\" (UID: \"b398ad44-84ce-42af-98e9-4f097895b3fd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-26310f-778676cc8c-ppxzh" Nov 01 16:10:52.785812 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:52.785720 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8krqj\" (UniqueName: \"kubernetes.io/projected/8227cf4e-1b88-4ba8-914d-30c14e972787-kube-api-access-8krqj\") pod \"payload-processing-e2e-api-b-26310f-58f7f46c57-hqjhw\" (UID: \"8227cf4e-1b88-4ba8-914d-30c14e972787\") " pod="openshift-ingress/payload-processing-e2e-api-b-26310f-58f7f46c57-hqjhw" Nov 01 16:10:52.786178 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:52.785883 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fwdt\" (UniqueName: \"kubernetes.io/projected/b398ad44-84ce-42af-98e9-4f097895b3fd-kube-api-access-9fwdt\") pod \"payload-pre-processing-e2e-api-b-26310f-778676cc8c-ppxzh\" (UID: \"b398ad44-84ce-42af-98e9-4f097895b3fd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-26310f-778676cc8c-ppxzh" Nov 01 16:10:52.786178 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:52.785959 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8227cf4e-1b88-4ba8-914d-30c14e972787-plugins-config-volume\") pod \"payload-processing-e2e-api-b-26310f-58f7f46c57-hqjhw\" (UID: \"8227cf4e-1b88-4ba8-914d-30c14e972787\") " pod="openshift-ingress/payload-processing-e2e-api-b-26310f-58f7f46c57-hqjhw" Nov 01 16:10:52.786178 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:52.786058 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b398ad44-84ce-42af-98e9-4f097895b3fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-26310f-778676cc8c-ppxzh\" (UID: \"b398ad44-84ce-42af-98e9-4f097895b3fd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-26310f-778676cc8c-ppxzh" Nov 01 16:10:52.791037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:52.787178 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b398ad44-84ce-42af-98e9-4f097895b3fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-26310f-778676cc8c-ppxzh\" (UID: \"b398ad44-84ce-42af-98e9-4f097895b3fd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-26310f-778676cc8c-ppxzh" Nov 01 16:10:52.805068 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:52.799822 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8krqj\" (UniqueName: \"kubernetes.io/projected/8227cf4e-1b88-4ba8-914d-30c14e972787-kube-api-access-8krqj\") pod \"payload-processing-e2e-api-b-26310f-58f7f46c57-hqjhw\" (UID: \"8227cf4e-1b88-4ba8-914d-30c14e972787\") " pod="openshift-ingress/payload-processing-e2e-api-b-26310f-58f7f46c57-hqjhw" Nov 01 16:10:52.805068 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:52.802584 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8227cf4e-1b88-4ba8-914d-30c14e972787-plugins-config-volume\") pod \"payload-processing-e2e-api-b-26310f-58f7f46c57-hqjhw\" (UID: \"8227cf4e-1b88-4ba8-914d-30c14e972787\") " pod="openshift-ingress/payload-processing-e2e-api-b-26310f-58f7f46c57-hqjhw" Nov 01 16:10:52.855076 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:52.849423 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fwdt\" (UniqueName: \"kubernetes.io/projected/b398ad44-84ce-42af-98e9-4f097895b3fd-kube-api-access-9fwdt\") pod \"payload-pre-processing-e2e-api-b-26310f-778676cc8c-ppxzh\" (UID: \"b398ad44-84ce-42af-98e9-4f097895b3fd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-26310f-778676cc8c-ppxzh" Nov 01 16:10:52.855076 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:52.849891 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-26310f-778676cc8c-ppxzh" Nov 01 16:10:52.855773 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:52.855746 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-26310f-58f7f46c57-hqjhw" Nov 01 16:10:53.502086 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:10:53.502056 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb398ad44_84ce_42af_98e9_4f097895b3fd.slice/crio-867692a46e3ad1a3bdc088e6fd696b02aaccd7036737854b57d6580566db9da2 WatchSource:0}: Error finding container 867692a46e3ad1a3bdc088e6fd696b02aaccd7036737854b57d6580566db9da2: Status 404 returned error can't find the container with id 867692a46e3ad1a3bdc088e6fd696b02aaccd7036737854b57d6580566db9da2 Nov 01 16:10:53.505448 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:53.505421 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-26310f-778676cc8c-ppxzh"] Nov 01 16:10:53.559327 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:53.559290 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-26310f-58f7f46c57-hqjhw"] Nov 01 16:10:53.562609 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:10:53.562584 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8227cf4e_1b88_4ba8_914d_30c14e972787.slice/crio-99c3e76070927f5613d92304211c7d9597d43dfd3b3079061304148cf9c7a728 WatchSource:0}: Error finding container 99c3e76070927f5613d92304211c7d9597d43dfd3b3079061304148cf9c7a728: Status 404 returned error can't find the container with id 99c3e76070927f5613d92304211c7d9597d43dfd3b3079061304148cf9c7a728 Nov 01 16:10:54.097180 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:54.097141 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-26310f-778676cc8c-ppxzh" event={"ID":"b398ad44-84ce-42af-98e9-4f097895b3fd","Type":"ContainerStarted","Data":"e1e8d1be81a6ec3f3eb2c3eb29d6042c8177b8e94e2b7bae8084fcb3438941f5"} Nov 01 16:10:54.097631 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:54.097207 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-26310f-778676cc8c-ppxzh" event={"ID":"b398ad44-84ce-42af-98e9-4f097895b3fd","Type":"ContainerStarted","Data":"867692a46e3ad1a3bdc088e6fd696b02aaccd7036737854b57d6580566db9da2"} Nov 01 16:10:54.097631 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:54.097240 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-26310f-778676cc8c-ppxzh" Nov 01 16:10:54.099278 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:54.099249 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-26310f-58f7f46c57-hqjhw" event={"ID":"8227cf4e-1b88-4ba8-914d-30c14e972787","Type":"ContainerStarted","Data":"a016506f764477f351c441bba45c516ce1bb8cff9a01fc7b0dd7fc506838ca07"} Nov 01 16:10:54.099404 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:54.099299 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-26310f-58f7f46c57-hqjhw" event={"ID":"8227cf4e-1b88-4ba8-914d-30c14e972787","Type":"ContainerStarted","Data":"99c3e76070927f5613d92304211c7d9597d43dfd3b3079061304148cf9c7a728"} Nov 01 16:10:54.099404 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:54.099351 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-26310f-58f7f46c57-hqjhw" Nov 01 16:10:54.120769 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:54.120724 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-26310f-778676cc8c-ppxzh" podStartSLOduration=2.120708271 podStartE2EDuration="2.120708271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:10:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:10:54.112576011 +0000 UTC m=+1387.444313662" watchObservedRunningTime="2026-11-01 16:10:54.120708271 +0000 UTC m=+1387.452445922" Nov 01 16:10:54.132255 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:54.132204 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-26310f-58f7f46c57-hqjhw" podStartSLOduration=2.132186349 podStartE2EDuration="2.132186349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:10:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:10:54.129259961 +0000 UTC m=+1387.460997605" watchObservedRunningTime="2026-11-01 16:10:54.132186349 +0000 UTC m=+1387.463924001" Nov 01 16:10:54.517086 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:54.516787 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-58b0d1-799d79584b-5ckb9"] Nov 01 16:10:54.528380 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:54.526780 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58b0d1-799d79584b-5ckb9" Nov 01 16:10:54.531306 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:54.531279 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-58b0d1-serving-cert\"" Nov 01 16:10:54.542540 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:54.542505 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-58b0d1-799d79584b-5ckb9"] Nov 01 16:10:54.595878 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:54.595841 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9mp8\" (UniqueName: \"kubernetes.io/projected/237a3f5e-83c1-43e0-a6f8-01dcfca57d32-kube-api-access-n9mp8\") pod \"maas-api-e2e-api-b-58b0d1-799d79584b-5ckb9\" (UID: \"237a3f5e-83c1-43e0-a6f8-01dcfca57d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58b0d1-799d79584b-5ckb9" Nov 01 16:10:54.596110 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:54.596093 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/237a3f5e-83c1-43e0-a6f8-01dcfca57d32-openshift-service-ca\") pod \"maas-api-e2e-api-b-58b0d1-799d79584b-5ckb9\" (UID: \"237a3f5e-83c1-43e0-a6f8-01dcfca57d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58b0d1-799d79584b-5ckb9" Nov 01 16:10:54.596304 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:54.596282 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/237a3f5e-83c1-43e0-a6f8-01dcfca57d32-metrics-certs\") pod \"maas-api-e2e-api-b-58b0d1-799d79584b-5ckb9\" (UID: \"237a3f5e-83c1-43e0-a6f8-01dcfca57d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58b0d1-799d79584b-5ckb9" Nov 01 16:10:54.596463 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:54.596446 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/237a3f5e-83c1-43e0-a6f8-01dcfca57d32-maas-api-tls\") pod \"maas-api-e2e-api-b-58b0d1-799d79584b-5ckb9\" (UID: \"237a3f5e-83c1-43e0-a6f8-01dcfca57d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58b0d1-799d79584b-5ckb9" Nov 01 16:10:54.698491 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:54.698445 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/237a3f5e-83c1-43e0-a6f8-01dcfca57d32-openshift-service-ca\") pod \"maas-api-e2e-api-b-58b0d1-799d79584b-5ckb9\" (UID: \"237a3f5e-83c1-43e0-a6f8-01dcfca57d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58b0d1-799d79584b-5ckb9" Nov 01 16:10:54.698705 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:54.698594 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/237a3f5e-83c1-43e0-a6f8-01dcfca57d32-metrics-certs\") pod \"maas-api-e2e-api-b-58b0d1-799d79584b-5ckb9\" (UID: \"237a3f5e-83c1-43e0-a6f8-01dcfca57d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58b0d1-799d79584b-5ckb9" Nov 01 16:10:54.698776 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:54.698711 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/237a3f5e-83c1-43e0-a6f8-01dcfca57d32-maas-api-tls\") pod \"maas-api-e2e-api-b-58b0d1-799d79584b-5ckb9\" (UID: \"237a3f5e-83c1-43e0-a6f8-01dcfca57d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58b0d1-799d79584b-5ckb9" Nov 01 16:10:54.698833 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:54.698812 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9mp8\" (UniqueName: \"kubernetes.io/projected/237a3f5e-83c1-43e0-a6f8-01dcfca57d32-kube-api-access-n9mp8\") pod \"maas-api-e2e-api-b-58b0d1-799d79584b-5ckb9\" (UID: \"237a3f5e-83c1-43e0-a6f8-01dcfca57d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58b0d1-799d79584b-5ckb9" Nov 01 16:10:54.699314 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:54.699283 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/237a3f5e-83c1-43e0-a6f8-01dcfca57d32-openshift-service-ca\") pod \"maas-api-e2e-api-b-58b0d1-799d79584b-5ckb9\" (UID: \"237a3f5e-83c1-43e0-a6f8-01dcfca57d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58b0d1-799d79584b-5ckb9" Nov 01 16:10:54.701341 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:54.701315 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/237a3f5e-83c1-43e0-a6f8-01dcfca57d32-metrics-certs\") pod \"maas-api-e2e-api-b-58b0d1-799d79584b-5ckb9\" (UID: \"237a3f5e-83c1-43e0-a6f8-01dcfca57d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58b0d1-799d79584b-5ckb9" Nov 01 16:10:54.701763 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:54.701730 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/237a3f5e-83c1-43e0-a6f8-01dcfca57d32-maas-api-tls\") pod \"maas-api-e2e-api-b-58b0d1-799d79584b-5ckb9\" (UID: \"237a3f5e-83c1-43e0-a6f8-01dcfca57d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58b0d1-799d79584b-5ckb9" Nov 01 16:10:54.706898 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:54.706872 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9mp8\" (UniqueName: \"kubernetes.io/projected/237a3f5e-83c1-43e0-a6f8-01dcfca57d32-kube-api-access-n9mp8\") pod \"maas-api-e2e-api-b-58b0d1-799d79584b-5ckb9\" (UID: \"237a3f5e-83c1-43e0-a6f8-01dcfca57d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58b0d1-799d79584b-5ckb9" Nov 01 16:10:54.868456 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:54.856857 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58b0d1-799d79584b-5ckb9" Nov 01 16:10:55.371645 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:55.371618 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-58b0d1-799d79584b-5ckb9"] Nov 01 16:10:56.395578 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:56.395483 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58b0d1-799d79584b-5ckb9" event={"ID":"237a3f5e-83c1-43e0-a6f8-01dcfca57d32","Type":"ContainerStarted","Data":"d55bfed49e9f73ba47012de2e9905e19852d56c634527b15f756a2fafd5a5a7c"} Nov 01 16:10:56.395578 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:56.395553 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58b0d1-799d79584b-5ckb9" event={"ID":"237a3f5e-83c1-43e0-a6f8-01dcfca57d32","Type":"ContainerStarted","Data":"3da9b5044a021a2fde1a0cc20a791bbff06cbf1d64029cb1f58e3b43066785a3"} Nov 01 16:10:56.396149 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:56.396088 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58b0d1-799d79584b-5ckb9" Nov 01 16:10:56.417098 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:56.417052 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58b0d1-799d79584b-5ckb9" podStartSLOduration=1.985417303 podStartE2EDuration="2.41703843s" totalImagesPullingTime="431.621127ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:10:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:10:56.413008033 +0000 UTC m=+1389.744745685" watchObservedRunningTime="2026-11-01 16:10:56.41703843 +0000 UTC m=+1389.748776122" Nov 01 16:10:56.594449 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:56.594413 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c65e3-77686b98cf-wzm9h" Nov 01 16:10:56.622040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:10:56.595539 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1c65e3-54b4ddd7d7-lbrcc" Nov 01 16:11:00.597039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:00.596131 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 01 16:11:00.688170 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:00.688122 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:00.772205 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:00.772105 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:00.892101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:00.872793 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:11:01.598470 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:01.598384 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:01.740041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:01.623752 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 16:11:01.810045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:01.777847 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:02.053093 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:02.040502 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:11:02.053093 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:02.043932 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 16:11:02.053093 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:01.785948 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" podUID="f1e164ab-e940-44e0-8829-8626e4162a9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:02.081178 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:02.054116 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:02.106041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:02.098432 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:11:02.154178 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:02.107086 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:11:02.625146 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:02.625093 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:02.626967 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:02.626304 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:11:02.640036 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:02.634913 4363 trace.go:236] Trace[1417997489]: "Calculate volume metrics of manifests for pod opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-l7sxc" (01-Nov-2026 16:11:01.597) (total time: 1037ms): Nov 01 16:11:02.640036 ip-10-0-128-238 kubenswrapper[4363]: Trace[1417997489]: [1.037033215s] [1.037033215s] END Nov 01 16:11:03.053587 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:03.028141 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:03.055042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:03.054982 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:03.055759 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:03.055728 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" podUID="f1e164ab-e940-44e0-8829-8626e4162a9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.115:8000/ready\": context deadline exceeded" Nov 01 16:11:03.068040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:03.064033 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-80c4c8-75974fbc49-zqjkh"] Nov 01 16:11:03.068355 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:03.068330 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58b0d1-799d79584b-5ckb9" Nov 01 16:11:03.076184 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:03.076156 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" Nov 01 16:11:03.076409 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:03.076391 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-80c4c8-5779b74587-k4gkg"] Nov 01 16:11:03.132160 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:03.114152 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:11:03.132160 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:03.114627 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:11:03.150859 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:03.150803 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:11:03.181063 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:03.154547 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:11:03.707762 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:03.707720 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:11:03.777059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:03.719116 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:11:03.777059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:03.722554 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:04.030263 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:04.030159 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:04.150375 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:04.150333 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:11:04.167627 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:04.167596 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" Nov 01 16:11:04.189165 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:04.189115 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:04.190662 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:04.190635 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:11:04.351100 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:04.325262 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:04.712102 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:04.711547 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:11:04.734563 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:04.734520 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:04.975389 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:04.975081 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:04.997049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:04.996993 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": context deadline exceeded" Nov 01 16:11:05.074959 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:05.042983 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:11:05.094504 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:05.092644 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" podUID="f1e164ab-e940-44e0-8829-8626e4162a9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:05.154161 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:05.154115 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:11:05.229096 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:05.215347 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:11:05.229096 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:05.229066 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:05.394378 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:05.394343 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-26310f-778676cc8c-ppxzh" Nov 01 16:11:05.397805 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:05.397775 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-26310f-58f7f46c57-hqjhw" Nov 01 16:11:05.735570 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:05.734940 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:05.778140 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:05.777466 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:05.828843 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:05.821175 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 16:11:06.114089 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:06.105225 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" podUID="f1e164ab-e940-44e0-8829-8626e4162a9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:06.114089 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:06.107998 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" Nov 01 16:11:06.290039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:06.285366 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:11:06.340167 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:06.337224 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:06.365116 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:06.364774 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:11:06.612044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:06.601836 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 16:11:06.652104 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:06.651698 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:11:06.672413 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:06.666529 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:11:06.672413 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:06.669221 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" podUID="f1e164ab-e940-44e0-8829-8626e4162a9f" containerName="main" containerID="cri-o://339bacc523e4532427358aaa627c777e82f8d61627e8b3613a97ccbf1fd9ab02" gracePeriod=45 Nov 01 16:11:06.677012 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:06.673338 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:11:07.197192 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:07.172064 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" podUID="f1e164ab-e940-44e0-8829-8626e4162a9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.115:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:07.236102 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:07.223825 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" podUID="f1e164ab-e940-44e0-8829-8626e4162a9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.115:8000/ready\": dial tcp 10.134.0.115:8000: connect: connection refused" Nov 01 16:11:07.294180 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:07.294144 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:11:08.100098 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:08.082683 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-80c4c8-75974fbc49-zqjkh" podUID="568c7dd7-469a-42a0-bb33-85167782b280" containerName="payload-processing" containerID="cri-o://5935d2652e6a2b3f3bc191cb5f9396949c0c35dde1d6775f28ee392d49102511" gracePeriod=25 Nov 01 16:11:08.100098 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:08.082851 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-80c4c8-5779b74587-k4gkg" podUID="0d6b57c1-aa75-406f-aa95-f455f8690078" containerName="payload-pre-processing" containerID="cri-o://5d429f533209d90a11bb526cdbb9aaaa0565efec86ae650726e1a4273e8e835f" gracePeriod=25 Nov 01 16:11:08.125724 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:08.115260 4363 generic.go:356] "Generic (PLEG): container finished" podID="f1e164ab-e940-44e0-8829-8626e4162a9f" containerID="339bacc523e4532427358aaa627c777e82f8d61627e8b3613a97ccbf1fd9ab02" exitCode=0 Nov 01 16:11:08.125724 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:08.115436 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" event={"ID":"f1e164ab-e940-44e0-8829-8626e4162a9f","Type":"ContainerDied","Data":"339bacc523e4532427358aaa627c777e82f8d61627e8b3613a97ccbf1fd9ab02"} Nov 01 16:11:08.271607 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:08.270950 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" Nov 01 16:11:08.285410 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:08.284789 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1e164ab-e940-44e0-8829-8626e4162a9f-kube-api-access-wqxgk\" (UniqueName: \"kubernetes.io/projected/f1e164ab-e940-44e0-8829-8626e4162a9f-kube-api-access-wqxgk\") pod \"f1e164ab-e940-44e0-8829-8626e4162a9f\" (UID: \"f1e164ab-e940-44e0-8829-8626e4162a9f\") " Nov 01 16:11:08.285410 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:08.284923 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1e164ab-e940-44e0-8829-8626e4162a9f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1e164ab-e940-44e0-8829-8626e4162a9f-tmp-dir\") pod \"f1e164ab-e940-44e0-8829-8626e4162a9f\" (UID: \"f1e164ab-e940-44e0-8829-8626e4162a9f\") " Nov 01 16:11:08.285410 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:08.284999 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1e164ab-e940-44e0-8829-8626e4162a9f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1e164ab-e940-44e0-8829-8626e4162a9f-dshm\") pod \"f1e164ab-e940-44e0-8829-8626e4162a9f\" (UID: \"f1e164ab-e940-44e0-8829-8626e4162a9f\") " Nov 01 16:11:08.285410 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:08.285071 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1e164ab-e940-44e0-8829-8626e4162a9f-tls-certs\" (UniqueName: \"kubernetes.io/secret/f1e164ab-e940-44e0-8829-8626e4162a9f-tls-certs\") pod \"f1e164ab-e940-44e0-8829-8626e4162a9f\" (UID: \"f1e164ab-e940-44e0-8829-8626e4162a9f\") " Nov 01 16:11:08.285410 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:08.285122 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1e164ab-e940-44e0-8829-8626e4162a9f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1e164ab-e940-44e0-8829-8626e4162a9f-model-cache\") pod \"f1e164ab-e940-44e0-8829-8626e4162a9f\" (UID: \"f1e164ab-e940-44e0-8829-8626e4162a9f\") " Nov 01 16:11:08.285410 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:08.285165 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1e164ab-e940-44e0-8829-8626e4162a9f-home\" (UniqueName: \"kubernetes.io/empty-dir/f1e164ab-e940-44e0-8829-8626e4162a9f-home\") pod \"f1e164ab-e940-44e0-8829-8626e4162a9f\" (UID: \"f1e164ab-e940-44e0-8829-8626e4162a9f\") " Nov 01 16:11:08.313291 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:08.306720 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1e164ab-e940-44e0-8829-8626e4162a9f-kube-api-access-wqxgk" pod "f1e164ab-e940-44e0-8829-8626e4162a9f" (UID: "f1e164ab-e940-44e0-8829-8626e4162a9f"). InnerVolumeSpecName "kube-api-access-wqxgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:11:08.313291 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:08.313227 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1e164ab-e940-44e0-8829-8626e4162a9f-model-cache" pod "f1e164ab-e940-44e0-8829-8626e4162a9f" (UID: "f1e164ab-e940-44e0-8829-8626e4162a9f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:11:08.369066 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:08.368136 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1e164ab-e940-44e0-8829-8626e4162a9f-tmp-dir" pod "f1e164ab-e940-44e0-8829-8626e4162a9f" (UID: "f1e164ab-e940-44e0-8829-8626e4162a9f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:11:08.369066 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:08.368199 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1e164ab-e940-44e0-8829-8626e4162a9f-home" pod "f1e164ab-e940-44e0-8829-8626e4162a9f" (UID: "f1e164ab-e940-44e0-8829-8626e4162a9f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:11:08.369066 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:08.368272 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1e164ab-e940-44e0-8829-8626e4162a9f-dshm" pod "f1e164ab-e940-44e0-8829-8626e4162a9f" (UID: "f1e164ab-e940-44e0-8829-8626e4162a9f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:11:08.420311 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:08.420206 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1e164ab-e940-44e0-8829-8626e4162a9f-tls-certs" pod "f1e164ab-e940-44e0-8829-8626e4162a9f" (UID: "f1e164ab-e940-44e0-8829-8626e4162a9f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:11:08.439584 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:08.422294 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqxgk\" (UniqueName: \"kubernetes.io/projected/f1e164ab-e940-44e0-8829-8626e4162a9f-kube-api-access-wqxgk\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:08.439584 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:08.422333 4363 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1e164ab-e940-44e0-8829-8626e4162a9f-tmp-dir\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:08.439584 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:08.422351 4363 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1e164ab-e940-44e0-8829-8626e4162a9f-dshm\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:08.439584 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:08.422365 4363 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1e164ab-e940-44e0-8829-8626e4162a9f-tls-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:08.439584 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:08.422379 4363 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1e164ab-e940-44e0-8829-8626e4162a9f-model-cache\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:08.439584 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:08.422394 4363 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1e164ab-e940-44e0-8829-8626e4162a9f-home\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:08.948041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:08.947275 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" podUID="f1e164ab-e940-44e0-8829-8626e4162a9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:09.103043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.100252 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-80c4c8-75974fbc49-zqjkh" Nov 01 16:11:09.169598 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.169548 4363 generic.go:356] "Generic (PLEG): container finished" podID="0d6b57c1-aa75-406f-aa95-f455f8690078" containerID="5d429f533209d90a11bb526cdbb9aaaa0565efec86ae650726e1a4273e8e835f" exitCode=0 Nov 01 16:11:09.169808 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.169664 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-80c4c8-5779b74587-k4gkg" event={"ID":"0d6b57c1-aa75-406f-aa95-f455f8690078","Type":"ContainerDied","Data":"5d429f533209d90a11bb526cdbb9aaaa0565efec86ae650726e1a4273e8e835f"} Nov 01 16:11:09.173411 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.173375 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" event={"ID":"f1e164ab-e940-44e0-8829-8626e4162a9f","Type":"ContainerDied","Data":"c9bf3fc3b589a648148d34b922917bfd80ec2b50bfbe5082985d8a4bcf403b7f"} Nov 01 16:11:09.173524 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.173434 4363 scope.go:122] "RemoveContainer" containerID="339bacc523e4532427358aaa627c777e82f8d61627e8b3613a97ccbf1fd9ab02" Nov 01 16:11:09.173635 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.173618 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm" Nov 01 16:11:09.208064 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.196270 4363 generic.go:356] "Generic (PLEG): container finished" podID="568c7dd7-469a-42a0-bb33-85167782b280" containerID="5935d2652e6a2b3f3bc191cb5f9396949c0c35dde1d6775f28ee392d49102511" exitCode=0 Nov 01 16:11:09.208064 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.201149 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-80c4c8-75974fbc49-zqjkh" Nov 01 16:11:09.217816 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.217766 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-80c4c8-75974fbc49-zqjkh" event={"ID":"568c7dd7-469a-42a0-bb33-85167782b280","Type":"ContainerDied","Data":"5935d2652e6a2b3f3bc191cb5f9396949c0c35dde1d6775f28ee392d49102511"} Nov 01 16:11:09.218128 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.218101 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-80c4c8-75974fbc49-zqjkh" event={"ID":"568c7dd7-469a-42a0-bb33-85167782b280","Type":"ContainerDied","Data":"16e268748326a91b4bf67e7a93f3e2b00de82a2190b1b141cbb59cc82745d43f"} Nov 01 16:11:09.335138 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.327907 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/568c7dd7-469a-42a0-bb33-85167782b280-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/568c7dd7-469a-42a0-bb33-85167782b280-plugins-config-volume\") pod \"568c7dd7-469a-42a0-bb33-85167782b280\" (UID: \"568c7dd7-469a-42a0-bb33-85167782b280\") " Nov 01 16:11:09.430044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.429582 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/568c7dd7-469a-42a0-bb33-85167782b280-kube-api-access-xnc7b\" (UniqueName: \"kubernetes.io/projected/568c7dd7-469a-42a0-bb33-85167782b280-kube-api-access-xnc7b\") pod \"568c7dd7-469a-42a0-bb33-85167782b280\" (UID: \"568c7dd7-469a-42a0-bb33-85167782b280\") " Nov 01 16:11:09.434957 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.387272 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/568c7dd7-469a-42a0-bb33-85167782b280-plugins-config-volume" pod "568c7dd7-469a-42a0-bb33-85167782b280" (UID: "568c7dd7-469a-42a0-bb33-85167782b280"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:11:09.435231 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.435188 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm"] Nov 01 16:11:09.439598 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.439569 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-76063d10/e2e-lifecycle-model-76063d10-kserve-5d698596bf-x5xvm"] Nov 01 16:11:09.441327 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.441303 4363 scope.go:122] "RemoveContainer" containerID="5935d2652e6a2b3f3bc191cb5f9396949c0c35dde1d6775f28ee392d49102511" Nov 01 16:11:09.446506 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.446465 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/568c7dd7-469a-42a0-bb33-85167782b280-kube-api-access-xnc7b" pod "568c7dd7-469a-42a0-bb33-85167782b280" (UID: "568c7dd7-469a-42a0-bb33-85167782b280"). InnerVolumeSpecName "kube-api-access-xnc7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:11:09.498717 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.498600 4363 scope.go:122] "RemoveContainer" containerID="5935d2652e6a2b3f3bc191cb5f9396949c0c35dde1d6775f28ee392d49102511" Nov 01 16:11:09.499172 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:11:09.499098 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5935d2652e6a2b3f3bc191cb5f9396949c0c35dde1d6775f28ee392d49102511\": container with ID starting with 5935d2652e6a2b3f3bc191cb5f9396949c0c35dde1d6775f28ee392d49102511 not found: ID does not exist" containerID="5935d2652e6a2b3f3bc191cb5f9396949c0c35dde1d6775f28ee392d49102511" Nov 01 16:11:09.499298 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.499185 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5935d2652e6a2b3f3bc191cb5f9396949c0c35dde1d6775f28ee392d49102511"} err="failed to get container status \"5935d2652e6a2b3f3bc191cb5f9396949c0c35dde1d6775f28ee392d49102511\": rpc error: code = NotFound desc = could not find container \"5935d2652e6a2b3f3bc191cb5f9396949c0c35dde1d6775f28ee392d49102511\": container with ID starting with 5935d2652e6a2b3f3bc191cb5f9396949c0c35dde1d6775f28ee392d49102511 not found: ID does not exist" Nov 01 16:11:09.517487 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.517449 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-80c4c8-5779b74587-k4gkg" Nov 01 16:11:09.535674 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.535623 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnc7b\" (UniqueName: \"kubernetes.io/projected/568c7dd7-469a-42a0-bb33-85167782b280-kube-api-access-xnc7b\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:09.535674 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.535661 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/568c7dd7-469a-42a0-bb33-85167782b280-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:09.558980 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.558941 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-80c4c8-75974fbc49-zqjkh"] Nov 01 16:11:09.566794 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.566698 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-80c4c8-75974fbc49-zqjkh"] Nov 01 16:11:09.636460 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.636426 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d6b57c1-aa75-406f-aa95-f455f8690078-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d6b57c1-aa75-406f-aa95-f455f8690078-plugins-config-volume\") pod \"0d6b57c1-aa75-406f-aa95-f455f8690078\" (UID: \"0d6b57c1-aa75-406f-aa95-f455f8690078\") " Nov 01 16:11:09.636622 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.636555 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d6b57c1-aa75-406f-aa95-f455f8690078-kube-api-access-hksd6\" (UniqueName: \"kubernetes.io/projected/0d6b57c1-aa75-406f-aa95-f455f8690078-kube-api-access-hksd6\") pod \"0d6b57c1-aa75-406f-aa95-f455f8690078\" (UID: \"0d6b57c1-aa75-406f-aa95-f455f8690078\") " Nov 01 16:11:09.636876 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.636850 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d6b57c1-aa75-406f-aa95-f455f8690078-plugins-config-volume" pod "0d6b57c1-aa75-406f-aa95-f455f8690078" (UID: "0d6b57c1-aa75-406f-aa95-f455f8690078"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:11:09.637459 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.637437 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d6b57c1-aa75-406f-aa95-f455f8690078-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:09.638788 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.638760 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d6b57c1-aa75-406f-aa95-f455f8690078-kube-api-access-hksd6" pod "0d6b57c1-aa75-406f-aa95-f455f8690078" (UID: "0d6b57c1-aa75-406f-aa95-f455f8690078"). InnerVolumeSpecName "kube-api-access-hksd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:11:09.739595 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.739555 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hksd6\" (UniqueName: \"kubernetes.io/projected/0d6b57c1-aa75-406f-aa95-f455f8690078-kube-api-access-hksd6\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:09.798929 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.797734 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv"] Nov 01 16:11:09.799170 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.799149 4363 cpu_manager.go:414] "containerMap: removing container" podUID="f1e164ab-e940-44e0-8829-8626e4162a9f" containerName="main" Nov 01 16:11:09.799260 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.799173 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1e164ab-e940-44e0-8829-8626e4162a9f" containerName="main" Nov 01 16:11:09.799260 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.799197 4363 cpu_manager.go:414] "containerMap: removing container" podUID="568c7dd7-469a-42a0-bb33-85167782b280" containerName="payload-processing" Nov 01 16:11:09.799260 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.799204 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="568c7dd7-469a-42a0-bb33-85167782b280" containerName="payload-processing" Nov 01 16:11:09.799260 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.799252 4363 cpu_manager.go:414] "containerMap: removing container" podUID="0d6b57c1-aa75-406f-aa95-f455f8690078" containerName="payload-pre-processing" Nov 01 16:11:09.799451 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.799261 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d6b57c1-aa75-406f-aa95-f455f8690078" containerName="payload-pre-processing" Nov 01 16:11:09.799504 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.799459 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1e164ab-e940-44e0-8829-8626e4162a9f" containerName="main" Nov 01 16:11:09.799504 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.799476 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="568c7dd7-469a-42a0-bb33-85167782b280" containerName="payload-processing" Nov 01 16:11:09.799504 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.799489 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d6b57c1-aa75-406f-aa95-f455f8690078" containerName="payload-pre-processing" Nov 01 16:11:09.816160 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.816113 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv"] Nov 01 16:11:09.816332 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.816234 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" Nov 01 16:11:09.825053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.824990 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f5dd93\"/\"openshift-service-ca.crt\"" Nov 01 16:11:09.825272 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.825235 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f5dd93\"/\"default-dockercfg-4jb7w\"" Nov 01 16:11:09.825369 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.825315 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f5dd93\"/\"test-model-f5dd93-kserve-self-signed-certs\"" Nov 01 16:11:09.825369 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.825354 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f5dd93\"/\"kube-root-ca.crt\"" Nov 01 16:11:09.943183 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.943100 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a15aa191-e376-490a-8763-16cdd7685b5c-tls-certs\") pod \"test-model-f5dd93-kserve-9b467c7d9-6tbdv\" (UID: \"a15aa191-e376-490a-8763-16cdd7685b5c\") " pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" Nov 01 16:11:09.943183 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.943139 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a15aa191-e376-490a-8763-16cdd7685b5c-model-cache\") pod \"test-model-f5dd93-kserve-9b467c7d9-6tbdv\" (UID: \"a15aa191-e376-490a-8763-16cdd7685b5c\") " pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" Nov 01 16:11:09.943462 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.943440 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a15aa191-e376-490a-8763-16cdd7685b5c-home\") pod \"test-model-f5dd93-kserve-9b467c7d9-6tbdv\" (UID: \"a15aa191-e376-490a-8763-16cdd7685b5c\") " pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" Nov 01 16:11:09.943542 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.943521 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a15aa191-e376-490a-8763-16cdd7685b5c-tmp-dir\") pod \"test-model-f5dd93-kserve-9b467c7d9-6tbdv\" (UID: \"a15aa191-e376-490a-8763-16cdd7685b5c\") " pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" Nov 01 16:11:09.943751 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.943720 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4t5m\" (UniqueName: \"kubernetes.io/projected/a15aa191-e376-490a-8763-16cdd7685b5c-kube-api-access-s4t5m\") pod \"test-model-f5dd93-kserve-9b467c7d9-6tbdv\" (UID: \"a15aa191-e376-490a-8763-16cdd7685b5c\") " pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" Nov 01 16:11:09.943934 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:09.943916 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a15aa191-e376-490a-8763-16cdd7685b5c-dshm\") pod \"test-model-f5dd93-kserve-9b467c7d9-6tbdv\" (UID: \"a15aa191-e376-490a-8763-16cdd7685b5c\") " pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" Nov 01 16:11:10.072041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:10.068164 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a15aa191-e376-490a-8763-16cdd7685b5c-home\") pod \"test-model-f5dd93-kserve-9b467c7d9-6tbdv\" (UID: \"a15aa191-e376-490a-8763-16cdd7685b5c\") " pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" Nov 01 16:11:10.072041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:10.068245 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a15aa191-e376-490a-8763-16cdd7685b5c-tmp-dir\") pod \"test-model-f5dd93-kserve-9b467c7d9-6tbdv\" (UID: \"a15aa191-e376-490a-8763-16cdd7685b5c\") " pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" Nov 01 16:11:10.072041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:10.068353 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4t5m\" (UniqueName: \"kubernetes.io/projected/a15aa191-e376-490a-8763-16cdd7685b5c-kube-api-access-s4t5m\") pod \"test-model-f5dd93-kserve-9b467c7d9-6tbdv\" (UID: \"a15aa191-e376-490a-8763-16cdd7685b5c\") " pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" Nov 01 16:11:10.072041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:10.068532 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a15aa191-e376-490a-8763-16cdd7685b5c-dshm\") pod \"test-model-f5dd93-kserve-9b467c7d9-6tbdv\" (UID: \"a15aa191-e376-490a-8763-16cdd7685b5c\") " pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" Nov 01 16:11:10.072041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:10.068599 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a15aa191-e376-490a-8763-16cdd7685b5c-tls-certs\") pod \"test-model-f5dd93-kserve-9b467c7d9-6tbdv\" (UID: \"a15aa191-e376-490a-8763-16cdd7685b5c\") " pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" Nov 01 16:11:10.072041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:10.068625 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a15aa191-e376-490a-8763-16cdd7685b5c-model-cache\") pod \"test-model-f5dd93-kserve-9b467c7d9-6tbdv\" (UID: \"a15aa191-e376-490a-8763-16cdd7685b5c\") " pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" Nov 01 16:11:10.072041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:10.069247 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a15aa191-e376-490a-8763-16cdd7685b5c-model-cache\") pod \"test-model-f5dd93-kserve-9b467c7d9-6tbdv\" (UID: \"a15aa191-e376-490a-8763-16cdd7685b5c\") " pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" Nov 01 16:11:10.072041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:10.069482 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a15aa191-e376-490a-8763-16cdd7685b5c-home\") pod \"test-model-f5dd93-kserve-9b467c7d9-6tbdv\" (UID: \"a15aa191-e376-490a-8763-16cdd7685b5c\") " pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" Nov 01 16:11:10.072041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:10.069684 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a15aa191-e376-490a-8763-16cdd7685b5c-tmp-dir\") pod \"test-model-f5dd93-kserve-9b467c7d9-6tbdv\" (UID: \"a15aa191-e376-490a-8763-16cdd7685b5c\") " pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" Nov 01 16:11:10.139141 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:10.131630 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a15aa191-e376-490a-8763-16cdd7685b5c-tls-certs\") pod \"test-model-f5dd93-kserve-9b467c7d9-6tbdv\" (UID: \"a15aa191-e376-490a-8763-16cdd7685b5c\") " pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" Nov 01 16:11:10.139404 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:10.136138 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4t5m\" (UniqueName: \"kubernetes.io/projected/a15aa191-e376-490a-8763-16cdd7685b5c-kube-api-access-s4t5m\") pod \"test-model-f5dd93-kserve-9b467c7d9-6tbdv\" (UID: \"a15aa191-e376-490a-8763-16cdd7685b5c\") " pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" Nov 01 16:11:10.157101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:10.148289 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a15aa191-e376-490a-8763-16cdd7685b5c-dshm\") pod \"test-model-f5dd93-kserve-9b467c7d9-6tbdv\" (UID: \"a15aa191-e376-490a-8763-16cdd7685b5c\") " pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" Nov 01 16:11:10.352471 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:10.352392 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-80c4c8-5779b74587-k4gkg" event={"ID":"0d6b57c1-aa75-406f-aa95-f455f8690078","Type":"ContainerDied","Data":"dbed6a283a6c11a2163bb80693edd2fb807aa427ea5477e84e5ee68009b352c1"} Nov 01 16:11:10.353094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:10.352504 4363 scope.go:122] "RemoveContainer" containerID="5d429f533209d90a11bb526cdbb9aaaa0565efec86ae650726e1a4273e8e835f" Nov 01 16:11:10.353094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:10.352666 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-80c4c8-5779b74587-k4gkg" Nov 01 16:11:10.385905 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:10.385864 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-80c4c8-5779b74587-k4gkg"] Nov 01 16:11:10.392440 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:10.392415 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-80c4c8-5779b74587-k4gkg"] Nov 01 16:11:10.436843 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:10.436808 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" Nov 01 16:11:10.738571 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:11:10.738528 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda15aa191_e376_490a_8763_16cdd7685b5c.slice/crio-99e1d2294aaa2b032bf227f67c32c73f305352abf7299e6d67ba6f278761cf1d WatchSource:0}: Error finding container 99e1d2294aaa2b032bf227f67c32c73f305352abf7299e6d67ba6f278761cf1d: Status 404 returned error can't find the container with id 99e1d2294aaa2b032bf227f67c32c73f305352abf7299e6d67ba6f278761cf1d Nov 01 16:11:10.739200 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:10.739177 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv"] Nov 01 16:11:10.803135 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:10.803103 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d6b57c1-aa75-406f-aa95-f455f8690078" path="/var/lib/kubelet/pods/0d6b57c1-aa75-406f-aa95-f455f8690078/volumes" Nov 01 16:11:10.803679 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:10.803663 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="568c7dd7-469a-42a0-bb33-85167782b280" path="/var/lib/kubelet/pods/568c7dd7-469a-42a0-bb33-85167782b280/volumes" Nov 01 16:11:10.804245 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:10.804227 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1e164ab-e940-44e0-8829-8626e4162a9f" path="/var/lib/kubelet/pods/f1e164ab-e940-44e0-8829-8626e4162a9f/volumes" Nov 01 16:11:11.367353 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:11.367317 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" event={"ID":"a15aa191-e376-490a-8763-16cdd7685b5c","Type":"ContainerStarted","Data":"ef485f1fb3ff984f14ae6058a97a1dafe21b4c7319c1dc2a89dfeda2ccf16116"} Nov 01 16:11:11.368047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:11.367975 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" event={"ID":"a15aa191-e376-490a-8763-16cdd7685b5c","Type":"ContainerStarted","Data":"99e1d2294aaa2b032bf227f67c32c73f305352abf7299e6d67ba6f278761cf1d"} Nov 01 16:11:11.387715 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:11.387656 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" podStartSLOduration=2.387637931 podStartE2EDuration="2.387637931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:11:11.384189915 +0000 UTC m=+1404.715927558" watchObservedRunningTime="2026-11-01 16:11:11.387637931 +0000 UTC m=+1404.719375581" Nov 01 16:11:11.437468 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:11.437431 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" Nov 01 16:11:11.627321 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:11.627234 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-26310f-ccf8d9588-wczxf"] Nov 01 16:11:11.661262 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:11.660172 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-26310f-ccf8d9588-wczxf"] Nov 01 16:11:11.661262 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:11.660301 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26310f-ccf8d9588-wczxf" Nov 01 16:11:11.669503 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:11.669472 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-26310f-serving-cert\"" Nov 01 16:11:11.800891 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:11.797716 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42b30c2b-3415-45cb-9120-eefb427210d5-metrics-certs\") pod \"maas-api-e2e-api-b-26310f-ccf8d9588-wczxf\" (UID: \"42b30c2b-3415-45cb-9120-eefb427210d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26310f-ccf8d9588-wczxf" Nov 01 16:11:11.800891 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:11.797980 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42b30c2b-3415-45cb-9120-eefb427210d5-maas-api-tls\") pod \"maas-api-e2e-api-b-26310f-ccf8d9588-wczxf\" (UID: \"42b30c2b-3415-45cb-9120-eefb427210d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26310f-ccf8d9588-wczxf" Nov 01 16:11:11.800891 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:11.798119 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvtsw\" (UniqueName: \"kubernetes.io/projected/42b30c2b-3415-45cb-9120-eefb427210d5-kube-api-access-qvtsw\") pod \"maas-api-e2e-api-b-26310f-ccf8d9588-wczxf\" (UID: \"42b30c2b-3415-45cb-9120-eefb427210d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26310f-ccf8d9588-wczxf" Nov 01 16:11:11.800891 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:11.798166 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42b30c2b-3415-45cb-9120-eefb427210d5-openshift-service-ca\") pod \"maas-api-e2e-api-b-26310f-ccf8d9588-wczxf\" (UID: \"42b30c2b-3415-45cb-9120-eefb427210d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26310f-ccf8d9588-wczxf" Nov 01 16:11:11.906991 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:11.899395 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvtsw\" (UniqueName: \"kubernetes.io/projected/42b30c2b-3415-45cb-9120-eefb427210d5-kube-api-access-qvtsw\") pod \"maas-api-e2e-api-b-26310f-ccf8d9588-wczxf\" (UID: \"42b30c2b-3415-45cb-9120-eefb427210d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26310f-ccf8d9588-wczxf" Nov 01 16:11:11.906991 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:11.899472 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42b30c2b-3415-45cb-9120-eefb427210d5-openshift-service-ca\") pod \"maas-api-e2e-api-b-26310f-ccf8d9588-wczxf\" (UID: \"42b30c2b-3415-45cb-9120-eefb427210d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26310f-ccf8d9588-wczxf" Nov 01 16:11:11.906991 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:11.899562 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42b30c2b-3415-45cb-9120-eefb427210d5-metrics-certs\") pod \"maas-api-e2e-api-b-26310f-ccf8d9588-wczxf\" (UID: \"42b30c2b-3415-45cb-9120-eefb427210d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26310f-ccf8d9588-wczxf" Nov 01 16:11:11.906991 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:11.899851 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42b30c2b-3415-45cb-9120-eefb427210d5-maas-api-tls\") pod \"maas-api-e2e-api-b-26310f-ccf8d9588-wczxf\" (UID: \"42b30c2b-3415-45cb-9120-eefb427210d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26310f-ccf8d9588-wczxf" Nov 01 16:11:11.906991 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:11.902655 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42b30c2b-3415-45cb-9120-eefb427210d5-openshift-service-ca\") pod \"maas-api-e2e-api-b-26310f-ccf8d9588-wczxf\" (UID: \"42b30c2b-3415-45cb-9120-eefb427210d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26310f-ccf8d9588-wczxf" Nov 01 16:11:11.942042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:11.914350 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42b30c2b-3415-45cb-9120-eefb427210d5-maas-api-tls\") pod \"maas-api-e2e-api-b-26310f-ccf8d9588-wczxf\" (UID: \"42b30c2b-3415-45cb-9120-eefb427210d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26310f-ccf8d9588-wczxf" Nov 01 16:11:11.942042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:11.926866 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvtsw\" (UniqueName: \"kubernetes.io/projected/42b30c2b-3415-45cb-9120-eefb427210d5-kube-api-access-qvtsw\") pod \"maas-api-e2e-api-b-26310f-ccf8d9588-wczxf\" (UID: \"42b30c2b-3415-45cb-9120-eefb427210d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26310f-ccf8d9588-wczxf" Nov 01 16:11:11.942042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:11.939730 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42b30c2b-3415-45cb-9120-eefb427210d5-metrics-certs\") pod \"maas-api-e2e-api-b-26310f-ccf8d9588-wczxf\" (UID: \"42b30c2b-3415-45cb-9120-eefb427210d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26310f-ccf8d9588-wczxf" Nov 01 16:11:11.992858 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:11.992410 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26310f-ccf8d9588-wczxf" Nov 01 16:11:12.764189 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:12.759645 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-26310f-ccf8d9588-wczxf"] Nov 01 16:11:13.475450 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:13.475355 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26310f-ccf8d9588-wczxf" event={"ID":"42b30c2b-3415-45cb-9120-eefb427210d5","Type":"ContainerStarted","Data":"ac67bfe29eb9783505b353966080b1488f7f0003a5a9299c96e2e709e25f7821"} Nov 01 16:11:13.475450 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:13.475413 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26310f-ccf8d9588-wczxf" event={"ID":"42b30c2b-3415-45cb-9120-eefb427210d5","Type":"ContainerStarted","Data":"17303043d02f389f6928993cd257ad335e87e3b5883034b04096d1caed631f26"} Nov 01 16:11:13.475688 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:13.475479 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26310f-ccf8d9588-wczxf" Nov 01 16:11:13.543105 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:13.540390 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26310f-ccf8d9588-wczxf" podStartSLOduration=2.149918905 podStartE2EDuration="2.54036436s" totalImagesPullingTime="390.445455ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:11:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:11:13.497162156 +0000 UTC m=+1406.828899807" watchObservedRunningTime="2026-11-01 16:11:13.54036436 +0000 UTC m=+1406.872102011" Nov 01 16:11:15.226050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:15.215662 4363 patch_prober.go:29] interesting pod/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:11:15.226050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:15.215763 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" podUID="ef9cc82b-caf7-4108-b80b-d10135f512b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:15.537424 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:15.537337 4363 generic.go:356] "Generic (PLEG): container finished" podID="eb765068-ab2a-42f7-9f46-66cfa9745814" containerID="a63f5e962708e8ed721205a200627275b54ed1319bcb23a29768210f81cbf0ce" exitCode=0 Nov 01 16:11:15.537424 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:15.537401 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" event={"ID":"eb765068-ab2a-42f7-9f46-66cfa9745814","Type":"ContainerDied","Data":"a63f5e962708e8ed721205a200627275b54ed1319bcb23a29768210f81cbf0ce"} Nov 01 16:11:15.547082 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:15.538845 4363 scope.go:122] "RemoveContainer" containerID="a63f5e962708e8ed721205a200627275b54ed1319bcb23a29768210f81cbf0ce" Nov 01 16:11:15.554692 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:15.554644 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:11:15.554881 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:15.554734 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:11:15.554881 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:15.554783 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:11:16.548871 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:16.548838 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" event={"ID":"eb765068-ab2a-42f7-9f46-66cfa9745814","Type":"ContainerStarted","Data":"6cddaca8108a4980ee2009dee85d2fc4fae942fd1c85c0cffcd416842260b8ff"} Nov 01 16:11:16.556690 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:16.556653 4363 generic.go:356] "Generic (PLEG): container finished" podID="ef9cc82b-caf7-4108-b80b-d10135f512b1" containerID="a2fdb4ea49a46cd17108656febd6573d9ebf7f3d7d40960311021f074ae2fca6" exitCode=0 Nov 01 16:11:16.556860 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:16.556721 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" event={"ID":"ef9cc82b-caf7-4108-b80b-d10135f512b1","Type":"ContainerDied","Data":"a2fdb4ea49a46cd17108656febd6573d9ebf7f3d7d40960311021f074ae2fca6"} Nov 01 16:11:16.557324 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:16.557298 4363 scope.go:122] "RemoveContainer" containerID="a2fdb4ea49a46cd17108656febd6573d9ebf7f3d7d40960311021f074ae2fca6" Nov 01 16:11:16.567473 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:16.567427 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:11:16.572204 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:16.572167 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:11:16.572455 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:16.572432 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:11:16.844449 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:16.844423 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" Nov 01 16:11:16.846210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:16.846183 4363 patch_prober.go:29] interesting pod/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 01 16:11:16.846430 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:16.846402 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" podUID="eb765068-ab2a-42f7-9f46-66cfa9745814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 01 16:11:17.640261 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:17.640210 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" event={"ID":"ef9cc82b-caf7-4108-b80b-d10135f512b1","Type":"ContainerStarted","Data":"c82a018a2f289f1106c4d3a0ef4e70cdd256a70d2b376e251eda93cb845ea2aa"} Nov 01 16:11:17.844875 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:17.844842 4363 patch_prober.go:29] interesting pod/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 01 16:11:17.845056 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:17.844894 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" podUID="eb765068-ab2a-42f7-9f46-66cfa9745814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 01 16:11:18.360168 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:11:18.360130 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:11:18.452671 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:18.452628 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" Nov 01 16:11:18.453112 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:18.453084 4363 patch_prober.go:29] interesting pod/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 01 16:11:18.453209 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:18.453145 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" podUID="ef9cc82b-caf7-4108-b80b-d10135f512b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 01 16:11:18.858725 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:18.850356 4363 patch_prober.go:29] interesting pod/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 01 16:11:18.858725 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:18.850437 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" podUID="eb765068-ab2a-42f7-9f46-66cfa9745814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 01 16:11:19.274636 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:19.274562 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" Nov 01 16:11:19.453396 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:19.453363 4363 patch_prober.go:29] interesting pod/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 01 16:11:19.453567 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:19.453443 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" podUID="ef9cc82b-caf7-4108-b80b-d10135f512b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 01 16:11:19.537286 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:19.537205 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26310f-ccf8d9588-wczxf" Nov 01 16:11:19.854051 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:19.845413 4363 patch_prober.go:29] interesting pod/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 01 16:11:19.854051 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:19.845474 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" podUID="eb765068-ab2a-42f7-9f46-66cfa9745814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 01 16:11:20.444050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:20.443193 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" Nov 01 16:11:20.453467 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:20.453436 4363 patch_prober.go:29] interesting pod/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 01 16:11:20.453605 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:20.453493 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" podUID="ef9cc82b-caf7-4108-b80b-d10135f512b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 01 16:11:20.456803 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:20.456783 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" Nov 01 16:11:20.868105 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:20.868036 4363 patch_prober.go:29] interesting pod/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 01 16:11:20.868105 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:20.868094 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" podUID="eb765068-ab2a-42f7-9f46-66cfa9745814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 01 16:11:20.943041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:20.938221 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" Nov 01 16:11:21.460038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:21.459453 4363 patch_prober.go:29] interesting pod/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 01 16:11:21.460038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:21.459517 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" podUID="ef9cc82b-caf7-4108-b80b-d10135f512b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 01 16:11:21.847106 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:21.846443 4363 patch_prober.go:29] interesting pod/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 01 16:11:21.847106 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:21.846501 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" podUID="eb765068-ab2a-42f7-9f46-66cfa9745814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 01 16:11:22.580929 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:22.575459 4363 patch_prober.go:29] interesting pod/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 01 16:11:22.580929 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:22.575540 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" podUID="ef9cc82b-caf7-4108-b80b-d10135f512b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 01 16:11:22.857244 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:22.857189 4363 patch_prober.go:29] interesting pod/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 01 16:11:22.931128 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:22.931071 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" podUID="eb765068-ab2a-42f7-9f46-66cfa9745814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 01 16:11:23.457042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:23.454218 4363 patch_prober.go:29] interesting pod/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 01 16:11:23.457042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:23.454299 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" podUID="ef9cc82b-caf7-4108-b80b-d10135f512b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 01 16:11:23.874095 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:23.874060 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" Nov 01 16:11:23.888784 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:23.888082 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" Nov 01 16:11:24.452942 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:24.452906 4363 patch_prober.go:29] interesting pod/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 01 16:11:24.453164 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:24.452966 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" podUID="ef9cc82b-caf7-4108-b80b-d10135f512b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 01 16:11:25.458362 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:25.458330 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" Nov 01 16:11:25.458929 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:25.458910 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" Nov 01 16:11:25.459901 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:25.459876 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" Nov 01 16:11:32.887059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:32.887008 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2ce08a-5bc49b65bf-qn2rr"] Nov 01 16:11:32.898382 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:32.898358 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2ce08a-5bc49b65bf-qn2rr" Nov 01 16:11:32.900851 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:32.900806 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2ce08a-5bc49b65bf-qn2rr"] Nov 01 16:11:32.902787 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:32.902764 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-2ce08a\"" Nov 01 16:11:32.932801 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:32.932659 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2ce08a-69744fddc5-cfpqd"] Nov 01 16:11:32.945455 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:32.945428 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2ce08a-69744fddc5-cfpqd"] Nov 01 16:11:32.945647 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:32.945551 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2ce08a-69744fddc5-cfpqd" Nov 01 16:11:32.951253 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:32.949173 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-2ce08a-dockercfg-4pjbn\"" Nov 01 16:11:33.026428 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:33.026390 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b44172c-d6bb-4766-b1da-3343a3b46cc6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2ce08a-69744fddc5-cfpqd\" (UID: \"1b44172c-d6bb-4766-b1da-3343a3b46cc6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2ce08a-69744fddc5-cfpqd" Nov 01 16:11:33.026780 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:33.026751 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51ac18a7-1176-48ff-9a97-c5b1dd3d490b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2ce08a-5bc49b65bf-qn2rr\" (UID: \"51ac18a7-1176-48ff-9a97-c5b1dd3d490b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2ce08a-5bc49b65bf-qn2rr" Nov 01 16:11:33.026962 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:33.026857 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xf65\" (UniqueName: \"kubernetes.io/projected/51ac18a7-1176-48ff-9a97-c5b1dd3d490b-kube-api-access-6xf65\") pod \"payload-processing-e2e-inf-b-2ce08a-5bc49b65bf-qn2rr\" (UID: \"51ac18a7-1176-48ff-9a97-c5b1dd3d490b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2ce08a-5bc49b65bf-qn2rr" Nov 01 16:11:33.026962 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:33.026892 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r7tp\" (UniqueName: \"kubernetes.io/projected/1b44172c-d6bb-4766-b1da-3343a3b46cc6-kube-api-access-5r7tp\") pod \"payload-pre-processing-e2e-inf-b-2ce08a-69744fddc5-cfpqd\" (UID: \"1b44172c-d6bb-4766-b1da-3343a3b46cc6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2ce08a-69744fddc5-cfpqd" Nov 01 16:11:33.139089 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:33.138166 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xf65\" (UniqueName: \"kubernetes.io/projected/51ac18a7-1176-48ff-9a97-c5b1dd3d490b-kube-api-access-6xf65\") pod \"payload-processing-e2e-inf-b-2ce08a-5bc49b65bf-qn2rr\" (UID: \"51ac18a7-1176-48ff-9a97-c5b1dd3d490b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2ce08a-5bc49b65bf-qn2rr" Nov 01 16:11:33.139089 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:33.138219 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r7tp\" (UniqueName: \"kubernetes.io/projected/1b44172c-d6bb-4766-b1da-3343a3b46cc6-kube-api-access-5r7tp\") pod \"payload-pre-processing-e2e-inf-b-2ce08a-69744fddc5-cfpqd\" (UID: \"1b44172c-d6bb-4766-b1da-3343a3b46cc6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2ce08a-69744fddc5-cfpqd" Nov 01 16:11:33.139089 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:33.138328 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b44172c-d6bb-4766-b1da-3343a3b46cc6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2ce08a-69744fddc5-cfpqd\" (UID: \"1b44172c-d6bb-4766-b1da-3343a3b46cc6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2ce08a-69744fddc5-cfpqd" Nov 01 16:11:33.139089 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:33.138654 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51ac18a7-1176-48ff-9a97-c5b1dd3d490b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2ce08a-5bc49b65bf-qn2rr\" (UID: \"51ac18a7-1176-48ff-9a97-c5b1dd3d490b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2ce08a-5bc49b65bf-qn2rr" Nov 01 16:11:33.158122 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:33.139587 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51ac18a7-1176-48ff-9a97-c5b1dd3d490b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2ce08a-5bc49b65bf-qn2rr\" (UID: \"51ac18a7-1176-48ff-9a97-c5b1dd3d490b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2ce08a-5bc49b65bf-qn2rr" Nov 01 16:11:33.158122 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:33.140721 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b44172c-d6bb-4766-b1da-3343a3b46cc6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2ce08a-69744fddc5-cfpqd\" (UID: \"1b44172c-d6bb-4766-b1da-3343a3b46cc6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2ce08a-69744fddc5-cfpqd" Nov 01 16:11:33.174953 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:33.174884 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xf65\" (UniqueName: \"kubernetes.io/projected/51ac18a7-1176-48ff-9a97-c5b1dd3d490b-kube-api-access-6xf65\") pod \"payload-processing-e2e-inf-b-2ce08a-5bc49b65bf-qn2rr\" (UID: \"51ac18a7-1176-48ff-9a97-c5b1dd3d490b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2ce08a-5bc49b65bf-qn2rr" Nov 01 16:11:33.245879 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:33.245840 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r7tp\" (UniqueName: \"kubernetes.io/projected/1b44172c-d6bb-4766-b1da-3343a3b46cc6-kube-api-access-5r7tp\") pod \"payload-pre-processing-e2e-inf-b-2ce08a-69744fddc5-cfpqd\" (UID: \"1b44172c-d6bb-4766-b1da-3343a3b46cc6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2ce08a-69744fddc5-cfpqd" Nov 01 16:11:33.265039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:33.251308 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2ce08a-5bc49b65bf-qn2rr" Nov 01 16:11:33.291626 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:33.286429 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2ce08a-69744fddc5-cfpqd" Nov 01 16:11:33.755079 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:33.755044 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-022f62-6b6cb577c9-w46bj"] Nov 01 16:11:33.769435 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:33.769396 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-022f62-6b68c9f45f-ls8hp"] Nov 01 16:11:33.855342 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:33.855309 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2ce08a-5bc49b65bf-qn2rr"] Nov 01 16:11:33.855935 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:11:33.855904 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51ac18a7_1176_48ff_9a97_c5b1dd3d490b.slice/crio-dcf84b2acb3dd454d9207daedb321c2867c201d86af133b1771e29349099b448 WatchSource:0}: Error finding container dcf84b2acb3dd454d9207daedb321c2867c201d86af133b1771e29349099b448: Status 404 returned error can't find the container with id dcf84b2acb3dd454d9207daedb321c2867c201d86af133b1771e29349099b448 Nov 01 16:11:33.896041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:33.876994 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2ce08a-69744fddc5-cfpqd"] Nov 01 16:11:34.142970 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:34.141455 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2ce08a-5bc49b65bf-qn2rr" event={"ID":"51ac18a7-1176-48ff-9a97-c5b1dd3d490b","Type":"ContainerStarted","Data":"16ba0ca48bb170bcdf14b41f60ebd4e462b2a55dca2edb4c69733bd3630a1a46"} Nov 01 16:11:34.142970 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:34.141529 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2ce08a-5bc49b65bf-qn2rr" event={"ID":"51ac18a7-1176-48ff-9a97-c5b1dd3d490b","Type":"ContainerStarted","Data":"dcf84b2acb3dd454d9207daedb321c2867c201d86af133b1771e29349099b448"} Nov 01 16:11:34.142970 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:34.142909 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2ce08a-5bc49b65bf-qn2rr" Nov 01 16:11:34.167863 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:34.166322 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2ce08a-69744fddc5-cfpqd" event={"ID":"1b44172c-d6bb-4766-b1da-3343a3b46cc6","Type":"ContainerStarted","Data":"5e11a0c126638284753185ee7abfb5336ef8e421c3331469958393c8d1859cfa"} Nov 01 16:11:34.167863 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:34.166387 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2ce08a-69744fddc5-cfpqd" event={"ID":"1b44172c-d6bb-4766-b1da-3343a3b46cc6","Type":"ContainerStarted","Data":"e5c32e592988463703bb6327a13e7dc6543d7caa6e89901e931671f9d6402c6d"} Nov 01 16:11:34.167863 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:34.167792 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2ce08a-69744fddc5-cfpqd" Nov 01 16:11:34.198434 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:34.198320 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-2ce08a-5bc49b65bf-qn2rr" podStartSLOduration=2.19830044 podStartE2EDuration="2.19830044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:11:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:11:34.163883743 +0000 UTC m=+1427.495621394" watchObservedRunningTime="2026-11-01 16:11:34.19830044 +0000 UTC m=+1427.530038091" Nov 01 16:11:34.229074 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:34.211302 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2ce08a-69744fddc5-cfpqd" podStartSLOduration=2.211284272 podStartE2EDuration="2.211284272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:11:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:11:34.186634027 +0000 UTC m=+1427.518371677" watchObservedRunningTime="2026-11-01 16:11:34.211284272 +0000 UTC m=+1427.543021924" Nov 01 16:11:38.756213 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:38.756175 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-022f62-6b6cb577c9-w46bj" podUID="ea6d9e7a-1a13-4ead-9cb2-c3408313dd09" containerName="payload-processing" containerID="cri-o://086287d47ac13eebc9a7aa898b2ad79ea2c0adf687902102e7867023bd9552bc" gracePeriod=25 Nov 01 16:11:38.769904 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:38.769872 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-022f62-6b68c9f45f-ls8hp" podUID="067dcb6b-1a6d-4818-b42d-9e3617e2302c" containerName="payload-pre-processing" containerID="cri-o://da77b63b1905450ae9d679325b0b975eabcecd922378298d69e5c7ade35272f4" gracePeriod=25 Nov 01 16:11:39.120512 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:39.120447 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-022f62-6b68c9f45f-ls8hp" Nov 01 16:11:39.214365 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:39.214341 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-022f62-6b6cb577c9-w46bj" Nov 01 16:11:39.218888 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:39.218858 4363 generic.go:356] "Generic (PLEG): container finished" podID="ea6d9e7a-1a13-4ead-9cb2-c3408313dd09" containerID="086287d47ac13eebc9a7aa898b2ad79ea2c0adf687902102e7867023bd9552bc" exitCode=0 Nov 01 16:11:39.219223 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:39.219202 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-022f62-6b6cb577c9-w46bj" event={"ID":"ea6d9e7a-1a13-4ead-9cb2-c3408313dd09","Type":"ContainerDied","Data":"086287d47ac13eebc9a7aa898b2ad79ea2c0adf687902102e7867023bd9552bc"} Nov 01 16:11:39.219364 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:39.219345 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-022f62-6b6cb577c9-w46bj" event={"ID":"ea6d9e7a-1a13-4ead-9cb2-c3408313dd09","Type":"ContainerDied","Data":"356c2ae8f22f0949612771e8407d9a4265f78b687ed31db782d6e164068dba3b"} Nov 01 16:11:39.219439 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:39.219384 4363 scope.go:122] "RemoveContainer" containerID="086287d47ac13eebc9a7aa898b2ad79ea2c0adf687902102e7867023bd9552bc" Nov 01 16:11:39.219503 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:39.219214 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-022f62-6b6cb577c9-w46bj" Nov 01 16:11:39.222086 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:39.221818 4363 generic.go:356] "Generic (PLEG): container finished" podID="067dcb6b-1a6d-4818-b42d-9e3617e2302c" containerID="da77b63b1905450ae9d679325b0b975eabcecd922378298d69e5c7ade35272f4" exitCode=0 Nov 01 16:11:39.222086 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:39.221889 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-022f62-6b68c9f45f-ls8hp" event={"ID":"067dcb6b-1a6d-4818-b42d-9e3617e2302c","Type":"ContainerDied","Data":"da77b63b1905450ae9d679325b0b975eabcecd922378298d69e5c7ade35272f4"} Nov 01 16:11:39.222086 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:39.221930 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-022f62-6b68c9f45f-ls8hp" event={"ID":"067dcb6b-1a6d-4818-b42d-9e3617e2302c","Type":"ContainerDied","Data":"2eaeb656b7269da709a52b21f437662be80699a6b2b12ba12979507639f66afe"} Nov 01 16:11:39.222086 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:39.222000 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-022f62-6b68c9f45f-ls8hp" Nov 01 16:11:39.230077 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:39.229845 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/067dcb6b-1a6d-4818-b42d-9e3617e2302c-kube-api-access-nrpsz\" (UniqueName: \"kubernetes.io/projected/067dcb6b-1a6d-4818-b42d-9e3617e2302c-kube-api-access-nrpsz\") pod \"067dcb6b-1a6d-4818-b42d-9e3617e2302c\" (UID: \"067dcb6b-1a6d-4818-b42d-9e3617e2302c\") " Nov 01 16:11:39.230982 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:39.230376 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/067dcb6b-1a6d-4818-b42d-9e3617e2302c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/067dcb6b-1a6d-4818-b42d-9e3617e2302c-plugins-config-volume\") pod \"067dcb6b-1a6d-4818-b42d-9e3617e2302c\" (UID: \"067dcb6b-1a6d-4818-b42d-9e3617e2302c\") " Nov 01 16:11:39.230982 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:39.230947 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/067dcb6b-1a6d-4818-b42d-9e3617e2302c-plugins-config-volume" pod "067dcb6b-1a6d-4818-b42d-9e3617e2302c" (UID: "067dcb6b-1a6d-4818-b42d-9e3617e2302c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:11:39.231283 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:39.231260 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/067dcb6b-1a6d-4818-b42d-9e3617e2302c-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:39.232837 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:39.232762 4363 scope.go:122] "RemoveContainer" containerID="086287d47ac13eebc9a7aa898b2ad79ea2c0adf687902102e7867023bd9552bc" Nov 01 16:11:39.232837 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:39.232792 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/067dcb6b-1a6d-4818-b42d-9e3617e2302c-kube-api-access-nrpsz" pod "067dcb6b-1a6d-4818-b42d-9e3617e2302c" (UID: "067dcb6b-1a6d-4818-b42d-9e3617e2302c"). InnerVolumeSpecName "kube-api-access-nrpsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:11:39.233608 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:11:39.233431 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"086287d47ac13eebc9a7aa898b2ad79ea2c0adf687902102e7867023bd9552bc\": container with ID starting with 086287d47ac13eebc9a7aa898b2ad79ea2c0adf687902102e7867023bd9552bc not found: ID does not exist" containerID="086287d47ac13eebc9a7aa898b2ad79ea2c0adf687902102e7867023bd9552bc" Nov 01 16:11:39.233608 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:39.233467 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"086287d47ac13eebc9a7aa898b2ad79ea2c0adf687902102e7867023bd9552bc"} err="failed to get container status \"086287d47ac13eebc9a7aa898b2ad79ea2c0adf687902102e7867023bd9552bc\": rpc error: code = NotFound desc = could not find container \"086287d47ac13eebc9a7aa898b2ad79ea2c0adf687902102e7867023bd9552bc\": container with ID starting with 086287d47ac13eebc9a7aa898b2ad79ea2c0adf687902102e7867023bd9552bc not found: ID does not exist" Nov 01 16:11:39.233608 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:39.233488 4363 scope.go:122] "RemoveContainer" containerID="da77b63b1905450ae9d679325b0b975eabcecd922378298d69e5c7ade35272f4" Nov 01 16:11:39.247314 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:39.247276 4363 scope.go:122] "RemoveContainer" containerID="da77b63b1905450ae9d679325b0b975eabcecd922378298d69e5c7ade35272f4" Nov 01 16:11:39.247586 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:11:39.247567 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da77b63b1905450ae9d679325b0b975eabcecd922378298d69e5c7ade35272f4\": container with ID starting with da77b63b1905450ae9d679325b0b975eabcecd922378298d69e5c7ade35272f4 not found: ID does not exist" containerID="da77b63b1905450ae9d679325b0b975eabcecd922378298d69e5c7ade35272f4" Nov 01 16:11:39.247721 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:39.247591 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da77b63b1905450ae9d679325b0b975eabcecd922378298d69e5c7ade35272f4"} err="failed to get container status \"da77b63b1905450ae9d679325b0b975eabcecd922378298d69e5c7ade35272f4\": rpc error: code = NotFound desc = could not find container \"da77b63b1905450ae9d679325b0b975eabcecd922378298d69e5c7ade35272f4\": container with ID starting with da77b63b1905450ae9d679325b0b975eabcecd922378298d69e5c7ade35272f4 not found: ID does not exist" Nov 01 16:11:39.332908 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:39.332839 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea6d9e7a-1a13-4ead-9cb2-c3408313dd09-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea6d9e7a-1a13-4ead-9cb2-c3408313dd09-plugins-config-volume\") pod \"ea6d9e7a-1a13-4ead-9cb2-c3408313dd09\" (UID: \"ea6d9e7a-1a13-4ead-9cb2-c3408313dd09\") " Nov 01 16:11:39.333067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:39.332947 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea6d9e7a-1a13-4ead-9cb2-c3408313dd09-kube-api-access-8jct8\" (UniqueName: \"kubernetes.io/projected/ea6d9e7a-1a13-4ead-9cb2-c3408313dd09-kube-api-access-8jct8\") pod \"ea6d9e7a-1a13-4ead-9cb2-c3408313dd09\" (UID: \"ea6d9e7a-1a13-4ead-9cb2-c3408313dd09\") " Nov 01 16:11:39.333250 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:39.333227 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea6d9e7a-1a13-4ead-9cb2-c3408313dd09-plugins-config-volume" pod "ea6d9e7a-1a13-4ead-9cb2-c3408313dd09" (UID: "ea6d9e7a-1a13-4ead-9cb2-c3408313dd09"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:11:39.333760 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:39.333741 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrpsz\" (UniqueName: \"kubernetes.io/projected/067dcb6b-1a6d-4818-b42d-9e3617e2302c-kube-api-access-nrpsz\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:39.333804 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:39.333765 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea6d9e7a-1a13-4ead-9cb2-c3408313dd09-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:39.335111 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:39.335087 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea6d9e7a-1a13-4ead-9cb2-c3408313dd09-kube-api-access-8jct8" pod "ea6d9e7a-1a13-4ead-9cb2-c3408313dd09" (UID: "ea6d9e7a-1a13-4ead-9cb2-c3408313dd09"). InnerVolumeSpecName "kube-api-access-8jct8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:11:39.435480 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:39.435448 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jct8\" (UniqueName: \"kubernetes.io/projected/ea6d9e7a-1a13-4ead-9cb2-c3408313dd09-kube-api-access-8jct8\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:39.634207 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:39.634170 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-022f62-6b6cb577c9-w46bj"] Nov 01 16:11:39.649836 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:39.649747 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-022f62-6b6cb577c9-w46bj"] Nov 01 16:11:39.660675 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:39.660649 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-022f62-6b68c9f45f-ls8hp"] Nov 01 16:11:39.665845 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:39.665818 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-022f62-6b68c9f45f-ls8hp"] Nov 01 16:11:40.802135 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:40.802104 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="067dcb6b-1a6d-4818-b42d-9e3617e2302c" path="/var/lib/kubelet/pods/067dcb6b-1a6d-4818-b42d-9e3617e2302c/volumes" Nov 01 16:11:40.802609 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:40.802594 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea6d9e7a-1a13-4ead-9cb2-c3408313dd09" path="/var/lib/kubelet/pods/ea6d9e7a-1a13-4ead-9cb2-c3408313dd09/volumes" Nov 01 16:11:46.187891 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:46.187861 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2ce08a-5bc49b65bf-qn2rr" Nov 01 16:11:46.188363 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:46.188007 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2ce08a-69744fddc5-cfpqd" Nov 01 16:11:48.312381 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:11:48.312341 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:11:49.037970 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:49.037936 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-166778-77d4bcfc6f-88bhr"] Nov 01 16:11:49.057131 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:49.057098 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-166778-7d8fcb5659-jq7tt"] Nov 01 16:11:50.257148 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:50.257111 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph"] Nov 01 16:11:50.257641 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:50.257431 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" podUID="a77a07cf-0f15-4724-8170-da59b2a1e5f7" containerName="istio-proxy" containerID="cri-o://87c70fd1295e0cc86bea738904f47c071d47b9f01313a765303d9b3e691c0613" gracePeriod=30 Nov 01 16:11:51.393046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:51.392177 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:51.445041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:51.441287 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" podUID="a15aa191-e376-490a-8763-16cdd7685b5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:51.751232 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:51.751125 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" podUID="a15aa191-e376-490a-8763-16cdd7685b5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:52.213129 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:52.194497 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:52.249397 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:52.249337 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:52.761088 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:52.761045 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" podUID="a15aa191-e376-490a-8763-16cdd7685b5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": context deadline exceeded" Nov 01 16:11:52.770189 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:52.765750 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" Nov 01 16:11:52.894373 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:52.894320 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:53.156178 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:53.156129 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:11:53.214038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:53.199247 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:53.300049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:53.279001 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 16:11:53.324170 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:53.324129 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:53.324460 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:53.324438 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:11:53.613701 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:53.613676 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 16:11:53.666189 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:53.665477 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:53.853043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:53.798131 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" podUID="a15aa191-e376-490a-8763-16cdd7685b5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:53.897795 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:53.897671 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:53.898652 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:53.898629 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:11:54.054040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:54.053077 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-166778-77d4bcfc6f-88bhr" podUID="5c1a1dba-28cb-489a-be20-1ae4bdc0ad85" containerName="payload-processing" containerID="cri-o://88eb0febc5b8f8df7440402b2c47891b9da369f21a363a9f3a6441710b497d28" gracePeriod=25 Nov 01 16:11:54.060151 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:54.060083 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-166778-7d8fcb5659-jq7tt" podUID="51609bfa-e393-49c1-a3c3-f93a6097183e" containerName="payload-pre-processing" containerID="cri-o://f50d05c3db4ac9a934b0e526891577d4f671756187de2b0fab7f4c23acca7f1a" gracePeriod=25 Nov 01 16:11:54.163696 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:54.163594 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:54.179231 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:54.179193 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:11:54.255531 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:54.255478 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:54.328398 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:54.328350 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:11:54.798123 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:54.788066 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:54.798123 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:54.789076 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 16:11:54.912076 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:54.902388 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" podUID="a15aa191-e376-490a-8763-16cdd7685b5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": context deadline exceeded" Nov 01 16:11:54.921516 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:54.921474 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:55.017157 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:55.002390 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": context deadline exceeded" Nov 01 16:11:55.068150 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:55.063196 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:55.214039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:55.193495 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:11:55.338288 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:55.295405 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:55.339001 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:55.331105 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:11:55.385189 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:55.379828 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:11:55.741134 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:55.716439 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:55.803043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:55.793613 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:55.928054 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:55.927738 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" podUID="a15aa191-e376-490a-8763-16cdd7685b5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": context deadline exceeded" Nov 01 16:11:55.967394 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:55.950330 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:56.043270 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:56.043180 4363 generic.go:356] "Generic (PLEG): container finished" podID="5c1a1dba-28cb-489a-be20-1ae4bdc0ad85" containerID="88eb0febc5b8f8df7440402b2c47891b9da369f21a363a9f3a6441710b497d28" exitCode=0 Nov 01 16:11:56.043447 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:56.043409 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-166778-77d4bcfc6f-88bhr" event={"ID":"5c1a1dba-28cb-489a-be20-1ae4bdc0ad85","Type":"ContainerDied","Data":"88eb0febc5b8f8df7440402b2c47891b9da369f21a363a9f3a6441710b497d28"} Nov 01 16:11:56.181135 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:56.171706 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-784444-55c46475-d78lt"] Nov 01 16:11:56.198763 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:56.185471 4363 cpu_manager.go:414] "containerMap: removing container" podUID="067dcb6b-1a6d-4818-b42d-9e3617e2302c" containerName="payload-pre-processing" Nov 01 16:11:56.199033 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:56.198987 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="067dcb6b-1a6d-4818-b42d-9e3617e2302c" containerName="payload-pre-processing" Nov 01 16:11:56.199202 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:56.199177 4363 cpu_manager.go:414] "containerMap: removing container" podUID="ea6d9e7a-1a13-4ead-9cb2-c3408313dd09" containerName="payload-processing" Nov 01 16:11:56.448193 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:56.448164 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea6d9e7a-1a13-4ead-9cb2-c3408313dd09" containerName="payload-processing" Nov 01 16:11:56.469041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:56.448803 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="067dcb6b-1a6d-4818-b42d-9e3617e2302c" containerName="payload-pre-processing" Nov 01 16:11:56.495078 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:56.471463 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea6d9e7a-1a13-4ead-9cb2-c3408313dd09" containerName="payload-processing" Nov 01 16:11:56.495078 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:56.425453 4363 generic.go:356] "Generic (PLEG): container finished" podID="51609bfa-e393-49c1-a3c3-f93a6097183e" containerID="f50d05c3db4ac9a934b0e526891577d4f671756187de2b0fab7f4c23acca7f1a" exitCode=0 Nov 01 16:11:56.495078 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:56.356229 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:11:56.495078 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:56.356172 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:11:56.518395 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:56.425885 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:56.521646 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:56.521586 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-166778-7d8fcb5659-jq7tt" event={"ID":"51609bfa-e393-49c1-a3c3-f93a6097183e","Type":"ContainerDied","Data":"f50d05c3db4ac9a934b0e526891577d4f671756187de2b0fab7f4c23acca7f1a"} Nov 01 16:11:56.522052 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:56.521947 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-784444-55c46475-d78lt" Nov 01 16:11:56.554475 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:56.554443 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-784444\"" Nov 01 16:11:56.653781 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:56.653745 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bkkq\" (UniqueName: \"kubernetes.io/projected/59ebabe0-34d2-4102-86de-52bb4d6d04e2-kube-api-access-8bkkq\") pod \"payload-processing-e2e-inf-a-784444-55c46475-d78lt\" (UID: \"59ebabe0-34d2-4102-86de-52bb4d6d04e2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-784444-55c46475-d78lt" Nov 01 16:11:56.655045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:56.655001 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ebabe0-34d2-4102-86de-52bb4d6d04e2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-784444-55c46475-d78lt\" (UID: \"59ebabe0-34d2-4102-86de-52bb4d6d04e2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-784444-55c46475-d78lt" Nov 01 16:11:56.732089 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:56.722638 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:56.779743 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:56.733356 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 16:11:56.779743 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:56.762751 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bkkq\" (UniqueName: \"kubernetes.io/projected/59ebabe0-34d2-4102-86de-52bb4d6d04e2-kube-api-access-8bkkq\") pod \"payload-processing-e2e-inf-a-784444-55c46475-d78lt\" (UID: \"59ebabe0-34d2-4102-86de-52bb4d6d04e2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-784444-55c46475-d78lt" Nov 01 16:11:56.779743 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:56.762822 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ebabe0-34d2-4102-86de-52bb4d6d04e2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-784444-55c46475-d78lt\" (UID: \"59ebabe0-34d2-4102-86de-52bb4d6d04e2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-784444-55c46475-d78lt" Nov 01 16:11:56.822042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:56.788569 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ebabe0-34d2-4102-86de-52bb4d6d04e2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-784444-55c46475-d78lt\" (UID: \"59ebabe0-34d2-4102-86de-52bb4d6d04e2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-784444-55c46475-d78lt" Nov 01 16:11:56.822042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:56.820121 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:56.931057 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:56.930988 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" podUID="a15aa191-e376-490a-8763-16cdd7685b5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:56.994085 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:56.988528 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:57.056850 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:57.042638 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bkkq\" (UniqueName: \"kubernetes.io/projected/59ebabe0-34d2-4102-86de-52bb4d6d04e2-kube-api-access-8bkkq\") pod \"payload-processing-e2e-inf-a-784444-55c46475-d78lt\" (UID: \"59ebabe0-34d2-4102-86de-52bb4d6d04e2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-784444-55c46475-d78lt" Nov 01 16:11:57.221042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:57.216320 4363 patch_prober.go:29] interesting pod/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 01 16:11:57.221042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:57.216389 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" podUID="a77a07cf-0f15-4724-8170-da59b2a1e5f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 01 16:11:57.306129 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:57.298694 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-784444-55c46475-d78lt" Nov 01 16:11:57.530039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:57.504088 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:11:57.865049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:57.840885 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:57.865049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:57.842419 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:11:58.435045 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:11:58.413874 4363 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.602s" Nov 01 16:11:58.435045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:58.413987 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:11:58.435045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:58.414050 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-784444-55c46475-d78lt"] Nov 01 16:11:58.435045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:58.414102 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-784444-79c887957f-gsjc5"] Nov 01 16:11:58.578878 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:58.578836 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:11:58.597537 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:58.597483 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:11:58.751049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:58.750641 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:11:59.147072 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:59.147011 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:59.158677 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:59.154811 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 01 16:11:59.158677 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:59.156242 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" podUID="a15aa191-e376-490a-8763-16cdd7685b5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:59.158677 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:59.157182 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr" podUID="305de5fb-52ed-401a-89fe-b7d802472f5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:59.266416 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:59.266319 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:59.406043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:59.377709 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-784444-79c887957f-gsjc5" Nov 01 16:11:59.482043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:59.406262 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f796692a-c6ab-48b9-a227-00e8af4f24df-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-784444-79c887957f-gsjc5\" (UID: \"f796692a-c6ab-48b9-a227-00e8af4f24df\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-784444-79c887957f-gsjc5" Nov 01 16:11:59.482043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:59.406470 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96mfm\" (UniqueName: \"kubernetes.io/projected/f796692a-c6ab-48b9-a227-00e8af4f24df-kube-api-access-96mfm\") pod \"payload-pre-processing-e2e-inf-a-784444-79c887957f-gsjc5\" (UID: \"f796692a-c6ab-48b9-a227-00e8af4f24df\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-784444-79c887957f-gsjc5" Nov 01 16:11:59.482043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:59.423454 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-784444-dockercfg-whpxn\"" Nov 01 16:11:59.977792 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:59.977755 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:12:00.006386 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:59.980109 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d121-5595856559-g7tk9" podUID="55c3d81b-79de-44cb-9838-8b07da7c911b" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.106:50052\" within 1s: context deadline exceeded" Nov 01 16:12:00.006386 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:11:59.982524 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:12:00.176868 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:00.176834 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f796692a-c6ab-48b9-a227-00e8af4f24df-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-784444-79c887957f-gsjc5\" (UID: \"f796692a-c6ab-48b9-a227-00e8af4f24df\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-784444-79c887957f-gsjc5" Nov 01 16:12:00.185279 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:00.185229 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96mfm\" (UniqueName: \"kubernetes.io/projected/f796692a-c6ab-48b9-a227-00e8af4f24df-kube-api-access-96mfm\") pod \"payload-pre-processing-e2e-inf-a-784444-79c887957f-gsjc5\" (UID: \"f796692a-c6ab-48b9-a227-00e8af4f24df\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-784444-79c887957f-gsjc5" Nov 01 16:12:00.341414 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:00.194935 4363 generic.go:356] "Generic (PLEG): container finished" podID="a77a07cf-0f15-4724-8170-da59b2a1e5f7" containerID="87c70fd1295e0cc86bea738904f47c071d47b9f01313a765303d9b3e691c0613" exitCode=0 Nov 01 16:12:00.341414 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:00.196518 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f796692a-c6ab-48b9-a227-00e8af4f24df-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-784444-79c887957f-gsjc5\" (UID: \"f796692a-c6ab-48b9-a227-00e8af4f24df\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-784444-79c887957f-gsjc5" Nov 01 16:12:00.341414 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:00.222705 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:12:00.341414 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:00.280351 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:12:00.341414 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:00.280743 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" podUID="a15aa191-e376-490a-8763-16cdd7685b5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": context deadline exceeded" Nov 01 16:12:00.386759 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:00.386718 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-166778-7d8fcb5659-jq7tt" Nov 01 16:12:00.485096 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:00.446349 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:12:00.526943 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:00.526896 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96mfm\" (UniqueName: \"kubernetes.io/projected/f796692a-c6ab-48b9-a227-00e8af4f24df-kube-api-access-96mfm\") pod \"payload-pre-processing-e2e-inf-a-784444-79c887957f-gsjc5\" (UID: \"f796692a-c6ab-48b9-a227-00e8af4f24df\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-784444-79c887957f-gsjc5" Nov 01 16:12:00.608940 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:00.608904 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51609bfa-e393-49c1-a3c3-f93a6097183e-kube-api-access-hkmrh\" (UniqueName: \"kubernetes.io/projected/51609bfa-e393-49c1-a3c3-f93a6097183e-kube-api-access-hkmrh\") pod \"51609bfa-e393-49c1-a3c3-f93a6097183e\" (UID: \"51609bfa-e393-49c1-a3c3-f93a6097183e\") " Nov 01 16:12:00.609320 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:00.609290 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51609bfa-e393-49c1-a3c3-f93a6097183e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51609bfa-e393-49c1-a3c3-f93a6097183e-plugins-config-volume\") pod \"51609bfa-e393-49c1-a3c3-f93a6097183e\" (UID: \"51609bfa-e393-49c1-a3c3-f93a6097183e\") " Nov 01 16:12:00.726763 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:12:00.696775 4363 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.764s" Nov 01 16:12:00.726763 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:00.696861 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:12:00.726763 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:00.696915 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" Nov 01 16:12:00.726763 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:00.696938 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-784444-79c887957f-gsjc5"] Nov 01 16:12:00.726763 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:00.696987 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:12:00.726763 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:00.697057 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:12:00.726763 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:00.697092 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" Nov 01 16:12:00.726763 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:00.697231 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 16:12:00.726763 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:00.697249 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" event={"ID":"a77a07cf-0f15-4724-8170-da59b2a1e5f7","Type":"ContainerDied","Data":"87c70fd1295e0cc86bea738904f47c071d47b9f01313a765303d9b3e691c0613"} Nov 01 16:12:00.726763 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:00.697296 4363 scope.go:122] "RemoveContainer" containerID="0285c96e956edbf8d64bb79c684597d677be6dfff08e15028b990bad54fcdbd6" Nov 01 16:12:00.800040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:00.796975 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": context deadline exceeded" Nov 01 16:12:01.074854 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:01.074718 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51609bfa-e393-49c1-a3c3-f93a6097183e-plugins-config-volume" pod "51609bfa-e393-49c1-a3c3-f93a6097183e" (UID: "51609bfa-e393-49c1-a3c3-f93a6097183e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:12:01.440041 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:12:01.332061 4363 logging.go:55] [core] [Channel #3162 SubChannel #3163]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.140:50052", ServerName: "10.134.0.140:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.140:50052: operation was canceled" Nov 01 16:12:01.440041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:01.412338 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51609bfa-e393-49c1-a3c3-f93a6097183e-kube-api-access-hkmrh" pod "51609bfa-e393-49c1-a3c3-f93a6097183e" (UID: "51609bfa-e393-49c1-a3c3-f93a6097183e"). InnerVolumeSpecName "kube-api-access-hkmrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:12:01.440041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:01.210950 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-a-166778-77d4bcfc6f-88bhr" podUID="5c1a1dba-28cb-489a-be20-1ae4bdc0ad85" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.140:50052\" within 1s: context deadline exceeded" Nov 01 16:12:01.440041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:01.091966 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:12:01.440041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:01.092004 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:12:01.440041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:01.244116 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:12:01.444037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:01.430203 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" podUID="a15aa191-e376-490a-8763-16cdd7685b5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:12:01.488214 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:01.430244 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 01 16:12:01.488214 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:01.481997 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:12:01.518038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:01.505342 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:12:01.518038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:01.511489 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-784444-79c887957f-gsjc5" Nov 01 16:12:01.665043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:01.664177 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" podUID="a15aa191-e376-490a-8763-16cdd7685b5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:12:01.693592 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:01.693562 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" Nov 01 16:12:01.733306 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:01.733269 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-166778-77d4bcfc6f-88bhr" Nov 01 16:12:01.769002 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:01.734787 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:12:01.907044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:01.856604 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" Nov 01 16:12:01.964124 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:01.908645 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51609bfa-e393-49c1-a3c3-f93a6097183e-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:02.049301 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.049269 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkmrh\" (UniqueName: \"kubernetes.io/projected/51609bfa-e393-49c1-a3c3-f93a6097183e-kube-api-access-hkmrh\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:02.144462 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:12:02.144405 4363 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51609bfa_e393_49c1_a3c3_f93a6097183e.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51609bfa_e393_49c1_a3c3_f93a6097183e.slice/crio-30e31d6a46d9d8b8d94123005c74858dc1616d849ab3f3477662fb3967ec1d4d\": RecentStats: unable to find data in memory cache]" Nov 01 16:12:02.157950 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.157763 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a77a07cf-0f15-4724-8170-da59b2a1e5f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77a07cf-0f15-4724-8170-da59b2a1e5f7-workload-socket\") pod \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\" (UID: \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\") " Nov 01 16:12:02.157950 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.157918 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a77a07cf-0f15-4724-8170-da59b2a1e5f7-kube-api-access-hxtnd\" (UniqueName: \"kubernetes.io/projected/a77a07cf-0f15-4724-8170-da59b2a1e5f7-kube-api-access-hxtnd\") pod \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\" (UID: \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\") " Nov 01 16:12:02.157950 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.157969 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a77a07cf-0f15-4724-8170-da59b2a1e5f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a77a07cf-0f15-4724-8170-da59b2a1e5f7-istiod-ca-cert\") pod \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\" (UID: \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\") " Nov 01 16:12:02.158348 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.158064 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a77a07cf-0f15-4724-8170-da59b2a1e5f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a77a07cf-0f15-4724-8170-da59b2a1e5f7-workload-certs\") pod \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\" (UID: \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\") " Nov 01 16:12:02.158348 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.158163 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a77a07cf-0f15-4724-8170-da59b2a1e5f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a77a07cf-0f15-4724-8170-da59b2a1e5f7-istio-envoy\") pod \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\" (UID: \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\") " Nov 01 16:12:02.158348 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.158213 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a77a07cf-0f15-4724-8170-da59b2a1e5f7-istio-token\" (UniqueName: \"kubernetes.io/projected/a77a07cf-0f15-4724-8170-da59b2a1e5f7-istio-token\") pod \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\" (UID: \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\") " Nov 01 16:12:02.158348 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.158328 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a77a07cf-0f15-4724-8170-da59b2a1e5f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77a07cf-0f15-4724-8170-da59b2a1e5f7-credential-socket\") pod \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\" (UID: \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\") " Nov 01 16:12:02.158565 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.158359 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c1a1dba-28cb-489a-be20-1ae4bdc0ad85-kube-api-access-5hbq2\" (UniqueName: \"kubernetes.io/projected/5c1a1dba-28cb-489a-be20-1ae4bdc0ad85-kube-api-access-5hbq2\") pod \"5c1a1dba-28cb-489a-be20-1ae4bdc0ad85\" (UID: \"5c1a1dba-28cb-489a-be20-1ae4bdc0ad85\") " Nov 01 16:12:02.158620 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.158590 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a77a07cf-0f15-4724-8170-da59b2a1e5f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a77a07cf-0f15-4724-8170-da59b2a1e5f7-istio-podinfo\") pod \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\" (UID: \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\") " Nov 01 16:12:02.158682 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.158618 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a77a07cf-0f15-4724-8170-da59b2a1e5f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a77a07cf-0f15-4724-8170-da59b2a1e5f7-istio-data\") pod \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\" (UID: \"a77a07cf-0f15-4724-8170-da59b2a1e5f7\") " Nov 01 16:12:02.158682 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.158653 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c1a1dba-28cb-489a-be20-1ae4bdc0ad85-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c1a1dba-28cb-489a-be20-1ae4bdc0ad85-plugins-config-volume\") pod \"5c1a1dba-28cb-489a-be20-1ae4bdc0ad85\" (UID: \"5c1a1dba-28cb-489a-be20-1ae4bdc0ad85\") " Nov 01 16:12:02.162505 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.160153 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c1a1dba-28cb-489a-be20-1ae4bdc0ad85-plugins-config-volume" pod "5c1a1dba-28cb-489a-be20-1ae4bdc0ad85" (UID: "5c1a1dba-28cb-489a-be20-1ae4bdc0ad85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:12:02.164817 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.164782 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a77a07cf-0f15-4724-8170-da59b2a1e5f7-istio-envoy" pod "a77a07cf-0f15-4724-8170-da59b2a1e5f7" (UID: "a77a07cf-0f15-4724-8170-da59b2a1e5f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:12:02.165181 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.165152 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a77a07cf-0f15-4724-8170-da59b2a1e5f7-istiod-ca-cert" pod "a77a07cf-0f15-4724-8170-da59b2a1e5f7" (UID: "a77a07cf-0f15-4724-8170-da59b2a1e5f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:12:02.165526 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.165502 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a77a07cf-0f15-4724-8170-da59b2a1e5f7-workload-certs" pod "a77a07cf-0f15-4724-8170-da59b2a1e5f7" (UID: "a77a07cf-0f15-4724-8170-da59b2a1e5f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:12:02.165648 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.165622 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a77a07cf-0f15-4724-8170-da59b2a1e5f7-istio-podinfo" pod "a77a07cf-0f15-4724-8170-da59b2a1e5f7" (UID: "a77a07cf-0f15-4724-8170-da59b2a1e5f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:12:02.165851 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.165833 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a77a07cf-0f15-4724-8170-da59b2a1e5f7-istio-data" pod "a77a07cf-0f15-4724-8170-da59b2a1e5f7" (UID: "a77a07cf-0f15-4724-8170-da59b2a1e5f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:12:02.166125 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.166089 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a77a07cf-0f15-4724-8170-da59b2a1e5f7-credential-socket" pod "a77a07cf-0f15-4724-8170-da59b2a1e5f7" (UID: "a77a07cf-0f15-4724-8170-da59b2a1e5f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:12:02.191102 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.166265 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a77a07cf-0f15-4724-8170-da59b2a1e5f7-workload-socket" pod "a77a07cf-0f15-4724-8170-da59b2a1e5f7" (UID: "a77a07cf-0f15-4724-8170-da59b2a1e5f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:12:02.191102 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.168847 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c1a1dba-28cb-489a-be20-1ae4bdc0ad85-kube-api-access-5hbq2" pod "5c1a1dba-28cb-489a-be20-1ae4bdc0ad85" (UID: "5c1a1dba-28cb-489a-be20-1ae4bdc0ad85"). InnerVolumeSpecName "kube-api-access-5hbq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:12:02.191102 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.175302 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a77a07cf-0f15-4724-8170-da59b2a1e5f7-istio-token" pod "a77a07cf-0f15-4724-8170-da59b2a1e5f7" (UID: "a77a07cf-0f15-4724-8170-da59b2a1e5f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:12:02.191102 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.175369 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a77a07cf-0f15-4724-8170-da59b2a1e5f7-kube-api-access-hxtnd" pod "a77a07cf-0f15-4724-8170-da59b2a1e5f7" (UID: "a77a07cf-0f15-4724-8170-da59b2a1e5f7"). InnerVolumeSpecName "kube-api-access-hxtnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:12:02.191102 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.190894 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-166778-7d8fcb5659-jq7tt" Nov 01 16:12:02.197922 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:01.770142 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-166778-77d4bcfc6f-88bhr" event={"ID":"5c1a1dba-28cb-489a-be20-1ae4bdc0ad85","Type":"ContainerDied","Data":"ba52d713bcc70feacc9ab9f7e9259c95a312ac91517a902770380a607964b938"} Nov 01 16:12:02.198474 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.198006 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:12:02.198607 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.198498 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-784444-55c46475-d78lt" event={"ID":"59ebabe0-34d2-4102-86de-52bb4d6d04e2","Type":"ContainerStarted","Data":"26a8fe815d9f28ea406fa7d5f3a8ed9782d2d58c0892d282833d7859ad67545b"} Nov 01 16:12:02.199044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.198141 4363 scope.go:122] "RemoveContainer" containerID="88eb0febc5b8f8df7440402b2c47891b9da369f21a363a9f3a6441710b497d28" Nov 01 16:12:02.222596 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.217698 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph" event={"ID":"a77a07cf-0f15-4724-8170-da59b2a1e5f7","Type":"ContainerDied","Data":"6ac0e2455fb12c304350247a81450487db355c9996c21de1f88cd99bf45529d4"} Nov 01 16:12:02.222596 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.217769 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:12:02.222596 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.217801 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-166778-7d8fcb5659-jq7tt" event={"ID":"51609bfa-e393-49c1-a3c3-f93a6097183e","Type":"ContainerDied","Data":"30e31d6a46d9d8b8d94123005c74858dc1616d849ab3f3477662fb3967ec1d4d"} Nov 01 16:12:02.222596 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.217886 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" Nov 01 16:12:02.222596 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.217943 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 16:12:02.222596 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.217976 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-784444-55c46475-d78lt"] Nov 01 16:12:02.222596 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.219801 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:12:02.268795 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.268760 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a77a07cf-0f15-4724-8170-da59b2a1e5f7-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:02.274479 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.274446 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a77a07cf-0f15-4724-8170-da59b2a1e5f7-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:02.274692 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.274674 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c1a1dba-28cb-489a-be20-1ae4bdc0ad85-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:02.274796 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.274784 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77a07cf-0f15-4724-8170-da59b2a1e5f7-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:02.274890 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.274878 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxtnd\" (UniqueName: \"kubernetes.io/projected/a77a07cf-0f15-4724-8170-da59b2a1e5f7-kube-api-access-hxtnd\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:02.274977 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.274966 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a77a07cf-0f15-4724-8170-da59b2a1e5f7-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:02.275094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.275081 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a77a07cf-0f15-4724-8170-da59b2a1e5f7-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:02.275202 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.275190 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a77a07cf-0f15-4724-8170-da59b2a1e5f7-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:02.275293 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.275282 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a77a07cf-0f15-4724-8170-da59b2a1e5f7-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:02.275384 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.275369 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77a07cf-0f15-4724-8170-da59b2a1e5f7-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:02.275473 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.275461 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hbq2\" (UniqueName: \"kubernetes.io/projected/5c1a1dba-28cb-489a-be20-1ae4bdc0ad85-kube-api-access-5hbq2\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:02.293391 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.293363 4363 scope.go:122] "RemoveContainer" containerID="87c70fd1295e0cc86bea738904f47c071d47b9f01313a765303d9b3e691c0613" Nov 01 16:12:02.314953 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.314919 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:12:02.358248 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.358224 4363 scope.go:122] "RemoveContainer" containerID="0285c96e956edbf8d64bb79c684597d677be6dfff08e15028b990bad54fcdbd6" Nov 01 16:12:02.359900 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:12:02.359738 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0285c96e956edbf8d64bb79c684597d677be6dfff08e15028b990bad54fcdbd6\": container with ID starting with 0285c96e956edbf8d64bb79c684597d677be6dfff08e15028b990bad54fcdbd6 not found: ID does not exist" containerID="0285c96e956edbf8d64bb79c684597d677be6dfff08e15028b990bad54fcdbd6" Nov 01 16:12:02.359900 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.359782 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0285c96e956edbf8d64bb79c684597d677be6dfff08e15028b990bad54fcdbd6"} err="failed to get container status \"0285c96e956edbf8d64bb79c684597d677be6dfff08e15028b990bad54fcdbd6\": rpc error: code = NotFound desc = could not find container \"0285c96e956edbf8d64bb79c684597d677be6dfff08e15028b990bad54fcdbd6\": container with ID starting with 0285c96e956edbf8d64bb79c684597d677be6dfff08e15028b990bad54fcdbd6 not found: ID does not exist" Nov 01 16:12:02.359900 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.359807 4363 scope.go:122] "RemoveContainer" containerID="f50d05c3db4ac9a934b0e526891577d4f671756187de2b0fab7f4c23acca7f1a" Nov 01 16:12:02.426474 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.426353 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-166778-7d8fcb5659-jq7tt"] Nov 01 16:12:02.442305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.442278 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-166778-7d8fcb5659-jq7tt"] Nov 01 16:12:02.542764 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.542699 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph"] Nov 01 16:12:02.551480 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.551404 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-784444-79c887957f-gsjc5"] Nov 01 16:12:02.559107 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.559082 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-80c4c8-openshift-default-78ff9c847-vj9ph"] Nov 01 16:12:02.811092 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.811060 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51609bfa-e393-49c1-a3c3-f93a6097183e" path="/var/lib/kubelet/pods/51609bfa-e393-49c1-a3c3-f93a6097183e/volumes" Nov 01 16:12:02.812114 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.812092 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a77a07cf-0f15-4724-8170-da59b2a1e5f7" path="/var/lib/kubelet/pods/a77a07cf-0f15-4724-8170-da59b2a1e5f7/volumes" Nov 01 16:12:02.897385 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.894692 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj"] Nov 01 16:12:02.903347 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.903310 4363 cpu_manager.go:414] "containerMap: removing container" podUID="51609bfa-e393-49c1-a3c3-f93a6097183e" containerName="payload-pre-processing" Nov 01 16:12:02.903347 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.903350 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51609bfa-e393-49c1-a3c3-f93a6097183e" containerName="payload-pre-processing" Nov 01 16:12:02.904323 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.903397 4363 cpu_manager.go:414] "containerMap: removing container" podUID="a77a07cf-0f15-4724-8170-da59b2a1e5f7" containerName="istio-proxy" Nov 01 16:12:02.904323 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.903405 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a77a07cf-0f15-4724-8170-da59b2a1e5f7" containerName="istio-proxy" Nov 01 16:12:02.904323 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.903469 4363 cpu_manager.go:414] "containerMap: removing container" podUID="5c1a1dba-28cb-489a-be20-1ae4bdc0ad85" containerName="payload-processing" Nov 01 16:12:02.904323 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.903476 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c1a1dba-28cb-489a-be20-1ae4bdc0ad85" containerName="payload-processing" Nov 01 16:12:02.904323 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.903506 4363 cpu_manager.go:414] "containerMap: removing container" podUID="a77a07cf-0f15-4724-8170-da59b2a1e5f7" containerName="istio-proxy" Nov 01 16:12:02.904323 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.903514 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a77a07cf-0f15-4724-8170-da59b2a1e5f7" containerName="istio-proxy" Nov 01 16:12:02.904323 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.903779 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c1a1dba-28cb-489a-be20-1ae4bdc0ad85" containerName="payload-processing" Nov 01 16:12:02.904323 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.903805 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="a77a07cf-0f15-4724-8170-da59b2a1e5f7" containerName="istio-proxy" Nov 01 16:12:02.904323 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.903823 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="51609bfa-e393-49c1-a3c3-f93a6097183e" containerName="payload-pre-processing" Nov 01 16:12:02.904323 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.903840 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="a77a07cf-0f15-4724-8170-da59b2a1e5f7" containerName="istio-proxy" Nov 01 16:12:02.912833 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.910467 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" Nov 01 16:12:02.918775 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.916624 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-04f248-openshift-default-dockercfg-wtqqd\"" Nov 01 16:12:02.927422 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.927391 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/388de32a-ab53-48d9-983d-ce302916e7a8-workload-socket\") pod \"e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj\" (UID: \"388de32a-ab53-48d9-983d-ce302916e7a8\") " pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" Nov 01 16:12:02.927628 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.927608 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/388de32a-ab53-48d9-983d-ce302916e7a8-istio-envoy\") pod \"e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj\" (UID: \"388de32a-ab53-48d9-983d-ce302916e7a8\") " pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" Nov 01 16:12:02.927751 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.927736 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/388de32a-ab53-48d9-983d-ce302916e7a8-workload-certs\") pod \"e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj\" (UID: \"388de32a-ab53-48d9-983d-ce302916e7a8\") " pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" Nov 01 16:12:02.927876 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.927861 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/388de32a-ab53-48d9-983d-ce302916e7a8-istio-podinfo\") pod \"e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj\" (UID: \"388de32a-ab53-48d9-983d-ce302916e7a8\") " pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" Nov 01 16:12:02.928076 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.928057 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/388de32a-ab53-48d9-983d-ce302916e7a8-credential-socket\") pod \"e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj\" (UID: \"388de32a-ab53-48d9-983d-ce302916e7a8\") " pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" Nov 01 16:12:02.928327 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.928308 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/388de32a-ab53-48d9-983d-ce302916e7a8-istiod-ca-cert\") pod \"e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj\" (UID: \"388de32a-ab53-48d9-983d-ce302916e7a8\") " pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" Nov 01 16:12:02.928450 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.928434 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/388de32a-ab53-48d9-983d-ce302916e7a8-istio-data\") pod \"e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj\" (UID: \"388de32a-ab53-48d9-983d-ce302916e7a8\") " pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" Nov 01 16:12:02.928710 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.928691 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/388de32a-ab53-48d9-983d-ce302916e7a8-istio-token\") pod \"e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj\" (UID: \"388de32a-ab53-48d9-983d-ce302916e7a8\") " pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" Nov 01 16:12:02.929031 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.928997 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7kr4\" (UniqueName: \"kubernetes.io/projected/388de32a-ab53-48d9-983d-ce302916e7a8-kube-api-access-t7kr4\") pod \"e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj\" (UID: \"388de32a-ab53-48d9-983d-ce302916e7a8\") " pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" Nov 01 16:12:02.929299 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:02.929281 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj"] Nov 01 16:12:03.031180 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:03.031095 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7kr4\" (UniqueName: \"kubernetes.io/projected/388de32a-ab53-48d9-983d-ce302916e7a8-kube-api-access-t7kr4\") pod \"e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj\" (UID: \"388de32a-ab53-48d9-983d-ce302916e7a8\") " pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" Nov 01 16:12:03.031340 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:03.031198 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/388de32a-ab53-48d9-983d-ce302916e7a8-workload-socket\") pod \"e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj\" (UID: \"388de32a-ab53-48d9-983d-ce302916e7a8\") " pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" Nov 01 16:12:03.031340 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:03.031247 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/388de32a-ab53-48d9-983d-ce302916e7a8-istio-envoy\") pod \"e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj\" (UID: \"388de32a-ab53-48d9-983d-ce302916e7a8\") " pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" Nov 01 16:12:03.031340 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:03.031280 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/388de32a-ab53-48d9-983d-ce302916e7a8-workload-certs\") pod \"e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj\" (UID: \"388de32a-ab53-48d9-983d-ce302916e7a8\") " pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" Nov 01 16:12:03.031340 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:03.031313 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/388de32a-ab53-48d9-983d-ce302916e7a8-istio-podinfo\") pod \"e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj\" (UID: \"388de32a-ab53-48d9-983d-ce302916e7a8\") " pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" Nov 01 16:12:03.031544 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:03.031388 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/388de32a-ab53-48d9-983d-ce302916e7a8-credential-socket\") pod \"e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj\" (UID: \"388de32a-ab53-48d9-983d-ce302916e7a8\") " pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" Nov 01 16:12:03.031544 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:03.031528 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/388de32a-ab53-48d9-983d-ce302916e7a8-istiod-ca-cert\") pod \"e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj\" (UID: \"388de32a-ab53-48d9-983d-ce302916e7a8\") " pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" Nov 01 16:12:03.031616 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:03.031562 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/388de32a-ab53-48d9-983d-ce302916e7a8-istio-data\") pod \"e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj\" (UID: \"388de32a-ab53-48d9-983d-ce302916e7a8\") " pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" Nov 01 16:12:03.031616 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:03.031594 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/388de32a-ab53-48d9-983d-ce302916e7a8-istio-token\") pod \"e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj\" (UID: \"388de32a-ab53-48d9-983d-ce302916e7a8\") " pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" Nov 01 16:12:03.032439 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:03.032115 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/388de32a-ab53-48d9-983d-ce302916e7a8-credential-socket\") pod \"e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj\" (UID: \"388de32a-ab53-48d9-983d-ce302916e7a8\") " pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" Nov 01 16:12:03.032439 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:03.032166 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/388de32a-ab53-48d9-983d-ce302916e7a8-workload-certs\") pod \"e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj\" (UID: \"388de32a-ab53-48d9-983d-ce302916e7a8\") " pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" Nov 01 16:12:03.032439 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:03.032380 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/388de32a-ab53-48d9-983d-ce302916e7a8-istio-data\") pod \"e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj\" (UID: \"388de32a-ab53-48d9-983d-ce302916e7a8\") " pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" Nov 01 16:12:03.032439 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:03.032440 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/388de32a-ab53-48d9-983d-ce302916e7a8-workload-socket\") pod \"e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj\" (UID: \"388de32a-ab53-48d9-983d-ce302916e7a8\") " pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" Nov 01 16:12:03.032713 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:03.032682 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/388de32a-ab53-48d9-983d-ce302916e7a8-istiod-ca-cert\") pod \"e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj\" (UID: \"388de32a-ab53-48d9-983d-ce302916e7a8\") " pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" Nov 01 16:12:03.033982 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:03.033960 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/388de32a-ab53-48d9-983d-ce302916e7a8-istio-envoy\") pod \"e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj\" (UID: \"388de32a-ab53-48d9-983d-ce302916e7a8\") " pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" Nov 01 16:12:03.034704 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:03.034686 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/388de32a-ab53-48d9-983d-ce302916e7a8-istio-podinfo\") pod \"e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj\" (UID: \"388de32a-ab53-48d9-983d-ce302916e7a8\") " pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" Nov 01 16:12:03.040345 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:03.040151 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7kr4\" (UniqueName: \"kubernetes.io/projected/388de32a-ab53-48d9-983d-ce302916e7a8-kube-api-access-t7kr4\") pod \"e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj\" (UID: \"388de32a-ab53-48d9-983d-ce302916e7a8\") " pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" Nov 01 16:12:03.040345 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:03.040273 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/388de32a-ab53-48d9-983d-ce302916e7a8-istio-token\") pod \"e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj\" (UID: \"388de32a-ab53-48d9-983d-ce302916e7a8\") " pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" Nov 01 16:12:03.242043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:03.236747 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-166778-77d4bcfc6f-88bhr" Nov 01 16:12:03.242821 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:03.242574 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" Nov 01 16:12:03.280244 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:03.278666 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-784444-55c46475-d78lt" event={"ID":"59ebabe0-34d2-4102-86de-52bb4d6d04e2","Type":"ContainerStarted","Data":"0ade98c8ab5aecb81daf070285e586f52de6816ce0969851c7eed7e06c8cff1e"} Nov 01 16:12:03.280244 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:03.280007 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-784444-55c46475-d78lt" Nov 01 16:12:03.326892 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:03.326797 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-784444-79c887957f-gsjc5" event={"ID":"f796692a-c6ab-48b9-a227-00e8af4f24df","Type":"ContainerStarted","Data":"10dca15fcaf3a9014dfc285d4a55df18db1ca6cfe936347bfb3c4698f038915c"} Nov 01 16:12:03.327357 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:03.327240 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-784444-79c887957f-gsjc5" event={"ID":"f796692a-c6ab-48b9-a227-00e8af4f24df","Type":"ContainerStarted","Data":"29fcdac271afd3010ef9d7b196fef903a3ba58fff531ebd753b8bd92b0bd5245"} Nov 01 16:12:03.341182 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:03.338694 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-784444-79c887957f-gsjc5" Nov 01 16:12:03.378341 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:03.371669 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-784444-55c46475-d78lt" podStartSLOduration=7.371650561 podStartE2EDuration="7.371650561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:11:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:12:03.30958575 +0000 UTC m=+1456.641323401" watchObservedRunningTime="2026-11-01 16:12:03.371650561 +0000 UTC m=+1456.703388213" Nov 01 16:12:03.400073 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:03.398285 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-784444-79c887957f-gsjc5" podStartSLOduration=7.398264375 podStartE2EDuration="7.398264375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:11:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:12:03.338056239 +0000 UTC m=+1456.669793887" watchObservedRunningTime="2026-11-01 16:12:03.398264375 +0000 UTC m=+1456.730002026" Nov 01 16:12:03.436106 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:03.436068 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-166778-77d4bcfc6f-88bhr"] Nov 01 16:12:03.506839 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:03.506806 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-166778-77d4bcfc6f-88bhr"] Nov 01 16:12:04.247693 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:04.247660 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj"] Nov 01 16:12:04.248298 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:12:04.248271 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod388de32a_ab53_48d9_983d_ce302916e7a8.slice/crio-8488b5b2ba1e4ed9c4f6a693ae0909259643c10abaab03025d1798f9f4d6b976 WatchSource:0}: Error finding container 8488b5b2ba1e4ed9c4f6a693ae0909259643c10abaab03025d1798f9f4d6b976: Status 404 returned error can't find the container with id 8488b5b2ba1e4ed9c4f6a693ae0909259643c10abaab03025d1798f9f4d6b976 Nov 01 16:12:04.252547 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:04.252496 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:12:04.252666 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:04.252598 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:12:04.252666 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:04.252647 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:12:04.309816 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:04.309770 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" event={"ID":"388de32a-ab53-48d9-983d-ce302916e7a8","Type":"ContainerStarted","Data":"8488b5b2ba1e4ed9c4f6a693ae0909259643c10abaab03025d1798f9f4d6b976"} Nov 01 16:12:04.806503 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:04.806459 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c1a1dba-28cb-489a-be20-1ae4bdc0ad85" path="/var/lib/kubelet/pods/5c1a1dba-28cb-489a-be20-1ae4bdc0ad85/volumes" Nov 01 16:12:05.321616 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:05.321287 4363 patch_prober.go:29] interesting pod/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:12:05.321616 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:05.321354 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" podUID="905f9285-a9c5-4a2a-abbf-8114e5ac9f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:12:05.331660 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:05.331624 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" event={"ID":"388de32a-ab53-48d9-983d-ce302916e7a8","Type":"ContainerStarted","Data":"4781f143604d4c1c79729186e3dd8913f8dd19537443e0fa4c9bba5a1c4a0081"} Nov 01 16:12:05.359353 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:05.359290 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" podStartSLOduration=3.359272558 podStartE2EDuration="3.359272558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:12:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:12:05.351251146 +0000 UTC m=+1458.682988798" watchObservedRunningTime="2026-11-01 16:12:05.359272558 +0000 UTC m=+1458.691010208" Nov 01 16:12:06.244068 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:06.244012 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" Nov 01 16:12:06.249915 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:06.249886 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" Nov 01 16:12:06.374689 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:06.367411 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" Nov 01 16:12:06.374689 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:06.373588 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" Nov 01 16:12:08.937279 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:08.937200 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-04f248-6f57f7dc7-snmtt"] Nov 01 16:12:10.037557 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:08.971248 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-04f248-65b9669fcf-7fzhs"] Nov 01 16:12:10.037557 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:08.971370 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-04f248-6f57f7dc7-snmtt" Nov 01 16:12:10.037557 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:08.975966 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-04f248\"" Nov 01 16:12:10.037557 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:08.990849 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-04f248-6f57f7dc7-snmtt"] Nov 01 16:12:10.037557 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:08.990896 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-04f248-65b9669fcf-7fzhs"] Nov 01 16:12:10.037557 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:08.991073 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04f248-65b9669fcf-7fzhs" Nov 01 16:12:10.037557 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:08.995181 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-04f248-dockercfg-5gf5n\"" Nov 01 16:12:10.037557 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:09.049865 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvr2t\" (UniqueName: \"kubernetes.io/projected/6c62027d-8353-4200-82f7-365988964d51-kube-api-access-mvr2t\") pod \"payload-processing-e2e-inf-a-04f248-6f57f7dc7-snmtt\" (UID: \"6c62027d-8353-4200-82f7-365988964d51\") " pod="openshift-ingress/payload-processing-e2e-inf-a-04f248-6f57f7dc7-snmtt" Nov 01 16:12:10.037557 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:09.049940 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e923b2d1-5a22-4eca-973d-6a94650574b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-04f248-65b9669fcf-7fzhs\" (UID: \"e923b2d1-5a22-4eca-973d-6a94650574b4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04f248-65b9669fcf-7fzhs" Nov 01 16:12:10.037557 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:09.050228 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vdng\" (UniqueName: \"kubernetes.io/projected/e923b2d1-5a22-4eca-973d-6a94650574b4-kube-api-access-8vdng\") pod \"payload-pre-processing-e2e-inf-a-04f248-65b9669fcf-7fzhs\" (UID: \"e923b2d1-5a22-4eca-973d-6a94650574b4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04f248-65b9669fcf-7fzhs" Nov 01 16:12:10.037557 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:09.050426 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c62027d-8353-4200-82f7-365988964d51-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-04f248-6f57f7dc7-snmtt\" (UID: \"6c62027d-8353-4200-82f7-365988964d51\") " pod="openshift-ingress/payload-processing-e2e-inf-a-04f248-6f57f7dc7-snmtt" Nov 01 16:12:10.037557 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:09.151608 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c62027d-8353-4200-82f7-365988964d51-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-04f248-6f57f7dc7-snmtt\" (UID: \"6c62027d-8353-4200-82f7-365988964d51\") " pod="openshift-ingress/payload-processing-e2e-inf-a-04f248-6f57f7dc7-snmtt" Nov 01 16:12:10.037557 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:09.151684 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvr2t\" (UniqueName: \"kubernetes.io/projected/6c62027d-8353-4200-82f7-365988964d51-kube-api-access-mvr2t\") pod \"payload-processing-e2e-inf-a-04f248-6f57f7dc7-snmtt\" (UID: \"6c62027d-8353-4200-82f7-365988964d51\") " pod="openshift-ingress/payload-processing-e2e-inf-a-04f248-6f57f7dc7-snmtt" Nov 01 16:12:10.037557 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:09.151727 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e923b2d1-5a22-4eca-973d-6a94650574b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-04f248-65b9669fcf-7fzhs\" (UID: \"e923b2d1-5a22-4eca-973d-6a94650574b4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04f248-65b9669fcf-7fzhs" Nov 01 16:12:10.037557 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:09.151936 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vdng\" (UniqueName: \"kubernetes.io/projected/e923b2d1-5a22-4eca-973d-6a94650574b4-kube-api-access-8vdng\") pod \"payload-pre-processing-e2e-inf-a-04f248-65b9669fcf-7fzhs\" (UID: \"e923b2d1-5a22-4eca-973d-6a94650574b4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04f248-65b9669fcf-7fzhs" Nov 01 16:12:10.037557 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:09.154514 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e923b2d1-5a22-4eca-973d-6a94650574b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-04f248-65b9669fcf-7fzhs\" (UID: \"e923b2d1-5a22-4eca-973d-6a94650574b4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04f248-65b9669fcf-7fzhs" Nov 01 16:12:10.037557 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:09.174420 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvr2t\" (UniqueName: \"kubernetes.io/projected/6c62027d-8353-4200-82f7-365988964d51-kube-api-access-mvr2t\") pod \"payload-processing-e2e-inf-a-04f248-6f57f7dc7-snmtt\" (UID: \"6c62027d-8353-4200-82f7-365988964d51\") " pod="openshift-ingress/payload-processing-e2e-inf-a-04f248-6f57f7dc7-snmtt" Nov 01 16:12:10.037557 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:09.179746 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c62027d-8353-4200-82f7-365988964d51-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-04f248-6f57f7dc7-snmtt\" (UID: \"6c62027d-8353-4200-82f7-365988964d51\") " pod="openshift-ingress/payload-processing-e2e-inf-a-04f248-6f57f7dc7-snmtt" Nov 01 16:12:10.037557 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:09.205818 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vdng\" (UniqueName: \"kubernetes.io/projected/e923b2d1-5a22-4eca-973d-6a94650574b4-kube-api-access-8vdng\") pod \"payload-pre-processing-e2e-inf-a-04f248-65b9669fcf-7fzhs\" (UID: \"e923b2d1-5a22-4eca-973d-6a94650574b4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04f248-65b9669fcf-7fzhs" Nov 01 16:12:10.037557 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:09.332713 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-04f248-6f57f7dc7-snmtt" Nov 01 16:12:10.037557 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:09.375182 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04f248-65b9669fcf-7fzhs" Nov 01 16:12:10.037557 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:09.602470 4363 patch_prober.go:29] interesting pod/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": read tcp 10.134.0.2:47700->10.134.0.144:15021: read: connection reset by peer" start-of-body= Nov 01 16:12:10.037557 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:09.602569 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" podUID="ba075b49-a3d6-4813-bbcf-37f11e5550a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": read tcp 10.134.0.2:47700->10.134.0.144:15021: read: connection reset by peer" Nov 01 16:12:10.037557 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:09.946117 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-04f248-65b9669fcf-7fzhs"] Nov 01 16:12:10.037557 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:09.957944 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-04f248-6f57f7dc7-snmtt"] Nov 01 16:12:10.037557 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:12:09.959504 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c62027d_8353_4200_82f7_365988964d51.slice/crio-06621aa34d1f53ad83e85c7f56c1338c21986c5d4eec9f4eee2a846458bd9f91 WatchSource:0}: Error finding container 06621aa34d1f53ad83e85c7f56c1338c21986c5d4eec9f4eee2a846458bd9f91: Status 404 returned error can't find the container with id 06621aa34d1f53ad83e85c7f56c1338c21986c5d4eec9f4eee2a846458bd9f91 Nov 01 16:12:10.463399 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:10.463361 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-04f248-6f57f7dc7-snmtt" event={"ID":"6c62027d-8353-4200-82f7-365988964d51","Type":"ContainerStarted","Data":"8b28ce784160139d75f45aa92a1e35273a7aa3706ffecb8bd813a31e2e7a52a9"} Nov 01 16:12:10.463621 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:10.463446 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-04f248-6f57f7dc7-snmtt" event={"ID":"6c62027d-8353-4200-82f7-365988964d51","Type":"ContainerStarted","Data":"06621aa34d1f53ad83e85c7f56c1338c21986c5d4eec9f4eee2a846458bd9f91"} Nov 01 16:12:10.463621 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:10.463473 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-04f248-6f57f7dc7-snmtt" Nov 01 16:12:10.465235 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:10.465209 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04f248-65b9669fcf-7fzhs" event={"ID":"e923b2d1-5a22-4eca-973d-6a94650574b4","Type":"ContainerStarted","Data":"f6d385b745181d874ca57a877ccc4eeccf0d85a85c48699a2b64864c3a0c2fe0"} Nov 01 16:12:10.465385 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:10.465251 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04f248-65b9669fcf-7fzhs" event={"ID":"e923b2d1-5a22-4eca-973d-6a94650574b4","Type":"ContainerStarted","Data":"8d1490118ad470482bf7b471a41cf01b6d7db97de602578b4a5b142ba0f6a1f7"} Nov 01 16:12:10.465385 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:10.465295 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04f248-65b9669fcf-7fzhs" Nov 01 16:12:10.466969 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:10.466948 4363 generic.go:356] "Generic (PLEG): container finished" podID="ba075b49-a3d6-4813-bbcf-37f11e5550a1" containerID="57d261958ee14cf1fb89882b6a7e49c3f10510781bfa7b4b0ce85668171dae1f" exitCode=0 Nov 01 16:12:10.467077 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:10.467031 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" event={"ID":"ba075b49-a3d6-4813-bbcf-37f11e5550a1","Type":"ContainerDied","Data":"57d261958ee14cf1fb89882b6a7e49c3f10510781bfa7b4b0ce85668171dae1f"} Nov 01 16:12:10.467441 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:10.467420 4363 scope.go:122] "RemoveContainer" containerID="57d261958ee14cf1fb89882b6a7e49c3f10510781bfa7b4b0ce85668171dae1f" Nov 01 16:12:10.468847 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:10.468827 4363 generic.go:356] "Generic (PLEG): container finished" podID="145ad297-6cf1-4c70-ba0e-db5047a25f20" containerID="2afc294017df9e90a1e8837a1cb74a7ff430675a829c41007f669d7b80469874" exitCode=0 Nov 01 16:12:10.468926 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:10.468898 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" event={"ID":"145ad297-6cf1-4c70-ba0e-db5047a25f20","Type":"ContainerDied","Data":"2afc294017df9e90a1e8837a1cb74a7ff430675a829c41007f669d7b80469874"} Nov 01 16:12:10.469234 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:10.469216 4363 scope.go:122] "RemoveContainer" containerID="2afc294017df9e90a1e8837a1cb74a7ff430675a829c41007f669d7b80469874" Nov 01 16:12:10.469460 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:10.469429 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:12:10.469559 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:10.469509 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:12:10.469623 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:10.469556 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:12:10.470584 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:10.470565 4363 generic.go:356] "Generic (PLEG): container finished" podID="905f9285-a9c5-4a2a-abbf-8114e5ac9f26" containerID="d4ed8c081c6ccf18dbbda3c56e45e17bc2f2c8535f043a60ffef029233d40292" exitCode=0 Nov 01 16:12:10.470670 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:10.470611 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" event={"ID":"905f9285-a9c5-4a2a-abbf-8114e5ac9f26","Type":"ContainerDied","Data":"d4ed8c081c6ccf18dbbda3c56e45e17bc2f2c8535f043a60ffef029233d40292"} Nov 01 16:12:10.470965 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:10.470946 4363 scope.go:122] "RemoveContainer" containerID="d4ed8c081c6ccf18dbbda3c56e45e17bc2f2c8535f043a60ffef029233d40292" Nov 01 16:12:10.471063 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:10.471002 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:12:10.471115 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:10.471098 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:12:10.471152 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:10.471138 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:12:10.473405 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:10.473383 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:12:10.473458 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:10.473445 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:12:10.473494 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:10.473479 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:12:10.484291 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:10.483594 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-04f248-6f57f7dc7-snmtt" podStartSLOduration=2.483577783 podStartE2EDuration="2.483577783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:12:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:12:10.478417484 +0000 UTC m=+1463.810155134" watchObservedRunningTime="2026-11-01 16:12:10.483577783 +0000 UTC m=+1463.815315436" Nov 01 16:12:10.559554 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:10.557195 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04f248-65b9669fcf-7fzhs" podStartSLOduration=2.557176341 podStartE2EDuration="2.557176341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:12:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:12:10.540759726 +0000 UTC m=+1463.872497371" watchObservedRunningTime="2026-11-01 16:12:10.557176341 +0000 UTC m=+1463.888913991" Nov 01 16:12:10.563227 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:10.563157 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" Nov 01 16:12:11.480989 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:11.480942 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" event={"ID":"ba075b49-a3d6-4813-bbcf-37f11e5550a1","Type":"ContainerStarted","Data":"41dcc89d213d23ad2ed60613e6b1a5f821c9952320dc43122e9f615a5e31ac94"} Nov 01 16:12:11.483730 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:11.483699 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" event={"ID":"145ad297-6cf1-4c70-ba0e-db5047a25f20","Type":"ContainerStarted","Data":"304915312cbd6fcb813ed142f9f16e61ff87306bd617931ffe23646853f2b632"} Nov 01 16:12:11.486201 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:11.486171 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" event={"ID":"905f9285-a9c5-4a2a-abbf-8114e5ac9f26","Type":"ContainerStarted","Data":"0436bad70374c2e8dfd0177fc35ad9fe7e7ff215cb4a8f53fffc136130c1b01c"} Nov 01 16:12:11.621042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:11.610775 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" Nov 01 16:12:11.621042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:11.610891 4363 patch_prober.go:29] interesting pod/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 01 16:12:11.621042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:11.610932 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" podUID="905f9285-a9c5-4a2a-abbf-8114e5ac9f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 01 16:12:11.761725 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:11.760557 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" Nov 01 16:12:11.773050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:11.766332 4363 patch_prober.go:29] interesting pod/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 01 16:12:11.773050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:11.766396 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" podUID="145ad297-6cf1-4c70-ba0e-db5047a25f20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 01 16:12:12.417041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:12.416446 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" Nov 01 16:12:12.417041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:12.416601 4363 patch_prober.go:29] interesting pod/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 01 16:12:12.417041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:12.416677 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" podUID="ba075b49-a3d6-4813-bbcf-37f11e5550a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 01 16:12:12.604063 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:12.600194 4363 patch_prober.go:29] interesting pod/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 01 16:12:12.604063 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:12.600265 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" podUID="905f9285-a9c5-4a2a-abbf-8114e5ac9f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 01 16:12:12.770099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:12.767895 4363 patch_prober.go:29] interesting pod/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 01 16:12:12.770099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:12.767980 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" podUID="145ad297-6cf1-4c70-ba0e-db5047a25f20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 01 16:12:13.414891 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:13.414440 4363 patch_prober.go:29] interesting pod/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 01 16:12:13.414891 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:13.414502 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" podUID="ba075b49-a3d6-4813-bbcf-37f11e5550a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 01 16:12:13.603135 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:13.603104 4363 patch_prober.go:29] interesting pod/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 01 16:12:13.603427 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:13.603391 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" podUID="905f9285-a9c5-4a2a-abbf-8114e5ac9f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 01 16:12:13.787269 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:13.787177 4363 patch_prober.go:29] interesting pod/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 01 16:12:13.787872 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:13.787839 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" podUID="145ad297-6cf1-4c70-ba0e-db5047a25f20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 01 16:12:14.268042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:14.260095 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded" Nov 01 16:12:14.288070 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:14.287117 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:12:14.288380 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:14.288346 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:12:14.392644 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:14.392606 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9"] Nov 01 16:12:14.411039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:14.393120 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9" podUID="375990c6-6bd9-4c31-9a12-74732ca46a47" containerName="maas-api" containerID="cri-o://10797d2c688855ad8b840577990403ba4b74814ae8ce61d85d349771ff2e2301" gracePeriod=30 Nov 01 16:12:14.411426 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:14.409902 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-784444-79c887957f-gsjc5" Nov 01 16:12:14.445918 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:14.445862 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:12:14.448147 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:14.448102 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" podUID="a15aa191-e376-490a-8763-16cdd7685b5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": context deadline exceeded" Nov 01 16:12:14.460745 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:14.456205 4363 patch_prober.go:29] interesting pod/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 01 16:12:14.460745 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:14.456261 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" podUID="ba075b49-a3d6-4813-bbcf-37f11e5550a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 01 16:12:14.654100 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:14.632302 4363 patch_prober.go:29] interesting pod/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 01 16:12:14.654100 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:14.632364 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" podUID="905f9285-a9c5-4a2a-abbf-8114e5ac9f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 01 16:12:14.817812 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:14.817712 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 01 16:12:15.018041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:14.840211 4363 patch_prober.go:29] interesting pod/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 01 16:12:15.018041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:15.011198 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" podUID="145ad297-6cf1-4c70-ba0e-db5047a25f20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 01 16:12:15.018041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:14.993637 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:12:15.070802 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:14.993008 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:12:15.070802 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:15.056443 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": context deadline exceeded" Nov 01 16:12:15.294296 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:15.294248 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:12:15.398692 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:15.315454 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:12:15.412084 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:15.407795 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:12:15.412084 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:15.407972 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 16:12:15.434091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:15.414165 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-784444-55c46475-d78lt" Nov 01 16:12:15.434091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:15.415797 4363 patch_prober.go:29] interesting pod/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 01 16:12:15.434091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:15.415854 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" podUID="ba075b49-a3d6-4813-bbcf-37f11e5550a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 01 16:12:15.454791 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:15.454729 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:12:15.467382 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:15.467336 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" podUID="a15aa191-e376-490a-8763-16cdd7685b5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:12:15.469644 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:15.469616 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" Nov 01 16:12:15.602552 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:15.602499 4363 patch_prober.go:29] interesting pod/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 01 16:12:15.867477 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:15.867435 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" podUID="905f9285-a9c5-4a2a-abbf-8114e5ac9f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 01 16:12:15.868487 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:15.867363 4363 patch_prober.go:29] interesting pod/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 01 16:12:15.868687 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:15.868662 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" podUID="145ad297-6cf1-4c70-ba0e-db5047a25f20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 01 16:12:16.368043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:16.305553 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 01 16:12:16.368043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:16.333115 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:12:16.368043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:16.334298 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 16:12:16.368043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:16.334372 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:12:16.567092 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:16.475109 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:12:16.567092 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:16.490205 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" Nov 01 16:12:16.640045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:16.614143 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:12:16.849148 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:16.810725 4363 patch_prober.go:29] interesting pod/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 01 16:12:16.849148 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:16.810799 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" podUID="145ad297-6cf1-4c70-ba0e-db5047a25f20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 01 16:12:17.055772 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:16.515131 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:12:17.135187 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:17.135085 4363 patch_prober.go:29] interesting pod/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 01 16:12:17.215043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:17.174081 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" podUID="905f9285-a9c5-4a2a-abbf-8114e5ac9f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 01 16:12:17.318989 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:17.310935 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:12:17.331175 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:17.331133 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 01 16:12:17.397113 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:17.388489 4363 generic.go:356] "Generic (PLEG): container finished" podID="375990c6-6bd9-4c31-9a12-74732ca46a47" containerID="10797d2c688855ad8b840577990403ba4b74814ae8ce61d85d349771ff2e2301" exitCode=0 Nov 01 16:12:17.405296 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:17.389230 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" podUID="a15aa191-e376-490a-8763-16cdd7685b5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:12:17.415229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:17.335124 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:12:17.484111 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:17.425329 4363 patch_prober.go:29] interesting pod/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 01 16:12:17.484111 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:17.425378 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" podUID="ba075b49-a3d6-4813-bbcf-37f11e5550a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 01 16:12:17.484111 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:17.480158 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:12:17.498004 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:17.488618 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f47890-84fdfb9c59-fs5nn" podUID="18faa082-6418-4665-b862-09efa76dfcc0" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.114:50052\" within 1s: context deadline exceeded" Nov 01 16:12:17.528306 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:17.527301 4363 patch_prober.go:29] interesting pod/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 01 16:12:17.528306 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:17.527363 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" podUID="ba075b49-a3d6-4813-bbcf-37f11e5550a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 01 16:12:17.605278 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:12:17.599752 4363 logging.go:55] [core] [Channel #3290 SubChannel #3291]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.114:50052", ServerName: "10.134.0.114:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.114:50052: operation was canceled" Nov 01 16:12:17.698517 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:17.698442 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9" Nov 01 16:12:17.715417 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:17.715381 4363 patch_prober.go:29] interesting pod/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 01 16:12:17.715706 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:17.715674 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" podUID="905f9285-a9c5-4a2a-abbf-8114e5ac9f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 01 16:12:17.750051 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:17.734596 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:12:17.801976 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:17.776938 4363 patch_prober.go:29] interesting pod/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 01 16:12:17.801976 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:17.776994 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" podUID="145ad297-6cf1-4c70-ba0e-db5047a25f20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 01 16:12:17.801976 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:17.798498 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9" event={"ID":"375990c6-6bd9-4c31-9a12-74732ca46a47","Type":"ContainerDied","Data":"10797d2c688855ad8b840577990403ba4b74814ae8ce61d85d349771ff2e2301"} Nov 01 16:12:17.801976 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:17.798579 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 16:12:17.801976 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:17.798827 4363 scope.go:122] "RemoveContainer" containerID="10797d2c688855ad8b840577990403ba4b74814ae8ce61d85d349771ff2e2301" Nov 01 16:12:18.103836 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:18.010798 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/375990c6-6bd9-4c31-9a12-74732ca46a47-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/375990c6-6bd9-4c31-9a12-74732ca46a47-maas-api-tls\") pod \"375990c6-6bd9-4c31-9a12-74732ca46a47\" (UID: \"375990c6-6bd9-4c31-9a12-74732ca46a47\") " Nov 01 16:12:18.103836 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:18.010956 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/375990c6-6bd9-4c31-9a12-74732ca46a47-metrics-certs\" (UniqueName: \"kubernetes.io/secret/375990c6-6bd9-4c31-9a12-74732ca46a47-metrics-certs\") pod \"375990c6-6bd9-4c31-9a12-74732ca46a47\" (UID: \"375990c6-6bd9-4c31-9a12-74732ca46a47\") " Nov 01 16:12:18.103836 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:18.011125 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/375990c6-6bd9-4c31-9a12-74732ca46a47-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/375990c6-6bd9-4c31-9a12-74732ca46a47-openshift-service-ca\") pod \"375990c6-6bd9-4c31-9a12-74732ca46a47\" (UID: \"375990c6-6bd9-4c31-9a12-74732ca46a47\") " Nov 01 16:12:18.103836 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:18.026382 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/375990c6-6bd9-4c31-9a12-74732ca46a47-openshift-service-ca" pod "375990c6-6bd9-4c31-9a12-74732ca46a47" (UID: "375990c6-6bd9-4c31-9a12-74732ca46a47"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:12:18.103836 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:18.044355 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/375990c6-6bd9-4c31-9a12-74732ca46a47-kube-api-access-dtf9f\" (UniqueName: \"kubernetes.io/projected/375990c6-6bd9-4c31-9a12-74732ca46a47-kube-api-access-dtf9f\") pod \"375990c6-6bd9-4c31-9a12-74732ca46a47\" (UID: \"375990c6-6bd9-4c31-9a12-74732ca46a47\") " Nov 01 16:12:18.103836 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:18.045601 4363 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/375990c6-6bd9-4c31-9a12-74732ca46a47-openshift-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:18.103836 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:18.059310 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:12:18.103836 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:18.059924 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:12:18.142563 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:18.122141 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:12:18.142563 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:18.123053 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/375990c6-6bd9-4c31-9a12-74732ca46a47-maas-api-tls" pod "375990c6-6bd9-4c31-9a12-74732ca46a47" (UID: "375990c6-6bd9-4c31-9a12-74732ca46a47"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:12:18.165119 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:18.157883 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/375990c6-6bd9-4c31-9a12-74732ca46a47-maas-api-tls\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:18.276621 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:18.231164 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/375990c6-6bd9-4c31-9a12-74732ca46a47-metrics-certs" pod "375990c6-6bd9-4c31-9a12-74732ca46a47" (UID: "375990c6-6bd9-4c31-9a12-74732ca46a47"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:12:18.276621 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:18.266574 4363 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/375990c6-6bd9-4c31-9a12-74732ca46a47-metrics-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:18.329117 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:18.328183 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:12:18.369124 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:18.340223 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/375990c6-6bd9-4c31-9a12-74732ca46a47-kube-api-access-dtf9f" pod "375990c6-6bd9-4c31-9a12-74732ca46a47" (UID: "375990c6-6bd9-4c31-9a12-74732ca46a47"). InnerVolumeSpecName "kube-api-access-dtf9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:12:18.369124 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:18.352900 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 01 16:12:18.389986 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:18.389940 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtf9f\" (UniqueName: \"kubernetes.io/projected/375990c6-6bd9-4c31-9a12-74732ca46a47-kube-api-access-dtf9f\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:18.447112 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:18.415111 4363 patch_prober.go:29] interesting pod/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 01 16:12:18.447112 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:18.415170 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" podUID="ba075b49-a3d6-4813-bbcf-37f11e5550a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 01 16:12:18.460860 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:18.460037 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9" Nov 01 16:12:18.461137 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:18.461108 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9" event={"ID":"375990c6-6bd9-4c31-9a12-74732ca46a47","Type":"ContainerDied","Data":"35f7b1db7e9acc8dbd5bcc052c880a070010cfba068b37be4c87399a019fcefe"} Nov 01 16:12:18.464776 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:18.464744 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" podUID="a15aa191-e376-490a-8763-16cdd7685b5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:12:18.466515 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:18.466494 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" Nov 01 16:12:18.660099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:18.650851 4363 patch_prober.go:29] interesting pod/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 01 16:12:18.660099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:18.650913 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" podUID="905f9285-a9c5-4a2a-abbf-8114e5ac9f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 01 16:12:18.823292 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:18.823048 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9"] Nov 01 16:12:18.847533 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:18.829854 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:12:18.847533 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:18.841550 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:12:18.848587 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:18.848056 4363 patch_prober.go:29] interesting pod/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 01 16:12:18.848587 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:18.848108 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" podUID="145ad297-6cf1-4c70-ba0e-db5047a25f20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 01 16:12:19.072575 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:12:19.056901 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:12:19.075523 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:19.075148 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:12:19.105435 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:19.105389 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:12:19.281069 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:19.280230 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" Nov 01 16:12:19.322051 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:19.320351 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9"] Nov 01 16:12:19.341854 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:19.339656 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:12:19.590390 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:19.590346 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" podUID="a15aa191-e376-490a-8763-16cdd7685b5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": context deadline exceeded" Nov 01 16:12:19.629794 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:19.629734 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-022f62-54b6c8bb99-wl7h9" podUID="375990c6-6bd9-4c31-9a12-74732ca46a47" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.142:8443/health\": context deadline exceeded" Nov 01 16:12:19.647189 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:19.647122 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:12:19.911089 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:19.894176 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:12:20.011681 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:20.011643 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2001b0-678dffcb5f-c94ht"] Nov 01 16:12:20.018002 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:20.017961 4363 patch_prober.go:29] interesting pod/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 01 16:12:20.018328 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:20.018286 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" podUID="905f9285-a9c5-4a2a-abbf-8114e5ac9f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 01 16:12:20.024468 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:20.024442 4363 patch_prober.go:29] interesting pod/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 01 16:12:20.024701 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:20.024677 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" podUID="145ad297-6cf1-4c70-ba0e-db5047a25f20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 01 16:12:20.034901 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:20.034852 4363 patch_prober.go:29] interesting pod/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 01 16:12:20.035160 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:20.035132 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" podUID="ba075b49-a3d6-4813-bbcf-37f11e5550a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 01 16:12:20.035301 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:20.035277 4363 cpu_manager.go:414] "containerMap: removing container" podUID="375990c6-6bd9-4c31-9a12-74732ca46a47" containerName="maas-api" Nov 01 16:12:20.035483 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:20.035305 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="375990c6-6bd9-4c31-9a12-74732ca46a47" containerName="maas-api" Nov 01 16:12:20.038249 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:20.036695 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="375990c6-6bd9-4c31-9a12-74732ca46a47" containerName="maas-api" Nov 01 16:12:20.070854 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:20.070820 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2001b0-678dffcb5f-c94ht"] Nov 01 16:12:20.071215 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:20.071194 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2001b0-678dffcb5f-c94ht" Nov 01 16:12:20.076491 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:20.076459 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:12:20.099675 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:20.099636 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-2001b0-serving-cert\"" Nov 01 16:12:20.100955 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:20.100938 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 16:12:20.150369 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:20.150318 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a8db0d8-0d46-4003-9dc9-679dae13b8a1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2001b0-678dffcb5f-c94ht\" (UID: \"9a8db0d8-0d46-4003-9dc9-679dae13b8a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2001b0-678dffcb5f-c94ht" Nov 01 16:12:20.151062 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:20.151035 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a8db0d8-0d46-4003-9dc9-679dae13b8a1-maas-api-tls\") pod \"maas-api-e2e-inf-a-2001b0-678dffcb5f-c94ht\" (UID: \"9a8db0d8-0d46-4003-9dc9-679dae13b8a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2001b0-678dffcb5f-c94ht" Nov 01 16:12:20.151330 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:20.151309 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxwx8\" (UniqueName: \"kubernetes.io/projected/9a8db0d8-0d46-4003-9dc9-679dae13b8a1-kube-api-access-lxwx8\") pod \"maas-api-e2e-inf-a-2001b0-678dffcb5f-c94ht\" (UID: \"9a8db0d8-0d46-4003-9dc9-679dae13b8a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2001b0-678dffcb5f-c94ht" Nov 01 16:12:20.151657 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:20.151638 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a8db0d8-0d46-4003-9dc9-679dae13b8a1-metrics-certs\") pod \"maas-api-e2e-inf-a-2001b0-678dffcb5f-c94ht\" (UID: \"9a8db0d8-0d46-4003-9dc9-679dae13b8a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2001b0-678dffcb5f-c94ht" Nov 01 16:12:20.152048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:20.152001 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:12:20.152649 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:20.152630 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:12:20.219114 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:20.219010 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:12:20.241061 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:20.235539 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:12:20.262704 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:20.261620 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a8db0d8-0d46-4003-9dc9-679dae13b8a1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2001b0-678dffcb5f-c94ht\" (UID: \"9a8db0d8-0d46-4003-9dc9-679dae13b8a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2001b0-678dffcb5f-c94ht" Nov 01 16:12:20.262704 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:20.261750 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a8db0d8-0d46-4003-9dc9-679dae13b8a1-maas-api-tls\") pod \"maas-api-e2e-inf-a-2001b0-678dffcb5f-c94ht\" (UID: \"9a8db0d8-0d46-4003-9dc9-679dae13b8a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2001b0-678dffcb5f-c94ht" Nov 01 16:12:20.262704 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:20.261869 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxwx8\" (UniqueName: \"kubernetes.io/projected/9a8db0d8-0d46-4003-9dc9-679dae13b8a1-kube-api-access-lxwx8\") pod \"maas-api-e2e-inf-a-2001b0-678dffcb5f-c94ht\" (UID: \"9a8db0d8-0d46-4003-9dc9-679dae13b8a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2001b0-678dffcb5f-c94ht" Nov 01 16:12:20.262704 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:20.262061 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a8db0d8-0d46-4003-9dc9-679dae13b8a1-metrics-certs\") pod \"maas-api-e2e-inf-a-2001b0-678dffcb5f-c94ht\" (UID: \"9a8db0d8-0d46-4003-9dc9-679dae13b8a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2001b0-678dffcb5f-c94ht" Nov 01 16:12:20.296153 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:20.296121 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a8db0d8-0d46-4003-9dc9-679dae13b8a1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2001b0-678dffcb5f-c94ht\" (UID: \"9a8db0d8-0d46-4003-9dc9-679dae13b8a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2001b0-678dffcb5f-c94ht" Nov 01 16:12:20.322080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:20.322048 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" Nov 01 16:12:20.831653 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:20.831618 4363 patch_prober.go:29] interesting pod/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 01 16:12:20.832010 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:20.831979 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" podUID="145ad297-6cf1-4c70-ba0e-db5047a25f20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 01 16:12:20.839046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:20.835989 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a8db0d8-0d46-4003-9dc9-679dae13b8a1-metrics-certs\") pod \"maas-api-e2e-inf-a-2001b0-678dffcb5f-c94ht\" (UID: \"9a8db0d8-0d46-4003-9dc9-679dae13b8a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2001b0-678dffcb5f-c94ht" Nov 01 16:12:20.866787 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:20.866757 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:12:20.870860 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:20.870832 4363 patch_prober.go:29] interesting pod/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 01 16:12:20.871246 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:20.871178 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" podUID="ba075b49-a3d6-4813-bbcf-37f11e5550a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 01 16:12:20.879261 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:20.877529 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a8db0d8-0d46-4003-9dc9-679dae13b8a1-maas-api-tls\") pod \"maas-api-e2e-inf-a-2001b0-678dffcb5f-c94ht\" (UID: \"9a8db0d8-0d46-4003-9dc9-679dae13b8a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2001b0-678dffcb5f-c94ht" Nov 01 16:12:20.879261 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:20.878660 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": context deadline exceeded" Nov 01 16:12:20.905182 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:20.879578 4363 patch_prober.go:29] interesting pod/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 01 16:12:20.905182 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:20.879617 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" podUID="905f9285-a9c5-4a2a-abbf-8114e5ac9f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 01 16:12:20.905182 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:20.903413 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:12:20.969065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:20.909095 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxwx8\" (UniqueName: \"kubernetes.io/projected/9a8db0d8-0d46-4003-9dc9-679dae13b8a1-kube-api-access-lxwx8\") pod \"maas-api-e2e-inf-a-2001b0-678dffcb5f-c94ht\" (UID: \"9a8db0d8-0d46-4003-9dc9-679dae13b8a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2001b0-678dffcb5f-c94ht" Nov 01 16:12:21.096090 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:21.070117 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2001b0-678dffcb5f-c94ht" Nov 01 16:12:21.157509 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:21.156995 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:12:21.397135 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:21.397043 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="375990c6-6bd9-4c31-9a12-74732ca46a47" path="/var/lib/kubelet/pods/375990c6-6bd9-4c31-9a12-74732ca46a47/volumes" Nov 01 16:12:21.458295 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:21.457817 4363 patch_prober.go:29] interesting pod/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 01 16:12:21.458295 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:21.457876 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" podUID="ba075b49-a3d6-4813-bbcf-37f11e5550a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 01 16:12:21.515550 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:21.515155 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-04f248-6f57f7dc7-snmtt" Nov 01 16:12:21.525926 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:21.525879 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04f248-65b9669fcf-7fzhs" Nov 01 16:12:21.594065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:21.588661 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p"] Nov 01 16:12:21.594065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:21.588908 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" podUID="eb765068-ab2a-42f7-9f46-66cfa9745814" containerName="istio-proxy" containerID="cri-o://6cddaca8108a4980ee2009dee85d2fc4fae942fd1c85c0cffcd416842260b8ff" gracePeriod=30 Nov 01 16:12:21.630040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:21.604666 4363 patch_prober.go:29] interesting pod/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 01 16:12:21.655289 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:21.655192 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" podUID="905f9285-a9c5-4a2a-abbf-8114e5ac9f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 01 16:12:21.798274 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:21.789101 4363 patch_prober.go:29] interesting pod/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 01 16:12:21.798274 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:21.789158 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" podUID="145ad297-6cf1-4c70-ba0e-db5047a25f20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 01 16:12:21.968095 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:21.966709 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:12:21.983105 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:21.974261 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 16:12:22.306881 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:22.293911 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 16:12:22.306881 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:22.294974 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:12:22.344911 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:22.320098 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:12:22.368510 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:22.368483 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:12:22.748189 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:22.745073 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2001b0-678dffcb5f-c94ht"] Nov 01 16:12:22.761061 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:22.755258 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" Nov 01 16:12:22.771352 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:22.771328 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" Nov 01 16:12:22.775520 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:22.775497 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" Nov 01 16:12:22.776530 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:22.776503 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" Nov 01 16:12:22.800795 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:22.800769 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" Nov 01 16:12:22.927192 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:22.925350 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" Nov 01 16:12:22.927192 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:22.926668 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" Nov 01 16:12:22.938132 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:22.934133 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" Nov 01 16:12:23.689364 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:23.689323 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2001b0-678dffcb5f-c94ht" event={"ID":"9a8db0d8-0d46-4003-9dc9-679dae13b8a1","Type":"ContainerStarted","Data":"c42381096a57c0eb99a00a5c101b7a6f0cd6235fbb4998e913b4bf5182a4ba01"} Nov 01 16:12:23.689791 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:23.689376 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2001b0-678dffcb5f-c94ht" event={"ID":"9a8db0d8-0d46-4003-9dc9-679dae13b8a1","Type":"ContainerStarted","Data":"326ed11e67d7428c99c123c3e9e32e5046ebded06b8f07514bd1eaf06bf7c359"} Nov 01 16:12:23.689791 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:23.689436 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2001b0-678dffcb5f-c94ht" Nov 01 16:12:23.710295 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:23.710238 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2001b0-678dffcb5f-c94ht" podStartSLOduration=5.270650527 podStartE2EDuration="5.710223689s" totalImagesPullingTime="439.573162ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:12:23.706893669 +0000 UTC m=+1477.038631321" watchObservedRunningTime="2026-11-01 16:12:23.710223689 +0000 UTC m=+1477.041961338" Nov 01 16:12:23.901041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:23.900981 4363 patch_prober.go:29] interesting pod/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:12:23.901237 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:23.901058 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" podUID="eb765068-ab2a-42f7-9f46-66cfa9745814" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:12:25.243139 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:25.243104 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s"] Nov 01 16:12:25.248754 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:25.248732 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" Nov 01 16:12:25.260684 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:25.260655 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s"] Nov 01 16:12:25.272394 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:25.262624 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1c65e3\"/\"kube-root-ca.crt\"" Nov 01 16:12:25.272394 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:25.262680 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1c65e3\"/\"openshift-service-ca.crt\"" Nov 01 16:12:25.272394 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:25.262784 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1c65e3\"/\"test-model-1c65e3-kserve-self-signed-certs\"" Nov 01 16:12:25.272394 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:25.262863 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1c65e3\"/\"default-dockercfg-jx58v\"" Nov 01 16:12:25.304942 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:25.304906 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d92f7196-153d-46b0-9e5b-51667329cd02-dshm\") pod \"test-model-1c65e3-kserve-57bc967dfc-kjh5s\" (UID: \"d92f7196-153d-46b0-9e5b-51667329cd02\") " pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" Nov 01 16:12:25.305116 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:25.305047 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d92f7196-153d-46b0-9e5b-51667329cd02-model-cache\") pod \"test-model-1c65e3-kserve-57bc967dfc-kjh5s\" (UID: \"d92f7196-153d-46b0-9e5b-51667329cd02\") " pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" Nov 01 16:12:25.305262 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:25.305247 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d92f7196-153d-46b0-9e5b-51667329cd02-home\") pod \"test-model-1c65e3-kserve-57bc967dfc-kjh5s\" (UID: \"d92f7196-153d-46b0-9e5b-51667329cd02\") " pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" Nov 01 16:12:25.305303 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:25.305289 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d92f7196-153d-46b0-9e5b-51667329cd02-tls-certs\") pod \"test-model-1c65e3-kserve-57bc967dfc-kjh5s\" (UID: \"d92f7196-153d-46b0-9e5b-51667329cd02\") " pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" Nov 01 16:12:25.305356 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:25.305341 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2jwx\" (UniqueName: \"kubernetes.io/projected/d92f7196-153d-46b0-9e5b-51667329cd02-kube-api-access-h2jwx\") pod \"test-model-1c65e3-kserve-57bc967dfc-kjh5s\" (UID: \"d92f7196-153d-46b0-9e5b-51667329cd02\") " pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" Nov 01 16:12:25.305394 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:25.305382 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d92f7196-153d-46b0-9e5b-51667329cd02-tmp-dir\") pod \"test-model-1c65e3-kserve-57bc967dfc-kjh5s\" (UID: \"d92f7196-153d-46b0-9e5b-51667329cd02\") " pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" Nov 01 16:12:25.448902 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:25.446467 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d92f7196-153d-46b0-9e5b-51667329cd02-home\") pod \"test-model-1c65e3-kserve-57bc967dfc-kjh5s\" (UID: \"d92f7196-153d-46b0-9e5b-51667329cd02\") " pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" Nov 01 16:12:25.448902 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:25.446551 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d92f7196-153d-46b0-9e5b-51667329cd02-tls-certs\") pod \"test-model-1c65e3-kserve-57bc967dfc-kjh5s\" (UID: \"d92f7196-153d-46b0-9e5b-51667329cd02\") " pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" Nov 01 16:12:25.448902 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:25.446634 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2jwx\" (UniqueName: \"kubernetes.io/projected/d92f7196-153d-46b0-9e5b-51667329cd02-kube-api-access-h2jwx\") pod \"test-model-1c65e3-kserve-57bc967dfc-kjh5s\" (UID: \"d92f7196-153d-46b0-9e5b-51667329cd02\") " pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" Nov 01 16:12:25.459059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:25.450092 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d92f7196-153d-46b0-9e5b-51667329cd02-tmp-dir\") pod \"test-model-1c65e3-kserve-57bc967dfc-kjh5s\" (UID: \"d92f7196-153d-46b0-9e5b-51667329cd02\") " pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" Nov 01 16:12:25.459059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:25.454083 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d92f7196-153d-46b0-9e5b-51667329cd02-dshm\") pod \"test-model-1c65e3-kserve-57bc967dfc-kjh5s\" (UID: \"d92f7196-153d-46b0-9e5b-51667329cd02\") " pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" Nov 01 16:12:25.472305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:25.470065 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d92f7196-153d-46b0-9e5b-51667329cd02-model-cache\") pod \"test-model-1c65e3-kserve-57bc967dfc-kjh5s\" (UID: \"d92f7196-153d-46b0-9e5b-51667329cd02\") " pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" Nov 01 16:12:25.472305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:25.471635 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d92f7196-153d-46b0-9e5b-51667329cd02-tmp-dir\") pod \"test-model-1c65e3-kserve-57bc967dfc-kjh5s\" (UID: \"d92f7196-153d-46b0-9e5b-51667329cd02\") " pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" Nov 01 16:12:25.487036 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:25.483966 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d92f7196-153d-46b0-9e5b-51667329cd02-home\") pod \"test-model-1c65e3-kserve-57bc967dfc-kjh5s\" (UID: \"d92f7196-153d-46b0-9e5b-51667329cd02\") " pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" Nov 01 16:12:25.487036 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:25.484195 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d92f7196-153d-46b0-9e5b-51667329cd02-model-cache\") pod \"test-model-1c65e3-kserve-57bc967dfc-kjh5s\" (UID: \"d92f7196-153d-46b0-9e5b-51667329cd02\") " pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" Nov 01 16:12:25.539125 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:25.529669 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2jwx\" (UniqueName: \"kubernetes.io/projected/d92f7196-153d-46b0-9e5b-51667329cd02-kube-api-access-h2jwx\") pod \"test-model-1c65e3-kserve-57bc967dfc-kjh5s\" (UID: \"d92f7196-153d-46b0-9e5b-51667329cd02\") " pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" Nov 01 16:12:25.539364 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:25.532848 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d92f7196-153d-46b0-9e5b-51667329cd02-tls-certs\") pod \"test-model-1c65e3-kserve-57bc967dfc-kjh5s\" (UID: \"d92f7196-153d-46b0-9e5b-51667329cd02\") " pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" Nov 01 16:12:25.548045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:25.539582 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d92f7196-153d-46b0-9e5b-51667329cd02-dshm\") pod \"test-model-1c65e3-kserve-57bc967dfc-kjh5s\" (UID: \"d92f7196-153d-46b0-9e5b-51667329cd02\") " pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" Nov 01 16:12:25.592044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:25.591686 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" Nov 01 16:12:25.989771 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:12:25.989735 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd92f7196_153d_46b0_9e5b_51667329cd02.slice/crio-39f38d451b096b879c5d4d8bbd6a6f376f342ca8ce3074ab359bedd958227ab2 WatchSource:0}: Error finding container 39f38d451b096b879c5d4d8bbd6a6f376f342ca8ce3074ab359bedd958227ab2: Status 404 returned error can't find the container with id 39f38d451b096b879c5d4d8bbd6a6f376f342ca8ce3074ab359bedd958227ab2 Nov 01 16:12:26.002442 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:26.002403 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s"] Nov 01 16:12:26.739442 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:26.739405 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" event={"ID":"d92f7196-153d-46b0-9e5b-51667329cd02","Type":"ContainerStarted","Data":"2b7bc55b89ac0c3303ba9b335155b86b7354d2adacb94b078a951619c8e9bf46"} Nov 01 16:12:26.739852 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:26.739457 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" event={"ID":"d92f7196-153d-46b0-9e5b-51667329cd02","Type":"ContainerStarted","Data":"39f38d451b096b879c5d4d8bbd6a6f376f342ca8ce3074ab359bedd958227ab2"} Nov 01 16:12:26.758260 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:26.758206 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" podStartSLOduration=1.758188424 podStartE2EDuration="1.758188424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:12:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:12:26.754299139 +0000 UTC m=+1480.086036788" watchObservedRunningTime="2026-11-01 16:12:26.758188424 +0000 UTC m=+1480.089926075" Nov 01 16:12:27.596493 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:27.596453 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" Nov 01 16:12:27.606392 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:27.606369 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" Nov 01 16:12:27.757810 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:27.757764 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb765068-ab2a-42f7-9f46-66cfa9745814-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb765068-ab2a-42f7-9f46-66cfa9745814-workload-certs\") pod \"eb765068-ab2a-42f7-9f46-66cfa9745814\" (UID: \"eb765068-ab2a-42f7-9f46-66cfa9745814\") " Nov 01 16:12:27.758239 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:27.757870 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb765068-ab2a-42f7-9f46-66cfa9745814-istio-token\" (UniqueName: \"kubernetes.io/projected/eb765068-ab2a-42f7-9f46-66cfa9745814-istio-token\") pod \"eb765068-ab2a-42f7-9f46-66cfa9745814\" (UID: \"eb765068-ab2a-42f7-9f46-66cfa9745814\") " Nov 01 16:12:27.758239 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:27.757912 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb765068-ab2a-42f7-9f46-66cfa9745814-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb765068-ab2a-42f7-9f46-66cfa9745814-istio-podinfo\") pod \"eb765068-ab2a-42f7-9f46-66cfa9745814\" (UID: \"eb765068-ab2a-42f7-9f46-66cfa9745814\") " Nov 01 16:12:27.758239 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:27.758005 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb765068-ab2a-42f7-9f46-66cfa9745814-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb765068-ab2a-42f7-9f46-66cfa9745814-credential-socket\") pod \"eb765068-ab2a-42f7-9f46-66cfa9745814\" (UID: \"eb765068-ab2a-42f7-9f46-66cfa9745814\") " Nov 01 16:12:27.758239 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:27.758044 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb765068-ab2a-42f7-9f46-66cfa9745814-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb765068-ab2a-42f7-9f46-66cfa9745814-istiod-ca-cert\") pod \"eb765068-ab2a-42f7-9f46-66cfa9745814\" (UID: \"eb765068-ab2a-42f7-9f46-66cfa9745814\") " Nov 01 16:12:27.758239 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:27.758113 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb765068-ab2a-42f7-9f46-66cfa9745814-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb765068-ab2a-42f7-9f46-66cfa9745814-workload-socket\") pod \"eb765068-ab2a-42f7-9f46-66cfa9745814\" (UID: \"eb765068-ab2a-42f7-9f46-66cfa9745814\") " Nov 01 16:12:27.758239 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:27.758215 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb765068-ab2a-42f7-9f46-66cfa9745814-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb765068-ab2a-42f7-9f46-66cfa9745814-istio-data\") pod \"eb765068-ab2a-42f7-9f46-66cfa9745814\" (UID: \"eb765068-ab2a-42f7-9f46-66cfa9745814\") " Nov 01 16:12:27.758486 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:27.758471 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb765068-ab2a-42f7-9f46-66cfa9745814-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb765068-ab2a-42f7-9f46-66cfa9745814-istio-envoy\") pod \"eb765068-ab2a-42f7-9f46-66cfa9745814\" (UID: \"eb765068-ab2a-42f7-9f46-66cfa9745814\") " Nov 01 16:12:27.758521 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:27.758511 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb765068-ab2a-42f7-9f46-66cfa9745814-kube-api-access-458gr\" (UniqueName: \"kubernetes.io/projected/eb765068-ab2a-42f7-9f46-66cfa9745814-kube-api-access-458gr\") pod \"eb765068-ab2a-42f7-9f46-66cfa9745814\" (UID: \"eb765068-ab2a-42f7-9f46-66cfa9745814\") " Nov 01 16:12:27.761280 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:27.760268 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb765068-ab2a-42f7-9f46-66cfa9745814-workload-certs" pod "eb765068-ab2a-42f7-9f46-66cfa9745814" (UID: "eb765068-ab2a-42f7-9f46-66cfa9745814"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:12:27.761280 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:27.760462 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb765068-ab2a-42f7-9f46-66cfa9745814-workload-socket" pod "eb765068-ab2a-42f7-9f46-66cfa9745814" (UID: "eb765068-ab2a-42f7-9f46-66cfa9745814"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:12:27.761280 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:27.760755 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb765068-ab2a-42f7-9f46-66cfa9745814-istiod-ca-cert" pod "eb765068-ab2a-42f7-9f46-66cfa9745814" (UID: "eb765068-ab2a-42f7-9f46-66cfa9745814"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:12:27.761280 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:27.760990 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb765068-ab2a-42f7-9f46-66cfa9745814-istio-data" pod "eb765068-ab2a-42f7-9f46-66cfa9745814" (UID: "eb765068-ab2a-42f7-9f46-66cfa9745814"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:12:27.765626 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:27.764543 4363 generic.go:356] "Generic (PLEG): container finished" podID="eb765068-ab2a-42f7-9f46-66cfa9745814" containerID="6cddaca8108a4980ee2009dee85d2fc4fae942fd1c85c0cffcd416842260b8ff" exitCode=0 Nov 01 16:12:27.765626 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:27.765150 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb765068-ab2a-42f7-9f46-66cfa9745814-credential-socket" pod "eb765068-ab2a-42f7-9f46-66cfa9745814" (UID: "eb765068-ab2a-42f7-9f46-66cfa9745814"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:12:27.765626 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:27.765222 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" Nov 01 16:12:27.765626 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:27.765416 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" event={"ID":"eb765068-ab2a-42f7-9f46-66cfa9745814","Type":"ContainerDied","Data":"6cddaca8108a4980ee2009dee85d2fc4fae942fd1c85c0cffcd416842260b8ff"} Nov 01 16:12:27.765626 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:27.765462 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p" event={"ID":"eb765068-ab2a-42f7-9f46-66cfa9745814","Type":"ContainerDied","Data":"381a5272707ac081ee454043c777ffd8186669485bcad4b3ce705e1cd43195da"} Nov 01 16:12:27.765626 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:27.765489 4363 scope.go:122] "RemoveContainer" containerID="6cddaca8108a4980ee2009dee85d2fc4fae942fd1c85c0cffcd416842260b8ff" Nov 01 16:12:27.776751 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:27.776705 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb765068-ab2a-42f7-9f46-66cfa9745814-istio-token" pod "eb765068-ab2a-42f7-9f46-66cfa9745814" (UID: "eb765068-ab2a-42f7-9f46-66cfa9745814"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:12:27.776882 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:27.776855 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb765068-ab2a-42f7-9f46-66cfa9745814-kube-api-access-458gr" pod "eb765068-ab2a-42f7-9f46-66cfa9745814" (UID: "eb765068-ab2a-42f7-9f46-66cfa9745814"). InnerVolumeSpecName "kube-api-access-458gr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:12:27.777146 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:27.777109 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb765068-ab2a-42f7-9f46-66cfa9745814-istio-envoy" pod "eb765068-ab2a-42f7-9f46-66cfa9745814" (UID: "eb765068-ab2a-42f7-9f46-66cfa9745814"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:12:27.778166 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:27.778133 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb765068-ab2a-42f7-9f46-66cfa9745814-istio-podinfo" pod "eb765068-ab2a-42f7-9f46-66cfa9745814" (UID: "eb765068-ab2a-42f7-9f46-66cfa9745814"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:12:27.818185 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:27.818149 4363 scope.go:122] "RemoveContainer" containerID="a63f5e962708e8ed721205a200627275b54ed1319bcb23a29768210f81cbf0ce" Nov 01 16:12:27.829405 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:27.829383 4363 scope.go:122] "RemoveContainer" containerID="6cddaca8108a4980ee2009dee85d2fc4fae942fd1c85c0cffcd416842260b8ff" Nov 01 16:12:27.829703 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:12:27.829679 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cddaca8108a4980ee2009dee85d2fc4fae942fd1c85c0cffcd416842260b8ff\": container with ID starting with 6cddaca8108a4980ee2009dee85d2fc4fae942fd1c85c0cffcd416842260b8ff not found: ID does not exist" containerID="6cddaca8108a4980ee2009dee85d2fc4fae942fd1c85c0cffcd416842260b8ff" Nov 01 16:12:27.829773 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:27.829712 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cddaca8108a4980ee2009dee85d2fc4fae942fd1c85c0cffcd416842260b8ff"} err="failed to get container status \"6cddaca8108a4980ee2009dee85d2fc4fae942fd1c85c0cffcd416842260b8ff\": rpc error: code = NotFound desc = could not find container \"6cddaca8108a4980ee2009dee85d2fc4fae942fd1c85c0cffcd416842260b8ff\": container with ID starting with 6cddaca8108a4980ee2009dee85d2fc4fae942fd1c85c0cffcd416842260b8ff not found: ID does not exist" Nov 01 16:12:27.829773 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:27.829730 4363 scope.go:122] "RemoveContainer" containerID="a63f5e962708e8ed721205a200627275b54ed1319bcb23a29768210f81cbf0ce" Nov 01 16:12:27.830003 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:12:27.829976 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a63f5e962708e8ed721205a200627275b54ed1319bcb23a29768210f81cbf0ce\": container with ID starting with a63f5e962708e8ed721205a200627275b54ed1319bcb23a29768210f81cbf0ce not found: ID does not exist" containerID="a63f5e962708e8ed721205a200627275b54ed1319bcb23a29768210f81cbf0ce" Nov 01 16:12:27.830072 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:27.830028 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a63f5e962708e8ed721205a200627275b54ed1319bcb23a29768210f81cbf0ce"} err="failed to get container status \"a63f5e962708e8ed721205a200627275b54ed1319bcb23a29768210f81cbf0ce\": rpc error: code = NotFound desc = could not find container \"a63f5e962708e8ed721205a200627275b54ed1319bcb23a29768210f81cbf0ce\": container with ID starting with a63f5e962708e8ed721205a200627275b54ed1319bcb23a29768210f81cbf0ce not found: ID does not exist" Nov 01 16:12:27.861688 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:27.861656 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb765068-ab2a-42f7-9f46-66cfa9745814-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:27.861805 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:27.861691 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-458gr\" (UniqueName: \"kubernetes.io/projected/eb765068-ab2a-42f7-9f46-66cfa9745814-kube-api-access-458gr\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:27.861805 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:27.861706 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb765068-ab2a-42f7-9f46-66cfa9745814-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:27.861805 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:27.861721 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb765068-ab2a-42f7-9f46-66cfa9745814-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:27.861805 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:27.861749 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb765068-ab2a-42f7-9f46-66cfa9745814-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:27.861805 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:27.861772 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb765068-ab2a-42f7-9f46-66cfa9745814-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:27.861805 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:27.861782 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb765068-ab2a-42f7-9f46-66cfa9745814-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:27.861805 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:27.861791 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb765068-ab2a-42f7-9f46-66cfa9745814-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:27.861805 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:27.861800 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb765068-ab2a-42f7-9f46-66cfa9745814-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:28.090036 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:28.089988 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p"] Nov 01 16:12:28.094757 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:28.094736 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-022f62-openshift-default-c7698fd8f-24f5p"] Nov 01 16:12:28.801166 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:28.801132 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb765068-ab2a-42f7-9f46-66cfa9745814" path="/var/lib/kubelet/pods/eb765068-ab2a-42f7-9f46-66cfa9745814/volumes" Nov 01 16:12:30.614304 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:30.614268 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-04f248-668fc47648-hptb8"] Nov 01 16:12:30.621048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:30.615821 4363 cpu_manager.go:414] "containerMap: removing container" podUID="eb765068-ab2a-42f7-9f46-66cfa9745814" containerName="istio-proxy" Nov 01 16:12:30.621048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:30.615845 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb765068-ab2a-42f7-9f46-66cfa9745814" containerName="istio-proxy" Nov 01 16:12:30.621048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:30.615876 4363 cpu_manager.go:414] "containerMap: removing container" podUID="eb765068-ab2a-42f7-9f46-66cfa9745814" containerName="istio-proxy" Nov 01 16:12:30.621048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:30.615884 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb765068-ab2a-42f7-9f46-66cfa9745814" containerName="istio-proxy" Nov 01 16:12:30.621048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:30.616128 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb765068-ab2a-42f7-9f46-66cfa9745814" containerName="istio-proxy" Nov 01 16:12:30.621048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:30.616149 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb765068-ab2a-42f7-9f46-66cfa9745814" containerName="istio-proxy" Nov 01 16:12:30.634511 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:30.634477 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-04f248-668fc47648-hptb8"] Nov 01 16:12:30.634792 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:30.634599 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04f248-668fc47648-hptb8" Nov 01 16:12:30.640652 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:30.640618 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-04f248-serving-cert\"" Nov 01 16:12:30.653766 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:30.653736 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67731def-6473-4c83-a367-6960146d4907-maas-api-tls\") pod \"maas-api-e2e-inf-a-04f248-668fc47648-hptb8\" (UID: \"67731def-6473-4c83-a367-6960146d4907\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04f248-668fc47648-hptb8" Nov 01 16:12:30.653898 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:30.653870 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67731def-6473-4c83-a367-6960146d4907-openshift-service-ca\") pod \"maas-api-e2e-inf-a-04f248-668fc47648-hptb8\" (UID: \"67731def-6473-4c83-a367-6960146d4907\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04f248-668fc47648-hptb8" Nov 01 16:12:30.653995 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:30.653981 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbbv2\" (UniqueName: \"kubernetes.io/projected/67731def-6473-4c83-a367-6960146d4907-kube-api-access-kbbv2\") pod \"maas-api-e2e-inf-a-04f248-668fc47648-hptb8\" (UID: \"67731def-6473-4c83-a367-6960146d4907\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04f248-668fc47648-hptb8" Nov 01 16:12:30.654107 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:30.654094 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67731def-6473-4c83-a367-6960146d4907-metrics-certs\") pod \"maas-api-e2e-inf-a-04f248-668fc47648-hptb8\" (UID: \"67731def-6473-4c83-a367-6960146d4907\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04f248-668fc47648-hptb8" Nov 01 16:12:30.732892 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:30.732811 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2001b0-678dffcb5f-c94ht" Nov 01 16:12:30.755853 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:30.755818 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67731def-6473-4c83-a367-6960146d4907-maas-api-tls\") pod \"maas-api-e2e-inf-a-04f248-668fc47648-hptb8\" (UID: \"67731def-6473-4c83-a367-6960146d4907\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04f248-668fc47648-hptb8" Nov 01 16:12:30.756064 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:30.755899 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67731def-6473-4c83-a367-6960146d4907-openshift-service-ca\") pod \"maas-api-e2e-inf-a-04f248-668fc47648-hptb8\" (UID: \"67731def-6473-4c83-a367-6960146d4907\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04f248-668fc47648-hptb8" Nov 01 16:12:30.756064 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:30.755962 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbbv2\" (UniqueName: \"kubernetes.io/projected/67731def-6473-4c83-a367-6960146d4907-kube-api-access-kbbv2\") pod \"maas-api-e2e-inf-a-04f248-668fc47648-hptb8\" (UID: \"67731def-6473-4c83-a367-6960146d4907\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04f248-668fc47648-hptb8" Nov 01 16:12:30.756259 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:30.756069 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67731def-6473-4c83-a367-6960146d4907-metrics-certs\") pod \"maas-api-e2e-inf-a-04f248-668fc47648-hptb8\" (UID: \"67731def-6473-4c83-a367-6960146d4907\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04f248-668fc47648-hptb8" Nov 01 16:12:30.757012 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:30.756982 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67731def-6473-4c83-a367-6960146d4907-openshift-service-ca\") pod \"maas-api-e2e-inf-a-04f248-668fc47648-hptb8\" (UID: \"67731def-6473-4c83-a367-6960146d4907\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04f248-668fc47648-hptb8" Nov 01 16:12:30.759327 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:30.759300 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67731def-6473-4c83-a367-6960146d4907-metrics-certs\") pod \"maas-api-e2e-inf-a-04f248-668fc47648-hptb8\" (UID: \"67731def-6473-4c83-a367-6960146d4907\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04f248-668fc47648-hptb8" Nov 01 16:12:30.759457 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:30.759431 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67731def-6473-4c83-a367-6960146d4907-maas-api-tls\") pod \"maas-api-e2e-inf-a-04f248-668fc47648-hptb8\" (UID: \"67731def-6473-4c83-a367-6960146d4907\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04f248-668fc47648-hptb8" Nov 01 16:12:30.764219 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:30.764194 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbbv2\" (UniqueName: \"kubernetes.io/projected/67731def-6473-4c83-a367-6960146d4907-kube-api-access-kbbv2\") pod \"maas-api-e2e-inf-a-04f248-668fc47648-hptb8\" (UID: \"67731def-6473-4c83-a367-6960146d4907\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04f248-668fc47648-hptb8" Nov 01 16:12:30.962184 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:30.960901 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04f248-668fc47648-hptb8" Nov 01 16:12:31.554395 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:31.554365 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-04f248-668fc47648-hptb8"] Nov 01 16:12:31.555965 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:12:31.555933 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67731def_6473_4c83_a367_6960146d4907.slice/crio-bea1e34ea5c7247d57a679217b6c1c05baa943d6b49df03c15881f11ce632453 WatchSource:0}: Error finding container bea1e34ea5c7247d57a679217b6c1c05baa943d6b49df03c15881f11ce632453: Status 404 returned error can't find the container with id bea1e34ea5c7247d57a679217b6c1c05baa943d6b49df03c15881f11ce632453 Nov 01 16:12:31.814666 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:31.814579 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04f248-668fc47648-hptb8" event={"ID":"67731def-6473-4c83-a367-6960146d4907","Type":"ContainerStarted","Data":"bea1e34ea5c7247d57a679217b6c1c05baa943d6b49df03c15881f11ce632453"} Nov 01 16:12:32.144413 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:32.144375 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-58b0d1-786cf8cdb9-cbftv"] Nov 01 16:12:32.164621 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:32.163950 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-58b0d1-5455d7568c-jh7m4"] Nov 01 16:12:32.246475 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:32.246439 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m"] Nov 01 16:12:32.246736 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:32.246708 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" podUID="420774ff-0dfc-44ef-a9c3-19a41f76a985" containerName="istio-proxy" containerID="cri-o://681ff7838753f6e911ecadf9f3b2eb5b04772a76a9cfbff05dece650caa4a517" gracePeriod=30 Nov 01 16:12:32.928089 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:32.915968 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04f248-668fc47648-hptb8" event={"ID":"67731def-6473-4c83-a367-6960146d4907","Type":"ContainerStarted","Data":"af61302b33c10503b5cddee91256dfa122c474e2923d7ccf92ffce91ba14d3da"} Nov 01 16:12:32.928089 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:32.917319 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04f248-668fc47648-hptb8" Nov 01 16:12:32.965847 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:32.965772 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04f248-668fc47648-hptb8" podStartSLOduration=2.595510418 podStartE2EDuration="2.965749052s" totalImagesPullingTime="370.238634ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:12:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:12:32.960197278 +0000 UTC m=+1486.291934925" watchObservedRunningTime="2026-11-01 16:12:32.965749052 +0000 UTC m=+1486.297486703" Nov 01 16:12:34.772738 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:34.772704 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6"] Nov 01 16:12:34.783060 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:34.778044 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" Nov 01 16:12:34.801383 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:34.801352 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2ce08a\"/\"kube-root-ca.crt\"" Nov 01 16:12:34.801383 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:34.801371 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2ce08a\"/\"openshift-service-ca.crt\"" Nov 01 16:12:34.806448 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:34.806222 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2ce08a\"/\"test-model-2ce08a-kserve-self-signed-certs\"" Nov 01 16:12:34.806448 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:34.806326 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2ce08a\"/\"default-dockercfg-4cmml\"" Nov 01 16:12:34.816027 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:34.815985 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6"] Nov 01 16:12:34.924649 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:34.924613 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37773d39-9d52-4b1d-94c7-68e02714a476-home\") pod \"test-model-2ce08a-kserve-64c8c486fc-vlgr6\" (UID: \"37773d39-9d52-4b1d-94c7-68e02714a476\") " pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" Nov 01 16:12:34.924848 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:34.924735 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37773d39-9d52-4b1d-94c7-68e02714a476-dshm\") pod \"test-model-2ce08a-kserve-64c8c486fc-vlgr6\" (UID: \"37773d39-9d52-4b1d-94c7-68e02714a476\") " pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" Nov 01 16:12:34.924960 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:34.924942 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37773d39-9d52-4b1d-94c7-68e02714a476-model-cache\") pod \"test-model-2ce08a-kserve-64c8c486fc-vlgr6\" (UID: \"37773d39-9d52-4b1d-94c7-68e02714a476\") " pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" Nov 01 16:12:34.925155 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:34.925139 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cwxh\" (UniqueName: \"kubernetes.io/projected/37773d39-9d52-4b1d-94c7-68e02714a476-kube-api-access-2cwxh\") pod \"test-model-2ce08a-kserve-64c8c486fc-vlgr6\" (UID: \"37773d39-9d52-4b1d-94c7-68e02714a476\") " pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" Nov 01 16:12:34.925204 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:34.925175 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37773d39-9d52-4b1d-94c7-68e02714a476-tls-certs\") pod \"test-model-2ce08a-kserve-64c8c486fc-vlgr6\" (UID: \"37773d39-9d52-4b1d-94c7-68e02714a476\") " pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" Nov 01 16:12:34.925247 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:34.925233 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37773d39-9d52-4b1d-94c7-68e02714a476-tmp-dir\") pod \"test-model-2ce08a-kserve-64c8c486fc-vlgr6\" (UID: \"37773d39-9d52-4b1d-94c7-68e02714a476\") " pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" Nov 01 16:12:35.035872 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:35.035783 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37773d39-9d52-4b1d-94c7-68e02714a476-model-cache\") pod \"test-model-2ce08a-kserve-64c8c486fc-vlgr6\" (UID: \"37773d39-9d52-4b1d-94c7-68e02714a476\") " pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" Nov 01 16:12:35.037432 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:35.037318 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37773d39-9d52-4b1d-94c7-68e02714a476-model-cache\") pod \"test-model-2ce08a-kserve-64c8c486fc-vlgr6\" (UID: \"37773d39-9d52-4b1d-94c7-68e02714a476\") " pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" Nov 01 16:12:35.037784 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:35.037762 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cwxh\" (UniqueName: \"kubernetes.io/projected/37773d39-9d52-4b1d-94c7-68e02714a476-kube-api-access-2cwxh\") pod \"test-model-2ce08a-kserve-64c8c486fc-vlgr6\" (UID: \"37773d39-9d52-4b1d-94c7-68e02714a476\") " pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" Nov 01 16:12:35.057591 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:35.057565 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37773d39-9d52-4b1d-94c7-68e02714a476-tls-certs\") pod \"test-model-2ce08a-kserve-64c8c486fc-vlgr6\" (UID: \"37773d39-9d52-4b1d-94c7-68e02714a476\") " pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" Nov 01 16:12:35.058500 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:35.058479 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37773d39-9d52-4b1d-94c7-68e02714a476-tmp-dir\") pod \"test-model-2ce08a-kserve-64c8c486fc-vlgr6\" (UID: \"37773d39-9d52-4b1d-94c7-68e02714a476\") " pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" Nov 01 16:12:35.058966 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:35.058950 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37773d39-9d52-4b1d-94c7-68e02714a476-home\") pod \"test-model-2ce08a-kserve-64c8c486fc-vlgr6\" (UID: \"37773d39-9d52-4b1d-94c7-68e02714a476\") " pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" Nov 01 16:12:35.059425 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:35.059410 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37773d39-9d52-4b1d-94c7-68e02714a476-dshm\") pod \"test-model-2ce08a-kserve-64c8c486fc-vlgr6\" (UID: \"37773d39-9d52-4b1d-94c7-68e02714a476\") " pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" Nov 01 16:12:35.060205 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:35.058866 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37773d39-9d52-4b1d-94c7-68e02714a476-tmp-dir\") pod \"test-model-2ce08a-kserve-64c8c486fc-vlgr6\" (UID: \"37773d39-9d52-4b1d-94c7-68e02714a476\") " pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" Nov 01 16:12:35.060350 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:35.059290 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37773d39-9d52-4b1d-94c7-68e02714a476-home\") pod \"test-model-2ce08a-kserve-64c8c486fc-vlgr6\" (UID: \"37773d39-9d52-4b1d-94c7-68e02714a476\") " pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" Nov 01 16:12:35.079039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:35.075700 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37773d39-9d52-4b1d-94c7-68e02714a476-tls-certs\") pod \"test-model-2ce08a-kserve-64c8c486fc-vlgr6\" (UID: \"37773d39-9d52-4b1d-94c7-68e02714a476\") " pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" Nov 01 16:12:35.079039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:35.077355 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37773d39-9d52-4b1d-94c7-68e02714a476-dshm\") pod \"test-model-2ce08a-kserve-64c8c486fc-vlgr6\" (UID: \"37773d39-9d52-4b1d-94c7-68e02714a476\") " pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" Nov 01 16:12:35.105039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:35.099135 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cwxh\" (UniqueName: \"kubernetes.io/projected/37773d39-9d52-4b1d-94c7-68e02714a476-kube-api-access-2cwxh\") pod \"test-model-2ce08a-kserve-64c8c486fc-vlgr6\" (UID: \"37773d39-9d52-4b1d-94c7-68e02714a476\") " pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" Nov 01 16:12:35.122049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:35.120355 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" Nov 01 16:12:35.441290 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:35.441251 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-166778-cd665bd8b-xhwj6"] Nov 01 16:12:35.441555 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:35.441499 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-166778-cd665bd8b-xhwj6" podUID="dcf26f07-340c-4852-8213-e5ab594f33b0" containerName="maas-api" containerID="cri-o://bdb2f5f39501aad360fb48d8ba8c9471a51599c9a354138238e2688251f85f32" gracePeriod=30 Nov 01 16:12:35.585886 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:35.585848 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6"] Nov 01 16:12:35.586389 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:12:35.586348 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37773d39_9d52_4b1d_94c7_68e02714a476.slice/crio-1e2eadc95d28c75dc5d3479e5b899551a3f4638d73cd21abed6f73edb0862892 WatchSource:0}: Error finding container 1e2eadc95d28c75dc5d3479e5b899551a3f4638d73cd21abed6f73edb0862892: Status 404 returned error can't find the container with id 1e2eadc95d28c75dc5d3479e5b899551a3f4638d73cd21abed6f73edb0862892 Nov 01 16:12:35.596189 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:35.596163 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" Nov 01 16:12:35.616681 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:35.616405 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" Nov 01 16:12:36.090775 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:36.090745 4363 generic.go:356] "Generic (PLEG): container finished" podID="dcf26f07-340c-4852-8213-e5ab594f33b0" containerID="bdb2f5f39501aad360fb48d8ba8c9471a51599c9a354138238e2688251f85f32" exitCode=0 Nov 01 16:12:36.091492 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:36.091464 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-166778-cd665bd8b-xhwj6" event={"ID":"dcf26f07-340c-4852-8213-e5ab594f33b0","Type":"ContainerDied","Data":"bdb2f5f39501aad360fb48d8ba8c9471a51599c9a354138238e2688251f85f32"} Nov 01 16:12:36.094234 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:36.094205 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" event={"ID":"37773d39-9d52-4b1d-94c7-68e02714a476","Type":"ContainerStarted","Data":"1e2eadc95d28c75dc5d3479e5b899551a3f4638d73cd21abed6f73edb0862892"} Nov 01 16:12:36.146703 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:36.146644 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" Nov 01 16:12:36.272431 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:36.272169 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-166778-cd665bd8b-xhwj6" Nov 01 16:12:36.472037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:36.426570 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcf26f07-340c-4852-8213-e5ab594f33b0-kube-api-access-fbdt6\" (UniqueName: \"kubernetes.io/projected/dcf26f07-340c-4852-8213-e5ab594f33b0-kube-api-access-fbdt6\") pod \"dcf26f07-340c-4852-8213-e5ab594f33b0\" (UID: \"dcf26f07-340c-4852-8213-e5ab594f33b0\") " Nov 01 16:12:36.472037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:36.450572 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcf26f07-340c-4852-8213-e5ab594f33b0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcf26f07-340c-4852-8213-e5ab594f33b0-maas-api-tls\") pod \"dcf26f07-340c-4852-8213-e5ab594f33b0\" (UID: \"dcf26f07-340c-4852-8213-e5ab594f33b0\") " Nov 01 16:12:36.472037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:36.450717 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcf26f07-340c-4852-8213-e5ab594f33b0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcf26f07-340c-4852-8213-e5ab594f33b0-metrics-certs\") pod \"dcf26f07-340c-4852-8213-e5ab594f33b0\" (UID: \"dcf26f07-340c-4852-8213-e5ab594f33b0\") " Nov 01 16:12:36.472037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:36.451138 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcf26f07-340c-4852-8213-e5ab594f33b0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcf26f07-340c-4852-8213-e5ab594f33b0-openshift-service-ca\") pod \"dcf26f07-340c-4852-8213-e5ab594f33b0\" (UID: \"dcf26f07-340c-4852-8213-e5ab594f33b0\") " Nov 01 16:12:36.494443 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:36.494396 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcf26f07-340c-4852-8213-e5ab594f33b0-openshift-service-ca" pod "dcf26f07-340c-4852-8213-e5ab594f33b0" (UID: "dcf26f07-340c-4852-8213-e5ab594f33b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:12:36.500211 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:36.497640 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcf26f07-340c-4852-8213-e5ab594f33b0-maas-api-tls" pod "dcf26f07-340c-4852-8213-e5ab594f33b0" (UID: "dcf26f07-340c-4852-8213-e5ab594f33b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:12:36.512086 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:36.510295 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcf26f07-340c-4852-8213-e5ab594f33b0-metrics-certs" pod "dcf26f07-340c-4852-8213-e5ab594f33b0" (UID: "dcf26f07-340c-4852-8213-e5ab594f33b0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:12:36.516673 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:36.516636 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcf26f07-340c-4852-8213-e5ab594f33b0-kube-api-access-fbdt6" pod "dcf26f07-340c-4852-8213-e5ab594f33b0" (UID: "dcf26f07-340c-4852-8213-e5ab594f33b0"). InnerVolumeSpecName "kube-api-access-fbdt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:12:36.554912 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:36.554883 4363 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcf26f07-340c-4852-8213-e5ab594f33b0-openshift-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:36.554912 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:36.554912 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbdt6\" (UniqueName: \"kubernetes.io/projected/dcf26f07-340c-4852-8213-e5ab594f33b0-kube-api-access-fbdt6\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:36.555075 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:36.554923 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcf26f07-340c-4852-8213-e5ab594f33b0-maas-api-tls\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:36.555075 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:36.554933 4363 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcf26f07-340c-4852-8213-e5ab594f33b0-metrics-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:37.107414 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.107378 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" event={"ID":"37773d39-9d52-4b1d-94c7-68e02714a476","Type":"ContainerStarted","Data":"b92daf01c2ff7518c17a6c102771978ffb6ef1ef7f54c62ac98853a7390fc354"} Nov 01 16:12:37.110227 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.110197 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-166778-cd665bd8b-xhwj6" event={"ID":"dcf26f07-340c-4852-8213-e5ab594f33b0","Type":"ContainerDied","Data":"07eefb8e53c48488e1fdde0f07f4198a72f3ad7a892647275012b1a52da46b73"} Nov 01 16:12:37.110377 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.110234 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-166778-cd665bd8b-xhwj6" Nov 01 16:12:37.110377 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.110251 4363 scope.go:122] "RemoveContainer" containerID="bdb2f5f39501aad360fb48d8ba8c9471a51599c9a354138238e2688251f85f32" Nov 01 16:12:37.122247 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.122226 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" Nov 01 16:12:37.144783 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.144753 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-58b0d1-786cf8cdb9-cbftv" podUID="4e4adc20-5cbc-485c-aaee-acca31664813" containerName="payload-processing" containerID="cri-o://9eef98366200810482cd73cc8152600f0fb7e132205dfcbe5c37a2413a45623e" gracePeriod=25 Nov 01 16:12:37.164298 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.164262 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-58b0d1-5455d7568c-jh7m4" podUID="c20f4fe4-124a-446f-89b5-e526085a8cea" containerName="payload-pre-processing" containerID="cri-o://4c12d3fb4ad3ff5de5e328b3345b40fe55c177923b08f016191b276813ab6de3" gracePeriod=25 Nov 01 16:12:37.179441 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.178135 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" podStartSLOduration=3.178117245 podStartE2EDuration="3.178117245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:12:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:12:37.173423478 +0000 UTC m=+1490.505161129" watchObservedRunningTime="2026-11-01 16:12:37.178117245 +0000 UTC m=+1490.509854897" Nov 01 16:12:37.233622 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.233586 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-166778-cd665bd8b-xhwj6"] Nov 01 16:12:37.239464 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.239440 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-166778-cd665bd8b-xhwj6"] Nov 01 16:12:37.605628 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.605603 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-58b0d1-786cf8cdb9-cbftv" Nov 01 16:12:37.666930 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.666904 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-58b0d1-5455d7568c-jh7m4" Nov 01 16:12:37.714339 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.714308 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e4adc20-5cbc-485c-aaee-acca31664813-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4adc20-5cbc-485c-aaee-acca31664813-plugins-config-volume\") pod \"4e4adc20-5cbc-485c-aaee-acca31664813\" (UID: \"4e4adc20-5cbc-485c-aaee-acca31664813\") " Nov 01 16:12:37.714484 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.714459 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c20f4fe4-124a-446f-89b5-e526085a8cea-kube-api-access-b4ccl\" (UniqueName: \"kubernetes.io/projected/c20f4fe4-124a-446f-89b5-e526085a8cea-kube-api-access-b4ccl\") pod \"c20f4fe4-124a-446f-89b5-e526085a8cea\" (UID: \"c20f4fe4-124a-446f-89b5-e526085a8cea\") " Nov 01 16:12:37.714732 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.714713 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e4adc20-5cbc-485c-aaee-acca31664813-kube-api-access-9qltc\" (UniqueName: \"kubernetes.io/projected/4e4adc20-5cbc-485c-aaee-acca31664813-kube-api-access-9qltc\") pod \"4e4adc20-5cbc-485c-aaee-acca31664813\" (UID: \"4e4adc20-5cbc-485c-aaee-acca31664813\") " Nov 01 16:12:37.714817 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.714775 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c20f4fe4-124a-446f-89b5-e526085a8cea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c20f4fe4-124a-446f-89b5-e526085a8cea-plugins-config-volume\") pod \"c20f4fe4-124a-446f-89b5-e526085a8cea\" (UID: \"c20f4fe4-124a-446f-89b5-e526085a8cea\") " Nov 01 16:12:37.716202 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.715321 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e4adc20-5cbc-485c-aaee-acca31664813-plugins-config-volume" pod "4e4adc20-5cbc-485c-aaee-acca31664813" (UID: "4e4adc20-5cbc-485c-aaee-acca31664813"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:12:37.716202 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.715742 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c20f4fe4-124a-446f-89b5-e526085a8cea-plugins-config-volume" pod "c20f4fe4-124a-446f-89b5-e526085a8cea" (UID: "c20f4fe4-124a-446f-89b5-e526085a8cea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:12:37.716202 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.716149 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c20f4fe4-124a-446f-89b5-e526085a8cea-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:37.716202 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.716172 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4adc20-5cbc-485c-aaee-acca31664813-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:37.719471 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.719436 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c20f4fe4-124a-446f-89b5-e526085a8cea-kube-api-access-b4ccl" pod "c20f4fe4-124a-446f-89b5-e526085a8cea" (UID: "c20f4fe4-124a-446f-89b5-e526085a8cea"). InnerVolumeSpecName "kube-api-access-b4ccl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:12:37.719686 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.719660 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4adc20-5cbc-485c-aaee-acca31664813-kube-api-access-9qltc" pod "4e4adc20-5cbc-485c-aaee-acca31664813" (UID: "4e4adc20-5cbc-485c-aaee-acca31664813"). InnerVolumeSpecName "kube-api-access-9qltc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:12:37.728517 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.728492 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" Nov 01 16:12:37.817108 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.817066 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/420774ff-0dfc-44ef-a9c3-19a41f76a985-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/420774ff-0dfc-44ef-a9c3-19a41f76a985-istio-data\") pod \"420774ff-0dfc-44ef-a9c3-19a41f76a985\" (UID: \"420774ff-0dfc-44ef-a9c3-19a41f76a985\") " Nov 01 16:12:37.817279 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.817144 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/420774ff-0dfc-44ef-a9c3-19a41f76a985-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/420774ff-0dfc-44ef-a9c3-19a41f76a985-credential-socket\") pod \"420774ff-0dfc-44ef-a9c3-19a41f76a985\" (UID: \"420774ff-0dfc-44ef-a9c3-19a41f76a985\") " Nov 01 16:12:37.817376 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.817322 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/420774ff-0dfc-44ef-a9c3-19a41f76a985-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/420774ff-0dfc-44ef-a9c3-19a41f76a985-istio-podinfo\") pod \"420774ff-0dfc-44ef-a9c3-19a41f76a985\" (UID: \"420774ff-0dfc-44ef-a9c3-19a41f76a985\") " Nov 01 16:12:37.817376 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.817359 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/420774ff-0dfc-44ef-a9c3-19a41f76a985-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/420774ff-0dfc-44ef-a9c3-19a41f76a985-workload-certs\") pod \"420774ff-0dfc-44ef-a9c3-19a41f76a985\" (UID: \"420774ff-0dfc-44ef-a9c3-19a41f76a985\") " Nov 01 16:12:37.817486 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.817388 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/420774ff-0dfc-44ef-a9c3-19a41f76a985-istio-data" pod "420774ff-0dfc-44ef-a9c3-19a41f76a985" (UID: "420774ff-0dfc-44ef-a9c3-19a41f76a985"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:12:37.817562 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.817538 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/420774ff-0dfc-44ef-a9c3-19a41f76a985-kube-api-access-fjqgv\" (UniqueName: \"kubernetes.io/projected/420774ff-0dfc-44ef-a9c3-19a41f76a985-kube-api-access-fjqgv\") pod \"420774ff-0dfc-44ef-a9c3-19a41f76a985\" (UID: \"420774ff-0dfc-44ef-a9c3-19a41f76a985\") " Nov 01 16:12:37.817637 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.817618 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/420774ff-0dfc-44ef-a9c3-19a41f76a985-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/420774ff-0dfc-44ef-a9c3-19a41f76a985-istiod-ca-cert\") pod \"420774ff-0dfc-44ef-a9c3-19a41f76a985\" (UID: \"420774ff-0dfc-44ef-a9c3-19a41f76a985\") " Nov 01 16:12:37.817704 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.817664 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/420774ff-0dfc-44ef-a9c3-19a41f76a985-istio-token\" (UniqueName: \"kubernetes.io/projected/420774ff-0dfc-44ef-a9c3-19a41f76a985-istio-token\") pod \"420774ff-0dfc-44ef-a9c3-19a41f76a985\" (UID: \"420774ff-0dfc-44ef-a9c3-19a41f76a985\") " Nov 01 16:12:37.817766 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.817733 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/420774ff-0dfc-44ef-a9c3-19a41f76a985-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/420774ff-0dfc-44ef-a9c3-19a41f76a985-workload-socket\") pod \"420774ff-0dfc-44ef-a9c3-19a41f76a985\" (UID: \"420774ff-0dfc-44ef-a9c3-19a41f76a985\") " Nov 01 16:12:37.817872 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.817855 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/420774ff-0dfc-44ef-a9c3-19a41f76a985-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/420774ff-0dfc-44ef-a9c3-19a41f76a985-istio-envoy\") pod \"420774ff-0dfc-44ef-a9c3-19a41f76a985\" (UID: \"420774ff-0dfc-44ef-a9c3-19a41f76a985\") " Nov 01 16:12:37.818054 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.818009 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/420774ff-0dfc-44ef-a9c3-19a41f76a985-istiod-ca-cert" pod "420774ff-0dfc-44ef-a9c3-19a41f76a985" (UID: "420774ff-0dfc-44ef-a9c3-19a41f76a985"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:12:37.818829 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.818262 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/420774ff-0dfc-44ef-a9c3-19a41f76a985-workload-socket" pod "420774ff-0dfc-44ef-a9c3-19a41f76a985" (UID: "420774ff-0dfc-44ef-a9c3-19a41f76a985"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:12:37.818829 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.818738 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4ccl\" (UniqueName: \"kubernetes.io/projected/c20f4fe4-124a-446f-89b5-e526085a8cea-kube-api-access-b4ccl\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:37.818829 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.818762 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/420774ff-0dfc-44ef-a9c3-19a41f76a985-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:37.818829 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.818776 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/420774ff-0dfc-44ef-a9c3-19a41f76a985-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:37.818829 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.818791 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/420774ff-0dfc-44ef-a9c3-19a41f76a985-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:37.818829 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.818805 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qltc\" (UniqueName: \"kubernetes.io/projected/4e4adc20-5cbc-485c-aaee-acca31664813-kube-api-access-9qltc\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:37.820116 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.819249 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/420774ff-0dfc-44ef-a9c3-19a41f76a985-credential-socket" pod "420774ff-0dfc-44ef-a9c3-19a41f76a985" (UID: "420774ff-0dfc-44ef-a9c3-19a41f76a985"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:12:37.820297 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.820267 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/420774ff-0dfc-44ef-a9c3-19a41f76a985-workload-certs" pod "420774ff-0dfc-44ef-a9c3-19a41f76a985" (UID: "420774ff-0dfc-44ef-a9c3-19a41f76a985"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:12:37.820297 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.820280 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/420774ff-0dfc-44ef-a9c3-19a41f76a985-istio-podinfo" pod "420774ff-0dfc-44ef-a9c3-19a41f76a985" (UID: "420774ff-0dfc-44ef-a9c3-19a41f76a985"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:12:37.820424 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.820407 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/420774ff-0dfc-44ef-a9c3-19a41f76a985-istio-token" pod "420774ff-0dfc-44ef-a9c3-19a41f76a985" (UID: "420774ff-0dfc-44ef-a9c3-19a41f76a985"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:12:37.820481 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.820471 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/420774ff-0dfc-44ef-a9c3-19a41f76a985-kube-api-access-fjqgv" pod "420774ff-0dfc-44ef-a9c3-19a41f76a985" (UID: "420774ff-0dfc-44ef-a9c3-19a41f76a985"). InnerVolumeSpecName "kube-api-access-fjqgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:12:37.821757 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.821732 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/420774ff-0dfc-44ef-a9c3-19a41f76a985-istio-envoy" pod "420774ff-0dfc-44ef-a9c3-19a41f76a985" (UID: "420774ff-0dfc-44ef-a9c3-19a41f76a985"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:12:37.920730 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.920646 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjqgv\" (UniqueName: \"kubernetes.io/projected/420774ff-0dfc-44ef-a9c3-19a41f76a985-kube-api-access-fjqgv\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:37.920730 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.920683 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/420774ff-0dfc-44ef-a9c3-19a41f76a985-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:37.920730 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.920700 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/420774ff-0dfc-44ef-a9c3-19a41f76a985-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:37.920730 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.920715 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/420774ff-0dfc-44ef-a9c3-19a41f76a985-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:37.920730 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.920729 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/420774ff-0dfc-44ef-a9c3-19a41f76a985-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:37.921130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:37.920743 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/420774ff-0dfc-44ef-a9c3-19a41f76a985-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:38.121236 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:38.121197 4363 generic.go:356] "Generic (PLEG): container finished" podID="4e4adc20-5cbc-485c-aaee-acca31664813" containerID="9eef98366200810482cd73cc8152600f0fb7e132205dfcbe5c37a2413a45623e" exitCode=0 Nov 01 16:12:38.121744 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:38.121256 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-58b0d1-786cf8cdb9-cbftv" Nov 01 16:12:38.121744 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:38.121283 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-58b0d1-786cf8cdb9-cbftv" event={"ID":"4e4adc20-5cbc-485c-aaee-acca31664813","Type":"ContainerDied","Data":"9eef98366200810482cd73cc8152600f0fb7e132205dfcbe5c37a2413a45623e"} Nov 01 16:12:38.121744 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:38.121350 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-58b0d1-786cf8cdb9-cbftv" event={"ID":"4e4adc20-5cbc-485c-aaee-acca31664813","Type":"ContainerDied","Data":"996228e4a28f6c1b94ad13193f86d949c9e49a8211d06739e846ac21e2aa4841"} Nov 01 16:12:38.121744 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:38.121381 4363 scope.go:122] "RemoveContainer" containerID="9eef98366200810482cd73cc8152600f0fb7e132205dfcbe5c37a2413a45623e" Nov 01 16:12:38.127032 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:38.126998 4363 generic.go:356] "Generic (PLEG): container finished" podID="420774ff-0dfc-44ef-a9c3-19a41f76a985" containerID="681ff7838753f6e911ecadf9f3b2eb5b04772a76a9cfbff05dece650caa4a517" exitCode=0 Nov 01 16:12:38.127182 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:38.127095 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" Nov 01 16:12:38.127182 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:38.127123 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" event={"ID":"420774ff-0dfc-44ef-a9c3-19a41f76a985","Type":"ContainerDied","Data":"681ff7838753f6e911ecadf9f3b2eb5b04772a76a9cfbff05dece650caa4a517"} Nov 01 16:12:38.127182 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:38.127159 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m" event={"ID":"420774ff-0dfc-44ef-a9c3-19a41f76a985","Type":"ContainerDied","Data":"afb3d68a77c7d155a871f29ac9a0d4bbc60b3ac92f18e103f033145c2364d02a"} Nov 01 16:12:38.128746 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:38.128722 4363 generic.go:356] "Generic (PLEG): container finished" podID="c20f4fe4-124a-446f-89b5-e526085a8cea" containerID="4c12d3fb4ad3ff5de5e328b3345b40fe55c177923b08f016191b276813ab6de3" exitCode=0 Nov 01 16:12:38.128890 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:38.128785 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-58b0d1-5455d7568c-jh7m4" Nov 01 16:12:38.128890 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:38.128812 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-58b0d1-5455d7568c-jh7m4" event={"ID":"c20f4fe4-124a-446f-89b5-e526085a8cea","Type":"ContainerDied","Data":"4c12d3fb4ad3ff5de5e328b3345b40fe55c177923b08f016191b276813ab6de3"} Nov 01 16:12:38.128890 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:38.128864 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-58b0d1-5455d7568c-jh7m4" event={"ID":"c20f4fe4-124a-446f-89b5-e526085a8cea","Type":"ContainerDied","Data":"5753e2d01b4fe0f5718a2fcb32c303a78eec7886685742bda122ad2b37bd8f00"} Nov 01 16:12:38.136397 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:38.136374 4363 scope.go:122] "RemoveContainer" containerID="9eef98366200810482cd73cc8152600f0fb7e132205dfcbe5c37a2413a45623e" Nov 01 16:12:38.136692 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:12:38.136669 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9eef98366200810482cd73cc8152600f0fb7e132205dfcbe5c37a2413a45623e\": container with ID starting with 9eef98366200810482cd73cc8152600f0fb7e132205dfcbe5c37a2413a45623e not found: ID does not exist" containerID="9eef98366200810482cd73cc8152600f0fb7e132205dfcbe5c37a2413a45623e" Nov 01 16:12:38.136821 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:38.136695 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9eef98366200810482cd73cc8152600f0fb7e132205dfcbe5c37a2413a45623e"} err="failed to get container status \"9eef98366200810482cd73cc8152600f0fb7e132205dfcbe5c37a2413a45623e\": rpc error: code = NotFound desc = could not find container \"9eef98366200810482cd73cc8152600f0fb7e132205dfcbe5c37a2413a45623e\": container with ID starting with 9eef98366200810482cd73cc8152600f0fb7e132205dfcbe5c37a2413a45623e not found: ID does not exist" Nov 01 16:12:38.136821 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:38.136713 4363 scope.go:122] "RemoveContainer" containerID="681ff7838753f6e911ecadf9f3b2eb5b04772a76a9cfbff05dece650caa4a517" Nov 01 16:12:38.148925 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:38.148900 4363 scope.go:122] "RemoveContainer" containerID="071c82dff01760262ca73b5e4ec7256daf0a0d77811695e658ab57db7052b1da" Nov 01 16:12:38.149539 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:38.149520 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-58b0d1-786cf8cdb9-cbftv"] Nov 01 16:12:38.197606 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:38.197537 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-58b0d1-786cf8cdb9-cbftv"] Nov 01 16:12:38.256961 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:38.255274 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m"] Nov 01 16:12:38.257733 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:38.257708 4363 scope.go:122] "RemoveContainer" containerID="681ff7838753f6e911ecadf9f3b2eb5b04772a76a9cfbff05dece650caa4a517" Nov 01 16:12:38.258820 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:12:38.258787 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"681ff7838753f6e911ecadf9f3b2eb5b04772a76a9cfbff05dece650caa4a517\": container with ID starting with 681ff7838753f6e911ecadf9f3b2eb5b04772a76a9cfbff05dece650caa4a517 not found: ID does not exist" containerID="681ff7838753f6e911ecadf9f3b2eb5b04772a76a9cfbff05dece650caa4a517" Nov 01 16:12:38.259077 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:38.259046 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"681ff7838753f6e911ecadf9f3b2eb5b04772a76a9cfbff05dece650caa4a517"} err="failed to get container status \"681ff7838753f6e911ecadf9f3b2eb5b04772a76a9cfbff05dece650caa4a517\": rpc error: code = NotFound desc = could not find container \"681ff7838753f6e911ecadf9f3b2eb5b04772a76a9cfbff05dece650caa4a517\": container with ID starting with 681ff7838753f6e911ecadf9f3b2eb5b04772a76a9cfbff05dece650caa4a517 not found: ID does not exist" Nov 01 16:12:38.259204 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:38.259189 4363 scope.go:122] "RemoveContainer" containerID="071c82dff01760262ca73b5e4ec7256daf0a0d77811695e658ab57db7052b1da" Nov 01 16:12:38.259844 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:12:38.259704 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"071c82dff01760262ca73b5e4ec7256daf0a0d77811695e658ab57db7052b1da\": container with ID starting with 071c82dff01760262ca73b5e4ec7256daf0a0d77811695e658ab57db7052b1da not found: ID does not exist" containerID="071c82dff01760262ca73b5e4ec7256daf0a0d77811695e658ab57db7052b1da" Nov 01 16:12:38.259844 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:38.259736 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"071c82dff01760262ca73b5e4ec7256daf0a0d77811695e658ab57db7052b1da"} err="failed to get container status \"071c82dff01760262ca73b5e4ec7256daf0a0d77811695e658ab57db7052b1da\": rpc error: code = NotFound desc = could not find container \"071c82dff01760262ca73b5e4ec7256daf0a0d77811695e658ab57db7052b1da\": container with ID starting with 071c82dff01760262ca73b5e4ec7256daf0a0d77811695e658ab57db7052b1da not found: ID does not exist" Nov 01 16:12:38.259844 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:38.259759 4363 scope.go:122] "RemoveContainer" containerID="4c12d3fb4ad3ff5de5e328b3345b40fe55c177923b08f016191b276813ab6de3" Nov 01 16:12:38.317091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:38.316941 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-ffx4m"] Nov 01 16:12:38.317091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:38.317006 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-58b0d1-5455d7568c-jh7m4"] Nov 01 16:12:38.328618 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:38.327332 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-58b0d1-5455d7568c-jh7m4"] Nov 01 16:12:38.353118 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:38.353049 4363 scope.go:122] "RemoveContainer" containerID="4c12d3fb4ad3ff5de5e328b3345b40fe55c177923b08f016191b276813ab6de3" Nov 01 16:12:38.355264 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:12:38.355237 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c12d3fb4ad3ff5de5e328b3345b40fe55c177923b08f016191b276813ab6de3\": container with ID starting with 4c12d3fb4ad3ff5de5e328b3345b40fe55c177923b08f016191b276813ab6de3 not found: ID does not exist" containerID="4c12d3fb4ad3ff5de5e328b3345b40fe55c177923b08f016191b276813ab6de3" Nov 01 16:12:38.355354 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:38.355271 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c12d3fb4ad3ff5de5e328b3345b40fe55c177923b08f016191b276813ab6de3"} err="failed to get container status \"4c12d3fb4ad3ff5de5e328b3345b40fe55c177923b08f016191b276813ab6de3\": rpc error: code = NotFound desc = could not find container \"4c12d3fb4ad3ff5de5e328b3345b40fe55c177923b08f016191b276813ab6de3\": container with ID starting with 4c12d3fb4ad3ff5de5e328b3345b40fe55c177923b08f016191b276813ab6de3 not found: ID does not exist" Nov 01 16:12:38.802552 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:38.802519 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="420774ff-0dfc-44ef-a9c3-19a41f76a985" path="/var/lib/kubelet/pods/420774ff-0dfc-44ef-a9c3-19a41f76a985/volumes" Nov 01 16:12:38.803153 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:38.803132 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e4adc20-5cbc-485c-aaee-acca31664813" path="/var/lib/kubelet/pods/4e4adc20-5cbc-485c-aaee-acca31664813/volumes" Nov 01 16:12:38.803592 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:38.803577 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c20f4fe4-124a-446f-89b5-e526085a8cea" path="/var/lib/kubelet/pods/c20f4fe4-124a-446f-89b5-e526085a8cea/volumes" Nov 01 16:12:38.804393 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:38.804377 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcf26f07-340c-4852-8213-e5ab594f33b0" path="/var/lib/kubelet/pods/dcf26f07-340c-4852-8213-e5ab594f33b0/volumes" Nov 01 16:12:39.938104 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:39.938075 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04f248-668fc47648-hptb8" Nov 01 16:12:45.122405 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:45.122371 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" Nov 01 16:12:45.164859 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:45.164827 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" Nov 01 16:12:45.264646 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:45.264616 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" Nov 01 16:12:46.405871 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.405837 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn"] Nov 01 16:12:46.407716 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.407690 4363 cpu_manager.go:414] "containerMap: removing container" podUID="dcf26f07-340c-4852-8213-e5ab594f33b0" containerName="maas-api" Nov 01 16:12:46.407877 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.407862 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcf26f07-340c-4852-8213-e5ab594f33b0" containerName="maas-api" Nov 01 16:12:46.407980 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.407969 4363 cpu_manager.go:414] "containerMap: removing container" podUID="4e4adc20-5cbc-485c-aaee-acca31664813" containerName="payload-processing" Nov 01 16:12:46.408077 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.408065 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e4adc20-5cbc-485c-aaee-acca31664813" containerName="payload-processing" Nov 01 16:12:46.408188 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.408176 4363 cpu_manager.go:414] "containerMap: removing container" podUID="c20f4fe4-124a-446f-89b5-e526085a8cea" containerName="payload-pre-processing" Nov 01 16:12:46.408271 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.408257 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c20f4fe4-124a-446f-89b5-e526085a8cea" containerName="payload-pre-processing" Nov 01 16:12:46.408379 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.408369 4363 cpu_manager.go:414] "containerMap: removing container" podUID="420774ff-0dfc-44ef-a9c3-19a41f76a985" containerName="istio-proxy" Nov 01 16:12:46.408493 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.408481 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="420774ff-0dfc-44ef-a9c3-19a41f76a985" containerName="istio-proxy" Nov 01 16:12:46.408646 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.408634 4363 cpu_manager.go:414] "containerMap: removing container" podUID="420774ff-0dfc-44ef-a9c3-19a41f76a985" containerName="istio-proxy" Nov 01 16:12:46.408722 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.408711 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="420774ff-0dfc-44ef-a9c3-19a41f76a985" containerName="istio-proxy" Nov 01 16:12:46.409092 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.409075 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcf26f07-340c-4852-8213-e5ab594f33b0" containerName="maas-api" Nov 01 16:12:46.409205 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.409195 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e4adc20-5cbc-485c-aaee-acca31664813" containerName="payload-processing" Nov 01 16:12:46.409293 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.409282 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="c20f4fe4-124a-446f-89b5-e526085a8cea" containerName="payload-pre-processing" Nov 01 16:12:46.409372 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.409363 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="420774ff-0dfc-44ef-a9c3-19a41f76a985" containerName="istio-proxy" Nov 01 16:12:46.409462 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.409453 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="420774ff-0dfc-44ef-a9c3-19a41f76a985" containerName="istio-proxy" Nov 01 16:12:46.435121 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.435088 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn"] Nov 01 16:12:46.435287 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.435218 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" Nov 01 16:12:46.453424 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.453396 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-99afca-openshift-default-dockercfg-7gj25\"" Nov 01 16:12:46.513054 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.512992 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6716c2c-b539-4e3c-ac09-7f2d1426af74-workload-socket\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn\" (UID: \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" Nov 01 16:12:46.513315 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.513290 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6716c2c-b539-4e3c-ac09-7f2d1426af74-istiod-ca-cert\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn\" (UID: \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" Nov 01 16:12:46.513539 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.513513 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6716c2c-b539-4e3c-ac09-7f2d1426af74-istio-token\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn\" (UID: \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" Nov 01 16:12:46.513631 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.513575 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6716c2c-b539-4e3c-ac09-7f2d1426af74-workload-certs\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn\" (UID: \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" Nov 01 16:12:46.513631 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.513625 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6716c2c-b539-4e3c-ac09-7f2d1426af74-istio-data\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn\" (UID: \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" Nov 01 16:12:46.513770 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.513753 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6716c2c-b539-4e3c-ac09-7f2d1426af74-credential-socket\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn\" (UID: \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" Nov 01 16:12:46.513835 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.513791 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6716c2c-b539-4e3c-ac09-7f2d1426af74-istio-envoy\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn\" (UID: \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" Nov 01 16:12:46.513998 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.513977 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq5b6\" (UniqueName: \"kubernetes.io/projected/b6716c2c-b539-4e3c-ac09-7f2d1426af74-kube-api-access-sq5b6\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn\" (UID: \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" Nov 01 16:12:46.514117 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.514069 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6716c2c-b539-4e3c-ac09-7f2d1426af74-istio-podinfo\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn\" (UID: \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" Nov 01 16:12:46.623130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.616198 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6716c2c-b539-4e3c-ac09-7f2d1426af74-credential-socket\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn\" (UID: \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" Nov 01 16:12:46.623130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.616257 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6716c2c-b539-4e3c-ac09-7f2d1426af74-istio-envoy\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn\" (UID: \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" Nov 01 16:12:46.623130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.616332 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq5b6\" (UniqueName: \"kubernetes.io/projected/b6716c2c-b539-4e3c-ac09-7f2d1426af74-kube-api-access-sq5b6\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn\" (UID: \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" Nov 01 16:12:46.623130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.616399 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6716c2c-b539-4e3c-ac09-7f2d1426af74-istio-podinfo\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn\" (UID: \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" Nov 01 16:12:46.623130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.616505 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6716c2c-b539-4e3c-ac09-7f2d1426af74-workload-socket\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn\" (UID: \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" Nov 01 16:12:46.623130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.616567 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6716c2c-b539-4e3c-ac09-7f2d1426af74-istiod-ca-cert\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn\" (UID: \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" Nov 01 16:12:46.623130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.616698 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6716c2c-b539-4e3c-ac09-7f2d1426af74-istio-token\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn\" (UID: \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" Nov 01 16:12:46.623130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.616761 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6716c2c-b539-4e3c-ac09-7f2d1426af74-workload-certs\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn\" (UID: \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" Nov 01 16:12:46.623130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.616807 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6716c2c-b539-4e3c-ac09-7f2d1426af74-istio-data\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn\" (UID: \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" Nov 01 16:12:46.623130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.617360 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6716c2c-b539-4e3c-ac09-7f2d1426af74-istio-data\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn\" (UID: \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" Nov 01 16:12:46.623130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.617609 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6716c2c-b539-4e3c-ac09-7f2d1426af74-credential-socket\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn\" (UID: \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" Nov 01 16:12:46.623130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.619435 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6716c2c-b539-4e3c-ac09-7f2d1426af74-workload-socket\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn\" (UID: \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" Nov 01 16:12:46.692763 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.629985 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6716c2c-b539-4e3c-ac09-7f2d1426af74-istiod-ca-cert\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn\" (UID: \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" Nov 01 16:12:46.692763 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.630275 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6716c2c-b539-4e3c-ac09-7f2d1426af74-workload-certs\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn\" (UID: \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" Nov 01 16:12:46.726038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.717601 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6716c2c-b539-4e3c-ac09-7f2d1426af74-istio-podinfo\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn\" (UID: \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" Nov 01 16:12:46.744039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.730889 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq5b6\" (UniqueName: \"kubernetes.io/projected/b6716c2c-b539-4e3c-ac09-7f2d1426af74-kube-api-access-sq5b6\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn\" (UID: \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" Nov 01 16:12:46.744039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.731225 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6716c2c-b539-4e3c-ac09-7f2d1426af74-istio-envoy\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn\" (UID: \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" Nov 01 16:12:46.744039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.742619 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6716c2c-b539-4e3c-ac09-7f2d1426af74-istio-token\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn\" (UID: \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" Nov 01 16:12:46.768703 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:46.758713 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" Nov 01 16:12:47.263059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:47.261527 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtwhw_8254aafa-dd5b-4150-87e4-d165dab6bcb5/ovn-acl-logging/0.log" Nov 01 16:12:47.306928 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:47.306900 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn"] Nov 01 16:12:47.307752 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:47.307736 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtwhw_8254aafa-dd5b-4150-87e4-d165dab6bcb5/ovn-acl-logging/0.log" Nov 01 16:12:47.308382 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:12:47.308357 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6716c2c_b539_4e3c_ac09_7f2d1426af74.slice/crio-d76521f60f9a38e58080b8f0e0a4ddc2d8998c98c9ebe736bd903631d87a4280 WatchSource:0}: Error finding container d76521f60f9a38e58080b8f0e0a4ddc2d8998c98c9ebe736bd903631d87a4280: Status 404 returned error can't find the container with id d76521f60f9a38e58080b8f0e0a4ddc2d8998c98c9ebe736bd903631d87a4280 Nov 01 16:12:47.313180 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:47.313142 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:12:47.313262 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:47.313226 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:12:47.313299 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:47.313273 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:12:47.409177 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:47.409148 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm"] Nov 01 16:12:47.431200 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:47.429672 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" Nov 01 16:12:47.437412 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:47.437337 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm"] Nov 01 16:12:47.468616 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:47.468587 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-99e837-openshift-default-dockercfg-dmd8v\"" Nov 01 16:12:47.473787 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:47.473755 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebf773ee-489a-418d-a031-327a467839c4-workload-certs\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm\" (UID: \"ebf773ee-489a-418d-a031-327a467839c4\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" Nov 01 16:12:47.473929 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:47.473834 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebf773ee-489a-418d-a031-327a467839c4-istio-envoy\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm\" (UID: \"ebf773ee-489a-418d-a031-327a467839c4\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" Nov 01 16:12:47.474003 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:47.473978 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebf773ee-489a-418d-a031-327a467839c4-istio-data\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm\" (UID: \"ebf773ee-489a-418d-a031-327a467839c4\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" Nov 01 16:12:47.474090 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:47.474012 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebf773ee-489a-418d-a031-327a467839c4-istio-podinfo\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm\" (UID: \"ebf773ee-489a-418d-a031-327a467839c4\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" Nov 01 16:12:47.474090 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:47.474080 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebf773ee-489a-418d-a031-327a467839c4-istio-token\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm\" (UID: \"ebf773ee-489a-418d-a031-327a467839c4\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" Nov 01 16:12:47.474198 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:47.474102 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xgkk\" (UniqueName: \"kubernetes.io/projected/ebf773ee-489a-418d-a031-327a467839c4-kube-api-access-4xgkk\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm\" (UID: \"ebf773ee-489a-418d-a031-327a467839c4\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" Nov 01 16:12:47.474198 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:47.474185 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebf773ee-489a-418d-a031-327a467839c4-workload-socket\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm\" (UID: \"ebf773ee-489a-418d-a031-327a467839c4\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" Nov 01 16:12:47.474300 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:47.474200 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebf773ee-489a-418d-a031-327a467839c4-istiod-ca-cert\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm\" (UID: \"ebf773ee-489a-418d-a031-327a467839c4\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" Nov 01 16:12:47.474300 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:47.474285 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebf773ee-489a-418d-a031-327a467839c4-credential-socket\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm\" (UID: \"ebf773ee-489a-418d-a031-327a467839c4\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" Nov 01 16:12:47.579187 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:47.576354 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebf773ee-489a-418d-a031-327a467839c4-istio-data\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm\" (UID: \"ebf773ee-489a-418d-a031-327a467839c4\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" Nov 01 16:12:47.579187 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:47.576416 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebf773ee-489a-418d-a031-327a467839c4-istio-podinfo\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm\" (UID: \"ebf773ee-489a-418d-a031-327a467839c4\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" Nov 01 16:12:47.579187 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:47.576473 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebf773ee-489a-418d-a031-327a467839c4-istio-token\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm\" (UID: \"ebf773ee-489a-418d-a031-327a467839c4\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" Nov 01 16:12:47.579187 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:47.576505 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xgkk\" (UniqueName: \"kubernetes.io/projected/ebf773ee-489a-418d-a031-327a467839c4-kube-api-access-4xgkk\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm\" (UID: \"ebf773ee-489a-418d-a031-327a467839c4\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" Nov 01 16:12:47.579187 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:47.576626 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebf773ee-489a-418d-a031-327a467839c4-workload-socket\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm\" (UID: \"ebf773ee-489a-418d-a031-327a467839c4\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" Nov 01 16:12:47.579187 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:47.576654 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebf773ee-489a-418d-a031-327a467839c4-istiod-ca-cert\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm\" (UID: \"ebf773ee-489a-418d-a031-327a467839c4\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" Nov 01 16:12:47.579187 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:47.576791 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebf773ee-489a-418d-a031-327a467839c4-credential-socket\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm\" (UID: \"ebf773ee-489a-418d-a031-327a467839c4\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" Nov 01 16:12:47.579187 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:47.576913 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebf773ee-489a-418d-a031-327a467839c4-workload-certs\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm\" (UID: \"ebf773ee-489a-418d-a031-327a467839c4\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" Nov 01 16:12:47.579187 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:47.577057 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebf773ee-489a-418d-a031-327a467839c4-istio-envoy\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm\" (UID: \"ebf773ee-489a-418d-a031-327a467839c4\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" Nov 01 16:12:47.604083 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:47.591739 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebf773ee-489a-418d-a031-327a467839c4-istio-data\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm\" (UID: \"ebf773ee-489a-418d-a031-327a467839c4\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" Nov 01 16:12:47.614100 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:47.597099 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebf773ee-489a-418d-a031-327a467839c4-istiod-ca-cert\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm\" (UID: \"ebf773ee-489a-418d-a031-327a467839c4\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" Nov 01 16:12:47.614341 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:47.597347 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebf773ee-489a-418d-a031-327a467839c4-workload-socket\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm\" (UID: \"ebf773ee-489a-418d-a031-327a467839c4\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" Nov 01 16:12:47.614487 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:47.604265 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebf773ee-489a-418d-a031-327a467839c4-credential-socket\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm\" (UID: \"ebf773ee-489a-418d-a031-327a467839c4\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" Nov 01 16:12:47.623165 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:47.604522 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebf773ee-489a-418d-a031-327a467839c4-workload-certs\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm\" (UID: \"ebf773ee-489a-418d-a031-327a467839c4\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" Nov 01 16:12:47.675040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:47.648106 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xgkk\" (UniqueName: \"kubernetes.io/projected/ebf773ee-489a-418d-a031-327a467839c4-kube-api-access-4xgkk\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm\" (UID: \"ebf773ee-489a-418d-a031-327a467839c4\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" Nov 01 16:12:47.675040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:47.668573 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebf773ee-489a-418d-a031-327a467839c4-istio-token\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm\" (UID: \"ebf773ee-489a-418d-a031-327a467839c4\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" Nov 01 16:12:47.675040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:47.671657 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebf773ee-489a-418d-a031-327a467839c4-istio-podinfo\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm\" (UID: \"ebf773ee-489a-418d-a031-327a467839c4\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" Nov 01 16:12:47.685524 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:47.685295 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebf773ee-489a-418d-a031-327a467839c4-istio-envoy\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm\" (UID: \"ebf773ee-489a-418d-a031-327a467839c4\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" Nov 01 16:12:47.769042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:47.762328 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" Nov 01 16:12:48.284762 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:48.284652 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" event={"ID":"b6716c2c-b539-4e3c-ac09-7f2d1426af74","Type":"ContainerStarted","Data":"95dd1549927c9b22958db180914ee1cfa32ed7449cd317694cbacfd7ff364525"} Nov 01 16:12:48.284762 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:48.284733 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" event={"ID":"b6716c2c-b539-4e3c-ac09-7f2d1426af74","Type":"ContainerStarted","Data":"d76521f60f9a38e58080b8f0e0a4ddc2d8998c98c9ebe736bd903631d87a4280"} Nov 01 16:12:48.337951 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:48.336271 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" podStartSLOduration=2.3362570160000002 podStartE2EDuration="2.336257016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:12:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:12:48.331351389 +0000 UTC m=+1501.663089051" watchObservedRunningTime="2026-11-01 16:12:48.336257016 +0000 UTC m=+1501.667994665" Nov 01 16:12:48.345580 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:48.345537 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm"] Nov 01 16:12:48.349675 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:12:48.349560 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebf773ee_489a_418d_a031_327a467839c4.slice/crio-2e4646bba90843876574a739b699881cc909d3d43547a9aa25992e92b9ff8dc0 WatchSource:0}: Error finding container 2e4646bba90843876574a739b699881cc909d3d43547a9aa25992e92b9ff8dc0: Status 404 returned error can't find the container with id 2e4646bba90843876574a739b699881cc909d3d43547a9aa25992e92b9ff8dc0 Nov 01 16:12:48.351225 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:12:48.351191 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:12:48.353414 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:48.353385 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:12:48.353508 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:48.353452 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:12:48.353508 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:48.353480 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:12:48.761590 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:48.761552 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" Nov 01 16:12:48.767280 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:48.767256 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" Nov 01 16:12:49.365443 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:49.365408 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" event={"ID":"ebf773ee-489a-418d-a031-327a467839c4","Type":"ContainerStarted","Data":"1fb51714c82041bd241aadebc9f1707c9825ad779c38ebfec5ffb437a84e719c"} Nov 01 16:12:49.365724 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:49.365701 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" event={"ID":"ebf773ee-489a-418d-a031-327a467839c4","Type":"ContainerStarted","Data":"2e4646bba90843876574a739b699881cc909d3d43547a9aa25992e92b9ff8dc0"} Nov 01 16:12:49.365864 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:49.365850 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" Nov 01 16:12:49.378282 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:49.377750 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" Nov 01 16:12:49.402913 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:49.402871 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" podStartSLOduration=2.402856154 podStartE2EDuration="2.402856154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:12:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:12:49.394554452 +0000 UTC m=+1502.726292102" watchObservedRunningTime="2026-11-01 16:12:49.402856154 +0000 UTC m=+1502.734593804" Nov 01 16:12:49.762658 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:49.762563 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" Nov 01 16:12:49.770329 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:49.770298 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" Nov 01 16:12:50.371957 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:50.371923 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" Nov 01 16:12:50.372803 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:50.372779 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" Nov 01 16:12:53.054786 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:53.054750 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-99afca-6df5cd4d75-kqpvz"] Nov 01 16:12:53.065285 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:53.065256 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99afca-6df5cd4d75-kqpvz" Nov 01 16:12:53.078125 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:53.078088 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-99afca-6df5cd4d75-kqpvz"] Nov 01 16:12:53.094115 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:53.094038 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-99afca\"" Nov 01 16:12:53.094969 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:53.094941 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-99afca-5bc4ccf5cc-qs8gg"] Nov 01 16:12:53.110561 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:53.110533 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-99afca-5bc4ccf5cc-qs8gg"] Nov 01 16:12:53.110695 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:53.110645 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-99afca-5bc4ccf5cc-qs8gg" Nov 01 16:12:53.114719 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:53.114695 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-99afca-dockercfg-2qpk7\"" Nov 01 16:12:53.224050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:53.223471 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd9e45d-cd61-4a97-9d05-839b5cb4698b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-99afca-6df5cd4d75-kqpvz\" (UID: \"cdd9e45d-cd61-4a97-9d05-839b5cb4698b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99afca-6df5cd4d75-kqpvz" Nov 01 16:12:53.224050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:53.223572 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz4rw\" (UniqueName: \"kubernetes.io/projected/15978381-170d-44fd-bfe3-725b8e60060d-kube-api-access-kz4rw\") pod \"payload-processing-e2e-inf-b-99afca-5bc4ccf5cc-qs8gg\" (UID: \"15978381-170d-44fd-bfe3-725b8e60060d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-99afca-5bc4ccf5cc-qs8gg" Nov 01 16:12:53.224050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:53.223610 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5lcf\" (UniqueName: \"kubernetes.io/projected/cdd9e45d-cd61-4a97-9d05-839b5cb4698b-kube-api-access-g5lcf\") pod \"payload-pre-processing-e2e-inf-b-99afca-6df5cd4d75-kqpvz\" (UID: \"cdd9e45d-cd61-4a97-9d05-839b5cb4698b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99afca-6df5cd4d75-kqpvz" Nov 01 16:12:53.224050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:53.223778 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15978381-170d-44fd-bfe3-725b8e60060d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-99afca-5bc4ccf5cc-qs8gg\" (UID: \"15978381-170d-44fd-bfe3-725b8e60060d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-99afca-5bc4ccf5cc-qs8gg" Nov 01 16:12:53.342094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:53.339408 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd9e45d-cd61-4a97-9d05-839b5cb4698b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-99afca-6df5cd4d75-kqpvz\" (UID: \"cdd9e45d-cd61-4a97-9d05-839b5cb4698b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99afca-6df5cd4d75-kqpvz" Nov 01 16:12:53.342094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:53.339595 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz4rw\" (UniqueName: \"kubernetes.io/projected/15978381-170d-44fd-bfe3-725b8e60060d-kube-api-access-kz4rw\") pod \"payload-processing-e2e-inf-b-99afca-5bc4ccf5cc-qs8gg\" (UID: \"15978381-170d-44fd-bfe3-725b8e60060d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-99afca-5bc4ccf5cc-qs8gg" Nov 01 16:12:53.342094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:53.339658 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5lcf\" (UniqueName: \"kubernetes.io/projected/cdd9e45d-cd61-4a97-9d05-839b5cb4698b-kube-api-access-g5lcf\") pod \"payload-pre-processing-e2e-inf-b-99afca-6df5cd4d75-kqpvz\" (UID: \"cdd9e45d-cd61-4a97-9d05-839b5cb4698b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99afca-6df5cd4d75-kqpvz" Nov 01 16:12:53.342094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:53.339881 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15978381-170d-44fd-bfe3-725b8e60060d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-99afca-5bc4ccf5cc-qs8gg\" (UID: \"15978381-170d-44fd-bfe3-725b8e60060d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-99afca-5bc4ccf5cc-qs8gg" Nov 01 16:12:53.350040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:53.347391 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15978381-170d-44fd-bfe3-725b8e60060d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-99afca-5bc4ccf5cc-qs8gg\" (UID: \"15978381-170d-44fd-bfe3-725b8e60060d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-99afca-5bc4ccf5cc-qs8gg" Nov 01 16:12:53.350040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:53.348185 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd9e45d-cd61-4a97-9d05-839b5cb4698b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-99afca-6df5cd4d75-kqpvz\" (UID: \"cdd9e45d-cd61-4a97-9d05-839b5cb4698b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99afca-6df5cd4d75-kqpvz" Nov 01 16:12:53.378037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:53.374675 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz4rw\" (UniqueName: \"kubernetes.io/projected/15978381-170d-44fd-bfe3-725b8e60060d-kube-api-access-kz4rw\") pod \"payload-processing-e2e-inf-b-99afca-5bc4ccf5cc-qs8gg\" (UID: \"15978381-170d-44fd-bfe3-725b8e60060d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-99afca-5bc4ccf5cc-qs8gg" Nov 01 16:12:53.397615 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:53.383669 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5lcf\" (UniqueName: \"kubernetes.io/projected/cdd9e45d-cd61-4a97-9d05-839b5cb4698b-kube-api-access-g5lcf\") pod \"payload-pre-processing-e2e-inf-b-99afca-6df5cd4d75-kqpvz\" (UID: \"cdd9e45d-cd61-4a97-9d05-839b5cb4698b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99afca-6df5cd4d75-kqpvz" Nov 01 16:12:53.448055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:53.425319 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99afca-6df5cd4d75-kqpvz" Nov 01 16:12:53.448055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:53.434897 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-99afca-5bc4ccf5cc-qs8gg" Nov 01 16:12:54.119044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:54.115070 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-99afca-5bc4ccf5cc-qs8gg"] Nov 01 16:12:54.268586 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:54.222233 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-99afca-6df5cd4d75-kqpvz"] Nov 01 16:12:54.641712 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:54.641603 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-99afca-5bc4ccf5cc-qs8gg" event={"ID":"15978381-170d-44fd-bfe3-725b8e60060d","Type":"ContainerStarted","Data":"b02783d7c63304189e747bc4fe3c899f80c592dd857f7209b748bfb4f317f784"} Nov 01 16:12:54.641873 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:54.641837 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-99afca-5bc4ccf5cc-qs8gg" Nov 01 16:12:54.641873 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:54.641860 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-99afca-5bc4ccf5cc-qs8gg" event={"ID":"15978381-170d-44fd-bfe3-725b8e60060d","Type":"ContainerStarted","Data":"8508a05f7958c7bf0bba748eaf96c5fb543807abc53260bed444a30fe6b15fec"} Nov 01 16:12:54.643473 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:54.643442 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99afca-6df5cd4d75-kqpvz" event={"ID":"cdd9e45d-cd61-4a97-9d05-839b5cb4698b","Type":"ContainerStarted","Data":"69f66ffc632eed9f5f72736c3230bb932dcc7cb07b8adacc8b51062186bb88b8"} Nov 01 16:12:54.643586 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:54.643490 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99afca-6df5cd4d75-kqpvz" event={"ID":"cdd9e45d-cd61-4a97-9d05-839b5cb4698b","Type":"ContainerStarted","Data":"54e9dd2a38df97f8220cb3ec685cf4b919e0fe5d38ed18d9371155e2af3d308f"} Nov 01 16:12:54.643586 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:54.643546 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99afca-6df5cd4d75-kqpvz" Nov 01 16:12:54.662101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:54.662054 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-99afca-5bc4ccf5cc-qs8gg" podStartSLOduration=1.662039884 podStartE2EDuration="1.662039884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:12:54.656675228 +0000 UTC m=+1507.988412879" watchObservedRunningTime="2026-11-01 16:12:54.662039884 +0000 UTC m=+1507.993777530" Nov 01 16:12:54.673966 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:54.673917 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99afca-6df5cd4d75-kqpvz" podStartSLOduration=1.673904314 podStartE2EDuration="1.673904314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:12:54.670965767 +0000 UTC m=+1508.002703417" watchObservedRunningTime="2026-11-01 16:12:54.673904314 +0000 UTC m=+1508.005641963" Nov 01 16:12:55.723257 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:55.723224 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-26310f-58f7f46c57-hqjhw"] Nov 01 16:12:55.759859 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:12:55.759828 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-26310f-778676cc8c-ppxzh"] Nov 01 16:13:00.731044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:00.725704 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-26310f-58f7f46c57-hqjhw" podUID="8227cf4e-1b88-4ba8-914d-30c14e972787" containerName="payload-processing" containerID="cri-o://a016506f764477f351c441bba45c516ce1bb8cff9a01fc7b0dd7fc506838ca07" gracePeriod=25 Nov 01 16:13:00.774048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:00.763678 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-26310f-778676cc8c-ppxzh" podUID="b398ad44-84ce-42af-98e9-4f097895b3fd" containerName="payload-pre-processing" containerID="cri-o://e1e8d1be81a6ec3f3eb2c3eb29d6042c8177b8e94e2b7bae8084fcb3438941f5" gracePeriod=25 Nov 01 16:13:01.031673 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:01.031582 4363 generic.go:356] "Generic (PLEG): container finished" podID="b398ad44-84ce-42af-98e9-4f097895b3fd" containerID="e1e8d1be81a6ec3f3eb2c3eb29d6042c8177b8e94e2b7bae8084fcb3438941f5" exitCode=0 Nov 01 16:13:01.031845 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:01.031664 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-26310f-778676cc8c-ppxzh" event={"ID":"b398ad44-84ce-42af-98e9-4f097895b3fd","Type":"ContainerDied","Data":"e1e8d1be81a6ec3f3eb2c3eb29d6042c8177b8e94e2b7bae8084fcb3438941f5"} Nov 01 16:13:01.033541 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:01.033516 4363 generic.go:356] "Generic (PLEG): container finished" podID="8227cf4e-1b88-4ba8-914d-30c14e972787" containerID="a016506f764477f351c441bba45c516ce1bb8cff9a01fc7b0dd7fc506838ca07" exitCode=0 Nov 01 16:13:01.033688 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:01.033560 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-26310f-58f7f46c57-hqjhw" event={"ID":"8227cf4e-1b88-4ba8-914d-30c14e972787","Type":"ContainerDied","Data":"a016506f764477f351c441bba45c516ce1bb8cff9a01fc7b0dd7fc506838ca07"} Nov 01 16:13:01.439556 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:01.439529 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-26310f-778676cc8c-ppxzh" Nov 01 16:13:01.739079 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:01.736446 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b398ad44-84ce-42af-98e9-4f097895b3fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b398ad44-84ce-42af-98e9-4f097895b3fd-plugins-config-volume\") pod \"b398ad44-84ce-42af-98e9-4f097895b3fd\" (UID: \"b398ad44-84ce-42af-98e9-4f097895b3fd\") " Nov 01 16:13:01.739524 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:01.739235 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b398ad44-84ce-42af-98e9-4f097895b3fd-kube-api-access-9fwdt\" (UniqueName: \"kubernetes.io/projected/b398ad44-84ce-42af-98e9-4f097895b3fd-kube-api-access-9fwdt\") pod \"b398ad44-84ce-42af-98e9-4f097895b3fd\" (UID: \"b398ad44-84ce-42af-98e9-4f097895b3fd\") " Nov 01 16:13:01.740276 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:01.740235 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b398ad44-84ce-42af-98e9-4f097895b3fd-plugins-config-volume" pod "b398ad44-84ce-42af-98e9-4f097895b3fd" (UID: "b398ad44-84ce-42af-98e9-4f097895b3fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:13:01.744375 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:01.743620 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b398ad44-84ce-42af-98e9-4f097895b3fd-kube-api-access-9fwdt" pod "b398ad44-84ce-42af-98e9-4f097895b3fd" (UID: "b398ad44-84ce-42af-98e9-4f097895b3fd"). InnerVolumeSpecName "kube-api-access-9fwdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:13:01.744942 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:01.744849 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fwdt\" (UniqueName: \"kubernetes.io/projected/b398ad44-84ce-42af-98e9-4f097895b3fd-kube-api-access-9fwdt\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:01.744942 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:01.744875 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b398ad44-84ce-42af-98e9-4f097895b3fd-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:01.752504 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:01.752476 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-26310f-58f7f46c57-hqjhw" Nov 01 16:13:01.845715 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:01.845682 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8227cf4e-1b88-4ba8-914d-30c14e972787-kube-api-access-8krqj\" (UniqueName: \"kubernetes.io/projected/8227cf4e-1b88-4ba8-914d-30c14e972787-kube-api-access-8krqj\") pod \"8227cf4e-1b88-4ba8-914d-30c14e972787\" (UID: \"8227cf4e-1b88-4ba8-914d-30c14e972787\") " Nov 01 16:13:01.846144 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:01.846130 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8227cf4e-1b88-4ba8-914d-30c14e972787-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8227cf4e-1b88-4ba8-914d-30c14e972787-plugins-config-volume\") pod \"8227cf4e-1b88-4ba8-914d-30c14e972787\" (UID: \"8227cf4e-1b88-4ba8-914d-30c14e972787\") " Nov 01 16:13:01.846493 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:01.846470 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8227cf4e-1b88-4ba8-914d-30c14e972787-plugins-config-volume" pod "8227cf4e-1b88-4ba8-914d-30c14e972787" (UID: "8227cf4e-1b88-4ba8-914d-30c14e972787"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:13:01.847252 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:01.847225 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8227cf4e-1b88-4ba8-914d-30c14e972787-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:01.848787 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:01.848761 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8227cf4e-1b88-4ba8-914d-30c14e972787-kube-api-access-8krqj" pod "8227cf4e-1b88-4ba8-914d-30c14e972787" (UID: "8227cf4e-1b88-4ba8-914d-30c14e972787"). InnerVolumeSpecName "kube-api-access-8krqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:13:01.948771 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:01.948692 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8krqj\" (UniqueName: \"kubernetes.io/projected/8227cf4e-1b88-4ba8-914d-30c14e972787-kube-api-access-8krqj\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:02.043155 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:02.043118 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-26310f-778676cc8c-ppxzh" Nov 01 16:13:02.043155 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:02.043140 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-26310f-778676cc8c-ppxzh" event={"ID":"b398ad44-84ce-42af-98e9-4f097895b3fd","Type":"ContainerDied","Data":"867692a46e3ad1a3bdc088e6fd696b02aaccd7036737854b57d6580566db9da2"} Nov 01 16:13:02.043367 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:02.043199 4363 scope.go:122] "RemoveContainer" containerID="e1e8d1be81a6ec3f3eb2c3eb29d6042c8177b8e94e2b7bae8084fcb3438941f5" Nov 01 16:13:02.044908 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:02.044888 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-26310f-58f7f46c57-hqjhw" Nov 01 16:13:02.045052 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:02.044913 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-26310f-58f7f46c57-hqjhw" event={"ID":"8227cf4e-1b88-4ba8-914d-30c14e972787","Type":"ContainerDied","Data":"99c3e76070927f5613d92304211c7d9597d43dfd3b3079061304148cf9c7a728"} Nov 01 16:13:02.054883 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:02.054864 4363 scope.go:122] "RemoveContainer" containerID="a016506f764477f351c441bba45c516ce1bb8cff9a01fc7b0dd7fc506838ca07" Nov 01 16:13:02.102069 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:02.100069 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-26310f-778676cc8c-ppxzh"] Nov 01 16:13:02.127400 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:02.127367 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-26310f-778676cc8c-ppxzh"] Nov 01 16:13:02.133570 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:02.133538 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-26310f-58f7f46c57-hqjhw"] Nov 01 16:13:02.138073 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:02.138034 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-26310f-58f7f46c57-hqjhw"] Nov 01 16:13:02.806867 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:02.806832 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8227cf4e-1b88-4ba8-914d-30c14e972787" path="/var/lib/kubelet/pods/8227cf4e-1b88-4ba8-914d-30c14e972787/volumes" Nov 01 16:13:02.807556 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:02.807537 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b398ad44-84ce-42af-98e9-4f097895b3fd" path="/var/lib/kubelet/pods/b398ad44-84ce-42af-98e9-4f097895b3fd/volumes" Nov 01 16:13:04.624291 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:04.621192 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-10b00a-5cdcfd8b5f-fwz5p"] Nov 01 16:13:04.624291 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:04.622690 4363 cpu_manager.go:414] "containerMap: removing container" podUID="b398ad44-84ce-42af-98e9-4f097895b3fd" containerName="payload-pre-processing" Nov 01 16:13:04.624291 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:04.622708 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b398ad44-84ce-42af-98e9-4f097895b3fd" containerName="payload-pre-processing" Nov 01 16:13:04.624291 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:04.622745 4363 cpu_manager.go:414] "containerMap: removing container" podUID="8227cf4e-1b88-4ba8-914d-30c14e972787" containerName="payload-processing" Nov 01 16:13:04.624291 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:04.622753 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8227cf4e-1b88-4ba8-914d-30c14e972787" containerName="payload-processing" Nov 01 16:13:04.624291 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:04.622976 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="b398ad44-84ce-42af-98e9-4f097895b3fd" containerName="payload-pre-processing" Nov 01 16:13:04.624291 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:04.622994 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="8227cf4e-1b88-4ba8-914d-30c14e972787" containerName="payload-processing" Nov 01 16:13:04.635039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:04.632587 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-10b00a-5cdcfd8b5f-fwz5p" Nov 01 16:13:04.643109 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:04.639295 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-10b00a-659db85879-m4pxv"] Nov 01 16:13:04.646174 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:04.646142 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-10b00a-5cdcfd8b5f-fwz5p"] Nov 01 16:13:04.646330 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:04.646196 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-10b00a-659db85879-m4pxv" Nov 01 16:13:04.647347 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:04.646706 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-10b00a\"" Nov 01 16:13:04.650217 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:04.649303 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-10b00a-dockercfg-khjvt\"" Nov 01 16:13:04.654264 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:04.654232 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-10b00a-659db85879-m4pxv"] Nov 01 16:13:04.718270 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:04.717571 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b39da192-a1b1-43f9-b6c4-b2186df9063f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-10b00a-659db85879-m4pxv\" (UID: \"b39da192-a1b1-43f9-b6c4-b2186df9063f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-10b00a-659db85879-m4pxv" Nov 01 16:13:04.718270 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:04.717658 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkthr\" (UniqueName: \"kubernetes.io/projected/b39da192-a1b1-43f9-b6c4-b2186df9063f-kube-api-access-kkthr\") pod \"payload-processing-e2e-inf-b-10b00a-659db85879-m4pxv\" (UID: \"b39da192-a1b1-43f9-b6c4-b2186df9063f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-10b00a-659db85879-m4pxv" Nov 01 16:13:04.718270 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:04.717964 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3007207-2dac-46d0-970c-8138373dc953-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-10b00a-5cdcfd8b5f-fwz5p\" (UID: \"d3007207-2dac-46d0-970c-8138373dc953\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-10b00a-5cdcfd8b5f-fwz5p" Nov 01 16:13:04.718270 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:04.718001 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc8gr\" (UniqueName: \"kubernetes.io/projected/d3007207-2dac-46d0-970c-8138373dc953-kube-api-access-qc8gr\") pod \"payload-pre-processing-e2e-inf-b-10b00a-5cdcfd8b5f-fwz5p\" (UID: \"d3007207-2dac-46d0-970c-8138373dc953\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-10b00a-5cdcfd8b5f-fwz5p" Nov 01 16:13:04.837042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:04.823347 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3007207-2dac-46d0-970c-8138373dc953-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-10b00a-5cdcfd8b5f-fwz5p\" (UID: \"d3007207-2dac-46d0-970c-8138373dc953\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-10b00a-5cdcfd8b5f-fwz5p" Nov 01 16:13:04.842617 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:04.842587 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc8gr\" (UniqueName: \"kubernetes.io/projected/d3007207-2dac-46d0-970c-8138373dc953-kube-api-access-qc8gr\") pod \"payload-pre-processing-e2e-inf-b-10b00a-5cdcfd8b5f-fwz5p\" (UID: \"d3007207-2dac-46d0-970c-8138373dc953\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-10b00a-5cdcfd8b5f-fwz5p" Nov 01 16:13:04.843105 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:04.842314 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3007207-2dac-46d0-970c-8138373dc953-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-10b00a-5cdcfd8b5f-fwz5p\" (UID: \"d3007207-2dac-46d0-970c-8138373dc953\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-10b00a-5cdcfd8b5f-fwz5p" Nov 01 16:13:04.843361 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:04.843339 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b39da192-a1b1-43f9-b6c4-b2186df9063f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-10b00a-659db85879-m4pxv\" (UID: \"b39da192-a1b1-43f9-b6c4-b2186df9063f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-10b00a-659db85879-m4pxv" Nov 01 16:13:04.843559 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:04.843541 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkthr\" (UniqueName: \"kubernetes.io/projected/b39da192-a1b1-43f9-b6c4-b2186df9063f-kube-api-access-kkthr\") pod \"payload-processing-e2e-inf-b-10b00a-659db85879-m4pxv\" (UID: \"b39da192-a1b1-43f9-b6c4-b2186df9063f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-10b00a-659db85879-m4pxv" Nov 01 16:13:04.845234 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:04.845211 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b39da192-a1b1-43f9-b6c4-b2186df9063f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-10b00a-659db85879-m4pxv\" (UID: \"b39da192-a1b1-43f9-b6c4-b2186df9063f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-10b00a-659db85879-m4pxv" Nov 01 16:13:05.039399 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:05.016727 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc8gr\" (UniqueName: \"kubernetes.io/projected/d3007207-2dac-46d0-970c-8138373dc953-kube-api-access-qc8gr\") pod \"payload-pre-processing-e2e-inf-b-10b00a-5cdcfd8b5f-fwz5p\" (UID: \"d3007207-2dac-46d0-970c-8138373dc953\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-10b00a-5cdcfd8b5f-fwz5p" Nov 01 16:13:05.113841 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:05.112876 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkthr\" (UniqueName: \"kubernetes.io/projected/b39da192-a1b1-43f9-b6c4-b2186df9063f-kube-api-access-kkthr\") pod \"payload-processing-e2e-inf-b-10b00a-659db85879-m4pxv\" (UID: \"b39da192-a1b1-43f9-b6c4-b2186df9063f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-10b00a-659db85879-m4pxv" Nov 01 16:13:05.274362 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:05.274332 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-10b00a-5cdcfd8b5f-fwz5p" Nov 01 16:13:05.303229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:05.295231 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-10b00a-659db85879-m4pxv" Nov 01 16:13:05.770786 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:13:05.770757 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3007207_2dac_46d0_970c_8138373dc953.slice/crio-c1d9a525b387441a6b0d7a7b05822c827960fc85cd122af6a2bf57a98e065922 WatchSource:0}: Error finding container c1d9a525b387441a6b0d7a7b05822c827960fc85cd122af6a2bf57a98e065922: Status 404 returned error can't find the container with id c1d9a525b387441a6b0d7a7b05822c827960fc85cd122af6a2bf57a98e065922 Nov 01 16:13:05.772232 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:13:05.772207 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb39da192_a1b1_43f9_b6c4_b2186df9063f.slice/crio-0c35554e3be4974c58abcc91938a608e7a23c88f6a3593e4fbfb94a070b4421d WatchSource:0}: Error finding container 0c35554e3be4974c58abcc91938a608e7a23c88f6a3593e4fbfb94a070b4421d: Status 404 returned error can't find the container with id 0c35554e3be4974c58abcc91938a608e7a23c88f6a3593e4fbfb94a070b4421d Nov 01 16:13:05.772743 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:05.772721 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-10b00a-5cdcfd8b5f-fwz5p"] Nov 01 16:13:05.782442 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:05.781000 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-10b00a-659db85879-m4pxv"] Nov 01 16:13:05.801594 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:05.801558 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-99afca-5bc4ccf5cc-qs8gg" Nov 01 16:13:05.802711 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:05.802687 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99afca-6df5cd4d75-kqpvz" Nov 01 16:13:06.024220 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:06.024169 4363 patch_prober.go:29] interesting pod/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:13:06.024432 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:06.024246 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" podUID="2d1193ea-2099-4ce6-b417-82d1e8e051b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:06.148742 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:06.148176 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-10b00a-659db85879-m4pxv" event={"ID":"b39da192-a1b1-43f9-b6c4-b2186df9063f","Type":"ContainerStarted","Data":"0a76156d8f3310f0e93ae9fe38ea98cfd8af56c01af6b4467c011d68eae2f380"} Nov 01 16:13:06.148742 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:06.148235 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-10b00a-659db85879-m4pxv" event={"ID":"b39da192-a1b1-43f9-b6c4-b2186df9063f","Type":"ContainerStarted","Data":"0c35554e3be4974c58abcc91938a608e7a23c88f6a3593e4fbfb94a070b4421d"} Nov 01 16:13:06.149047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:06.149004 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-10b00a-659db85879-m4pxv" Nov 01 16:13:06.155910 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:06.151763 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-10b00a-5cdcfd8b5f-fwz5p" event={"ID":"d3007207-2dac-46d0-970c-8138373dc953","Type":"ContainerStarted","Data":"58f84855ecde8d855316da17f8bb0bf0bac8979a691786fd023e1680f930dbda"} Nov 01 16:13:06.155910 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:06.151819 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-10b00a-5cdcfd8b5f-fwz5p" event={"ID":"d3007207-2dac-46d0-970c-8138373dc953","Type":"ContainerStarted","Data":"c1d9a525b387441a6b0d7a7b05822c827960fc85cd122af6a2bf57a98e065922"} Nov 01 16:13:06.155910 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:06.152729 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-10b00a-5cdcfd8b5f-fwz5p" Nov 01 16:13:06.172997 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:06.172951 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-10b00a-659db85879-m4pxv" podStartSLOduration=2.172935548 podStartE2EDuration="2.172935548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:13:06.166106514 +0000 UTC m=+1519.497844164" watchObservedRunningTime="2026-11-01 16:13:06.172935548 +0000 UTC m=+1519.504673199" Nov 01 16:13:06.195907 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:06.191953 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-10b00a-5cdcfd8b5f-fwz5p" podStartSLOduration=2.19193237 podStartE2EDuration="2.19193237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:13:06.186633698 +0000 UTC m=+1519.518371349" watchObservedRunningTime="2026-11-01 16:13:06.19193237 +0000 UTC m=+1519.523670027" Nov 01 16:13:08.171233 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:08.171152 4363 generic.go:356] "Generic (PLEG): container finished" podID="2d1193ea-2099-4ce6-b417-82d1e8e051b8" containerID="f2ce23c7cc7f14058bb2d2a272324caa25278b384841729f423191a077b7e365" exitCode=0 Nov 01 16:13:08.171642 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:08.171227 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" event={"ID":"2d1193ea-2099-4ce6-b417-82d1e8e051b8","Type":"ContainerDied","Data":"f2ce23c7cc7f14058bb2d2a272324caa25278b384841729f423191a077b7e365"} Nov 01 16:13:08.171849 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:08.171824 4363 scope.go:122] "RemoveContainer" containerID="f2ce23c7cc7f14058bb2d2a272324caa25278b384841729f423191a077b7e365" Nov 01 16:13:08.173625 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:08.173591 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:13:08.173764 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:08.173674 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:13:08.173764 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:08.173741 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:13:09.213078 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:09.198759 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" event={"ID":"2d1193ea-2099-4ce6-b417-82d1e8e051b8","Type":"ContainerStarted","Data":"7fd2600eea08863a59ea3299fe596b1fe3fea4ebf8abf813d7685d22a2a8192a"} Nov 01 16:13:10.040609 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:10.040521 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" Nov 01 16:13:10.040609 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:10.040552 4363 patch_prober.go:29] interesting pod/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 01 16:13:10.040609 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:10.040592 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" podUID="2d1193ea-2099-4ce6-b417-82d1e8e051b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 01 16:13:11.041343 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:11.041308 4363 patch_prober.go:29] interesting pod/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 01 16:13:11.041730 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:11.041363 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" podUID="2d1193ea-2099-4ce6-b417-82d1e8e051b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 01 16:13:12.040920 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:12.040886 4363 patch_prober.go:29] interesting pod/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 01 16:13:12.041145 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:12.040931 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" podUID="2d1193ea-2099-4ce6-b417-82d1e8e051b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 01 16:13:13.097516 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:13.097474 4363 patch_prober.go:29] interesting pod/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 01 16:13:13.108268 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:13.098562 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" podUID="2d1193ea-2099-4ce6-b417-82d1e8e051b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 01 16:13:13.886086 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:13.864739 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-58b0d1-799d79584b-5ckb9"] Nov 01 16:13:14.060038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:14.058517 4363 patch_prober.go:29] interesting pod/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 01 16:13:14.060038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:14.058592 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" podUID="2d1193ea-2099-4ce6-b417-82d1e8e051b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 01 16:13:14.080330 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:14.080246 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58b0d1-799d79584b-5ckb9" podUID="237a3f5e-83c1-43e0-a6f8-01dcfca57d32" containerName="maas-api" containerID="cri-o://d55bfed49e9f73ba47012de2e9905e19852d56c634527b15f756a2fafd5a5a7c" gracePeriod=30 Nov 01 16:13:14.886308 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:14.867849 4363 generic.go:356] "Generic (PLEG): container finished" podID="237a3f5e-83c1-43e0-a6f8-01dcfca57d32" containerID="d55bfed49e9f73ba47012de2e9905e19852d56c634527b15f756a2fafd5a5a7c" exitCode=0 Nov 01 16:13:15.045038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:15.044061 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58b0d1-799d79584b-5ckb9" event={"ID":"237a3f5e-83c1-43e0-a6f8-01dcfca57d32","Type":"ContainerDied","Data":"d55bfed49e9f73ba47012de2e9905e19852d56c634527b15f756a2fafd5a5a7c"} Nov 01 16:13:15.092208 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:15.071806 4363 patch_prober.go:29] interesting pod/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 01 16:13:15.092208 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:15.071910 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" podUID="2d1193ea-2099-4ce6-b417-82d1e8e051b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 01 16:13:15.778586 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:15.778507 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58b0d1-799d79584b-5ckb9" Nov 01 16:13:15.794873 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:15.794841 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/237a3f5e-83c1-43e0-a6f8-01dcfca57d32-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/237a3f5e-83c1-43e0-a6f8-01dcfca57d32-openshift-service-ca\") pod \"237a3f5e-83c1-43e0-a6f8-01dcfca57d32\" (UID: \"237a3f5e-83c1-43e0-a6f8-01dcfca57d32\") " Nov 01 16:13:15.795071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:15.794892 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/237a3f5e-83c1-43e0-a6f8-01dcfca57d32-kube-api-access-n9mp8\" (UniqueName: \"kubernetes.io/projected/237a3f5e-83c1-43e0-a6f8-01dcfca57d32-kube-api-access-n9mp8\") pod \"237a3f5e-83c1-43e0-a6f8-01dcfca57d32\" (UID: \"237a3f5e-83c1-43e0-a6f8-01dcfca57d32\") " Nov 01 16:13:15.795071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:15.794928 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/237a3f5e-83c1-43e0-a6f8-01dcfca57d32-metrics-certs\" (UniqueName: \"kubernetes.io/secret/237a3f5e-83c1-43e0-a6f8-01dcfca57d32-metrics-certs\") pod \"237a3f5e-83c1-43e0-a6f8-01dcfca57d32\" (UID: \"237a3f5e-83c1-43e0-a6f8-01dcfca57d32\") " Nov 01 16:13:15.795071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:15.794970 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/237a3f5e-83c1-43e0-a6f8-01dcfca57d32-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/237a3f5e-83c1-43e0-a6f8-01dcfca57d32-maas-api-tls\") pod \"237a3f5e-83c1-43e0-a6f8-01dcfca57d32\" (UID: \"237a3f5e-83c1-43e0-a6f8-01dcfca57d32\") " Nov 01 16:13:15.795897 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:15.795870 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/237a3f5e-83c1-43e0-a6f8-01dcfca57d32-openshift-service-ca" pod "237a3f5e-83c1-43e0-a6f8-01dcfca57d32" (UID: "237a3f5e-83c1-43e0-a6f8-01dcfca57d32"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:13:15.801982 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:15.799940 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/237a3f5e-83c1-43e0-a6f8-01dcfca57d32-maas-api-tls" pod "237a3f5e-83c1-43e0-a6f8-01dcfca57d32" (UID: "237a3f5e-83c1-43e0-a6f8-01dcfca57d32"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:13:15.817389 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:15.817349 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/237a3f5e-83c1-43e0-a6f8-01dcfca57d32-kube-api-access-n9mp8" pod "237a3f5e-83c1-43e0-a6f8-01dcfca57d32" (UID: "237a3f5e-83c1-43e0-a6f8-01dcfca57d32"). InnerVolumeSpecName "kube-api-access-n9mp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:13:15.817813 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:15.817786 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/237a3f5e-83c1-43e0-a6f8-01dcfca57d32-metrics-certs" pod "237a3f5e-83c1-43e0-a6f8-01dcfca57d32" (UID: "237a3f5e-83c1-43e0-a6f8-01dcfca57d32"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:13:15.884299 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:15.884265 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58b0d1-799d79584b-5ckb9" Nov 01 16:13:15.884551 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:15.884335 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-58b0d1-799d79584b-5ckb9" event={"ID":"237a3f5e-83c1-43e0-a6f8-01dcfca57d32","Type":"ContainerDied","Data":"3da9b5044a021a2fde1a0cc20a791bbff06cbf1d64029cb1f58e3b43066785a3"} Nov 01 16:13:15.884551 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:15.884396 4363 scope.go:122] "RemoveContainer" containerID="d55bfed49e9f73ba47012de2e9905e19852d56c634527b15f756a2fafd5a5a7c" Nov 01 16:13:15.901678 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:15.901465 4363 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/237a3f5e-83c1-43e0-a6f8-01dcfca57d32-openshift-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:15.901678 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:15.901487 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9mp8\" (UniqueName: \"kubernetes.io/projected/237a3f5e-83c1-43e0-a6f8-01dcfca57d32-kube-api-access-n9mp8\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:15.901678 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:15.901497 4363 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/237a3f5e-83c1-43e0-a6f8-01dcfca57d32-metrics-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:15.901678 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:15.901507 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/237a3f5e-83c1-43e0-a6f8-01dcfca57d32-maas-api-tls\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:16.003043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:16.001502 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-58b0d1-799d79584b-5ckb9"] Nov 01 16:13:16.062091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:16.055345 4363 patch_prober.go:29] interesting pod/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 01 16:13:16.062091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:16.055411 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" podUID="2d1193ea-2099-4ce6-b417-82d1e8e051b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 01 16:13:16.092780 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:16.092742 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-58b0d1-799d79584b-5ckb9"] Nov 01 16:13:16.800075 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:16.800042 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="237a3f5e-83c1-43e0-a6f8-01dcfca57d32" path="/var/lib/kubelet/pods/237a3f5e-83c1-43e0-a6f8-01dcfca57d32/volumes" Nov 01 16:13:17.045874 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:17.045845 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" Nov 01 16:13:17.046399 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:17.046199 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" Nov 01 16:13:17.046942 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:17.046925 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" Nov 01 16:13:17.164234 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:17.164206 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-10b00a-659db85879-m4pxv" Nov 01 16:13:18.174452 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:18.174420 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-10b00a-5cdcfd8b5f-fwz5p" Nov 01 16:13:18.316379 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:13:18.316346 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:13:21.293444 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:21.293414 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n"] Nov 01 16:13:21.294305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:21.294270 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" podUID="905f9285-a9c5-4a2a-abbf-8114e5ac9f26" containerName="istio-proxy" containerID="cri-o://0436bad70374c2e8dfd0177fc35ad9fe7e7ff215cb4a8f53fffc136130c1b01c" gracePeriod=30 Nov 01 16:13:22.510336 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:22.510293 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc"] Nov 01 16:13:22.511759 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:22.511735 4363 cpu_manager.go:414] "containerMap: removing container" podUID="237a3f5e-83c1-43e0-a6f8-01dcfca57d32" containerName="maas-api" Nov 01 16:13:22.511759 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:22.511758 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="237a3f5e-83c1-43e0-a6f8-01dcfca57d32" containerName="maas-api" Nov 01 16:13:22.512141 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:22.512101 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="237a3f5e-83c1-43e0-a6f8-01dcfca57d32" containerName="maas-api" Nov 01 16:13:22.528049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:22.519420 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" Nov 01 16:13:22.528049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:22.524951 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a4240a-openshift-default-dockercfg-r7gj2\"" Nov 01 16:13:22.530429 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:22.529729 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc"] Nov 01 16:13:22.593852 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:22.593820 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23798655-3db1-4155-9f07-7c46e4fc7a14-credential-socket\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc\" (UID: \"23798655-3db1-4155-9f07-7c46e4fc7a14\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" Nov 01 16:13:22.594088 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:22.593924 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23798655-3db1-4155-9f07-7c46e4fc7a14-istio-podinfo\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc\" (UID: \"23798655-3db1-4155-9f07-7c46e4fc7a14\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" Nov 01 16:13:22.594088 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:22.593959 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23798655-3db1-4155-9f07-7c46e4fc7a14-workload-certs\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc\" (UID: \"23798655-3db1-4155-9f07-7c46e4fc7a14\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" Nov 01 16:13:22.594088 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:22.593984 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23798655-3db1-4155-9f07-7c46e4fc7a14-istiod-ca-cert\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc\" (UID: \"23798655-3db1-4155-9f07-7c46e4fc7a14\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" Nov 01 16:13:22.594279 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:22.594209 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkpzf\" (UniqueName: \"kubernetes.io/projected/23798655-3db1-4155-9f07-7c46e4fc7a14-kube-api-access-zkpzf\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc\" (UID: \"23798655-3db1-4155-9f07-7c46e4fc7a14\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" Nov 01 16:13:22.594432 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:22.594403 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23798655-3db1-4155-9f07-7c46e4fc7a14-istio-data\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc\" (UID: \"23798655-3db1-4155-9f07-7c46e4fc7a14\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" Nov 01 16:13:22.594685 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:22.594649 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23798655-3db1-4155-9f07-7c46e4fc7a14-workload-socket\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc\" (UID: \"23798655-3db1-4155-9f07-7c46e4fc7a14\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" Nov 01 16:13:22.594795 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:22.594701 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23798655-3db1-4155-9f07-7c46e4fc7a14-istio-envoy\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc\" (UID: \"23798655-3db1-4155-9f07-7c46e4fc7a14\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" Nov 01 16:13:22.594795 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:22.594732 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23798655-3db1-4155-9f07-7c46e4fc7a14-istio-token\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc\" (UID: \"23798655-3db1-4155-9f07-7c46e4fc7a14\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" Nov 01 16:13:22.721944 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:22.703536 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23798655-3db1-4155-9f07-7c46e4fc7a14-workload-socket\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc\" (UID: \"23798655-3db1-4155-9f07-7c46e4fc7a14\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" Nov 01 16:13:22.722337 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:22.722306 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23798655-3db1-4155-9f07-7c46e4fc7a14-istio-envoy\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc\" (UID: \"23798655-3db1-4155-9f07-7c46e4fc7a14\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" Nov 01 16:13:22.722969 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:22.722946 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23798655-3db1-4155-9f07-7c46e4fc7a14-istio-token\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc\" (UID: \"23798655-3db1-4155-9f07-7c46e4fc7a14\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" Nov 01 16:13:22.723622 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:22.723598 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23798655-3db1-4155-9f07-7c46e4fc7a14-credential-socket\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc\" (UID: \"23798655-3db1-4155-9f07-7c46e4fc7a14\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" Nov 01 16:13:22.761495 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:22.752792 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23798655-3db1-4155-9f07-7c46e4fc7a14-istio-podinfo\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc\" (UID: \"23798655-3db1-4155-9f07-7c46e4fc7a14\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" Nov 01 16:13:22.761495 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:22.752851 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23798655-3db1-4155-9f07-7c46e4fc7a14-workload-certs\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc\" (UID: \"23798655-3db1-4155-9f07-7c46e4fc7a14\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" Nov 01 16:13:22.761495 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:22.752975 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23798655-3db1-4155-9f07-7c46e4fc7a14-istiod-ca-cert\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc\" (UID: \"23798655-3db1-4155-9f07-7c46e4fc7a14\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" Nov 01 16:13:22.761495 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:22.753564 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkpzf\" (UniqueName: \"kubernetes.io/projected/23798655-3db1-4155-9f07-7c46e4fc7a14-kube-api-access-zkpzf\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc\" (UID: \"23798655-3db1-4155-9f07-7c46e4fc7a14\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" Nov 01 16:13:22.761495 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:22.753865 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23798655-3db1-4155-9f07-7c46e4fc7a14-istio-data\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc\" (UID: \"23798655-3db1-4155-9f07-7c46e4fc7a14\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" Nov 01 16:13:22.761495 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:22.710780 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23798655-3db1-4155-9f07-7c46e4fc7a14-workload-socket\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc\" (UID: \"23798655-3db1-4155-9f07-7c46e4fc7a14\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" Nov 01 16:13:22.761495 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:22.727632 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23798655-3db1-4155-9f07-7c46e4fc7a14-istio-envoy\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc\" (UID: \"23798655-3db1-4155-9f07-7c46e4fc7a14\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" Nov 01 16:13:22.761495 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:22.727884 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23798655-3db1-4155-9f07-7c46e4fc7a14-credential-socket\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc\" (UID: \"23798655-3db1-4155-9f07-7c46e4fc7a14\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" Nov 01 16:13:22.761495 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:22.754976 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23798655-3db1-4155-9f07-7c46e4fc7a14-workload-certs\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc\" (UID: \"23798655-3db1-4155-9f07-7c46e4fc7a14\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" Nov 01 16:13:22.761495 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:22.755096 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23798655-3db1-4155-9f07-7c46e4fc7a14-istio-data\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc\" (UID: \"23798655-3db1-4155-9f07-7c46e4fc7a14\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" Nov 01 16:13:22.761495 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:22.755548 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23798655-3db1-4155-9f07-7c46e4fc7a14-istiod-ca-cert\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc\" (UID: \"23798655-3db1-4155-9f07-7c46e4fc7a14\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" Nov 01 16:13:22.805373 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:22.805338 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23798655-3db1-4155-9f07-7c46e4fc7a14-istio-podinfo\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc\" (UID: \"23798655-3db1-4155-9f07-7c46e4fc7a14\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" Nov 01 16:13:22.812088 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:22.807621 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23798655-3db1-4155-9f07-7c46e4fc7a14-istio-token\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc\" (UID: \"23798655-3db1-4155-9f07-7c46e4fc7a14\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" Nov 01 16:13:22.813369 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:22.813341 4363 patch_prober.go:29] interesting pod/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:13:22.830130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:22.829732 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" podUID="905f9285-a9c5-4a2a-abbf-8114e5ac9f26" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:13:22.831793 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:22.831765 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkpzf\" (UniqueName: \"kubernetes.io/projected/23798655-3db1-4155-9f07-7c46e4fc7a14-kube-api-access-zkpzf\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc\" (UID: \"23798655-3db1-4155-9f07-7c46e4fc7a14\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" Nov 01 16:13:22.860599 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:22.860567 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" Nov 01 16:13:23.462563 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:13:23.462528 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23798655_3db1_4155_9f07_7c46e4fc7a14.slice/crio-aa8782129b74ab5d5ffc1bf2788f7a144aa03b4c80f3572d964528183ebc1d4f WatchSource:0}: Error finding container aa8782129b74ab5d5ffc1bf2788f7a144aa03b4c80f3572d964528183ebc1d4f: Status 404 returned error can't find the container with id aa8782129b74ab5d5ffc1bf2788f7a144aa03b4c80f3572d964528183ebc1d4f Nov 01 16:13:23.463078 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:23.462919 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc"] Nov 01 16:13:23.465656 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:23.465620 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:13:23.465748 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:23.465694 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:13:23.465748 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:23.465737 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:13:24.078525 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:24.078482 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" event={"ID":"23798655-3db1-4155-9f07-7c46e4fc7a14","Type":"ContainerStarted","Data":"a0c6a86522b2aa01f5b55176dbc467ba847f72036ff9ec5cacac183d6d7faef8"} Nov 01 16:13:24.079047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:24.078554 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" event={"ID":"23798655-3db1-4155-9f07-7c46e4fc7a14","Type":"ContainerStarted","Data":"aa8782129b74ab5d5ffc1bf2788f7a144aa03b4c80f3572d964528183ebc1d4f"} Nov 01 16:13:24.105840 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:24.105790 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" podStartSLOduration=2.10577774 podStartE2EDuration="2.10577774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:13:24.098420609 +0000 UTC m=+1537.430158262" watchObservedRunningTime="2026-11-01 16:13:24.10577774 +0000 UTC m=+1537.437515390" Nov 01 16:13:25.039991 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:25.039925 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" Nov 01 16:13:25.045127 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:25.041996 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" Nov 01 16:13:25.088960 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:25.088926 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" Nov 01 16:13:25.089940 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:25.089915 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" Nov 01 16:13:26.646964 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:26.646938 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" Nov 01 16:13:26.736180 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:26.736146 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-istio-token\" (UniqueName: \"kubernetes.io/projected/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-istio-token\") pod \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\" (UID: \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\") " Nov 01 16:13:26.736366 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:26.736254 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-credential-socket\") pod \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\" (UID: \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\") " Nov 01 16:13:26.736366 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:26.736300 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-kube-api-access-62gfr\" (UniqueName: \"kubernetes.io/projected/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-kube-api-access-62gfr\") pod \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\" (UID: \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\") " Nov 01 16:13:26.736366 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:26.736352 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-workload-socket\") pod \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\" (UID: \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\") " Nov 01 16:13:26.736819 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:26.736410 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-istio-data\") pod \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\" (UID: \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\") " Nov 01 16:13:26.736819 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:26.736590 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-workload-certs\") pod \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\" (UID: \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\") " Nov 01 16:13:26.736819 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:26.736748 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-workload-socket" pod "905f9285-a9c5-4a2a-abbf-8114e5ac9f26" (UID: "905f9285-a9c5-4a2a-abbf-8114e5ac9f26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:13:26.736819 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:26.736812 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-istio-envoy\") pod \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\" (UID: \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\") " Nov 01 16:13:26.738541 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:26.736852 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-istiod-ca-cert\") pod \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\" (UID: \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\") " Nov 01 16:13:26.738541 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:26.736904 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-istio-podinfo\") pod \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\" (UID: \"905f9285-a9c5-4a2a-abbf-8114e5ac9f26\") " Nov 01 16:13:26.738541 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:26.736847 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-workload-certs" pod "905f9285-a9c5-4a2a-abbf-8114e5ac9f26" (UID: "905f9285-a9c5-4a2a-abbf-8114e5ac9f26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:13:26.738541 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:26.736898 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-credential-socket" pod "905f9285-a9c5-4a2a-abbf-8114e5ac9f26" (UID: "905f9285-a9c5-4a2a-abbf-8114e5ac9f26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:13:26.738541 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:26.737005 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-istio-data" pod "905f9285-a9c5-4a2a-abbf-8114e5ac9f26" (UID: "905f9285-a9c5-4a2a-abbf-8114e5ac9f26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:13:26.738541 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:26.737274 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-istiod-ca-cert" pod "905f9285-a9c5-4a2a-abbf-8114e5ac9f26" (UID: "905f9285-a9c5-4a2a-abbf-8114e5ac9f26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:13:26.738541 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:26.737935 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:26.738541 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:26.737960 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:26.738541 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:26.737977 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:26.738541 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:26.737993 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:26.738541 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:26.738009 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:26.739576 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:26.739518 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-istio-envoy" pod "905f9285-a9c5-4a2a-abbf-8114e5ac9f26" (UID: "905f9285-a9c5-4a2a-abbf-8114e5ac9f26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:13:26.739576 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:26.739547 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-istio-podinfo" pod "905f9285-a9c5-4a2a-abbf-8114e5ac9f26" (UID: "905f9285-a9c5-4a2a-abbf-8114e5ac9f26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:13:26.739874 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:26.739850 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-istio-token" pod "905f9285-a9c5-4a2a-abbf-8114e5ac9f26" (UID: "905f9285-a9c5-4a2a-abbf-8114e5ac9f26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:13:26.740699 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:26.740677 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-kube-api-access-62gfr" pod "905f9285-a9c5-4a2a-abbf-8114e5ac9f26" (UID: "905f9285-a9c5-4a2a-abbf-8114e5ac9f26"). InnerVolumeSpecName "kube-api-access-62gfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:13:26.840432 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:26.840403 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:26.840432 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:26.840430 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62gfr\" (UniqueName: \"kubernetes.io/projected/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-kube-api-access-62gfr\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:26.840695 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:26.840444 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:26.840695 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:26.840457 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/905f9285-a9c5-4a2a-abbf-8114e5ac9f26-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:27.115757 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:27.115724 4363 generic.go:356] "Generic (PLEG): container finished" podID="905f9285-a9c5-4a2a-abbf-8114e5ac9f26" containerID="0436bad70374c2e8dfd0177fc35ad9fe7e7ff215cb4a8f53fffc136130c1b01c" exitCode=0 Nov 01 16:13:27.115923 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:27.115772 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" Nov 01 16:13:27.115923 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:27.115802 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" event={"ID":"905f9285-a9c5-4a2a-abbf-8114e5ac9f26","Type":"ContainerDied","Data":"0436bad70374c2e8dfd0177fc35ad9fe7e7ff215cb4a8f53fffc136130c1b01c"} Nov 01 16:13:27.115923 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:27.115863 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n" event={"ID":"905f9285-a9c5-4a2a-abbf-8114e5ac9f26","Type":"ContainerDied","Data":"c5f51fe52ad144614d5f8a6c68b796c349aad0fb7aabce6e15503507f4b23f8c"} Nov 01 16:13:27.115923 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:27.115890 4363 scope.go:122] "RemoveContainer" containerID="0436bad70374c2e8dfd0177fc35ad9fe7e7ff215cb4a8f53fffc136130c1b01c" Nov 01 16:13:27.127475 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:27.127461 4363 scope.go:122] "RemoveContainer" containerID="d4ed8c081c6ccf18dbbda3c56e45e17bc2f2c8535f043a60ffef029233d40292" Nov 01 16:13:27.137372 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:27.137352 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n"] Nov 01 16:13:27.138473 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:27.138459 4363 scope.go:122] "RemoveContainer" containerID="0436bad70374c2e8dfd0177fc35ad9fe7e7ff215cb4a8f53fffc136130c1b01c" Nov 01 16:13:27.138752 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:13:27.138731 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0436bad70374c2e8dfd0177fc35ad9fe7e7ff215cb4a8f53fffc136130c1b01c\": container with ID starting with 0436bad70374c2e8dfd0177fc35ad9fe7e7ff215cb4a8f53fffc136130c1b01c not found: ID does not exist" containerID="0436bad70374c2e8dfd0177fc35ad9fe7e7ff215cb4a8f53fffc136130c1b01c" Nov 01 16:13:27.138807 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:27.138761 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0436bad70374c2e8dfd0177fc35ad9fe7e7ff215cb4a8f53fffc136130c1b01c"} err="failed to get container status \"0436bad70374c2e8dfd0177fc35ad9fe7e7ff215cb4a8f53fffc136130c1b01c\": rpc error: code = NotFound desc = could not find container \"0436bad70374c2e8dfd0177fc35ad9fe7e7ff215cb4a8f53fffc136130c1b01c\": container with ID starting with 0436bad70374c2e8dfd0177fc35ad9fe7e7ff215cb4a8f53fffc136130c1b01c not found: ID does not exist" Nov 01 16:13:27.138807 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:27.138778 4363 scope.go:122] "RemoveContainer" containerID="d4ed8c081c6ccf18dbbda3c56e45e17bc2f2c8535f043a60ffef029233d40292" Nov 01 16:13:27.139030 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:13:27.138991 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4ed8c081c6ccf18dbbda3c56e45e17bc2f2c8535f043a60ffef029233d40292\": container with ID starting with d4ed8c081c6ccf18dbbda3c56e45e17bc2f2c8535f043a60ffef029233d40292 not found: ID does not exist" containerID="d4ed8c081c6ccf18dbbda3c56e45e17bc2f2c8535f043a60ffef029233d40292" Nov 01 16:13:27.139157 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:27.139065 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4ed8c081c6ccf18dbbda3c56e45e17bc2f2c8535f043a60ffef029233d40292"} err="failed to get container status \"d4ed8c081c6ccf18dbbda3c56e45e17bc2f2c8535f043a60ffef029233d40292\": rpc error: code = NotFound desc = could not find container \"d4ed8c081c6ccf18dbbda3c56e45e17bc2f2c8535f043a60ffef029233d40292\": container with ID starting with d4ed8c081c6ccf18dbbda3c56e45e17bc2f2c8535f043a60ffef029233d40292 not found: ID does not exist" Nov 01 16:13:27.144718 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:27.144694 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-58b0d1-openshift-default-b5c977647-ltl2n"] Nov 01 16:13:27.807659 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:27.807627 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a4240a-745976f465-st69v"] Nov 01 16:13:27.808662 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:27.808643 4363 cpu_manager.go:414] "containerMap: removing container" podUID="905f9285-a9c5-4a2a-abbf-8114e5ac9f26" containerName="istio-proxy" Nov 01 16:13:27.808716 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:27.808662 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="905f9285-a9c5-4a2a-abbf-8114e5ac9f26" containerName="istio-proxy" Nov 01 16:13:27.808716 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:27.808699 4363 cpu_manager.go:414] "containerMap: removing container" podUID="905f9285-a9c5-4a2a-abbf-8114e5ac9f26" containerName="istio-proxy" Nov 01 16:13:27.808716 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:27.808706 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="905f9285-a9c5-4a2a-abbf-8114e5ac9f26" containerName="istio-proxy" Nov 01 16:13:27.808849 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:27.808840 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="905f9285-a9c5-4a2a-abbf-8114e5ac9f26" containerName="istio-proxy" Nov 01 16:13:27.814816 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:27.814797 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a4240a-745976f465-st69v" Nov 01 16:13:27.819622 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:27.818952 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a4240a\"" Nov 01 16:13:27.824064 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:27.823793 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a4240a-745976f465-st69v"] Nov 01 16:13:27.841643 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:27.841586 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a4240a-86f876f9cf-9bqvq"] Nov 01 16:13:27.843527 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:27.843500 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="905f9285-a9c5-4a2a-abbf-8114e5ac9f26" containerName="istio-proxy" Nov 01 16:13:27.855165 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:27.855137 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a4240a-86f876f9cf-9bqvq" Nov 01 16:13:27.856937 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:27.856910 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a4240a-86f876f9cf-9bqvq"] Nov 01 16:13:27.859334 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:27.859308 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a4240a-dockercfg-j5rdq\"" Nov 01 16:13:28.051069 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:27.966349 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1964cd1c-696f-4f45-83ba-8dec81dbc624-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a4240a-86f876f9cf-9bqvq\" (UID: \"1964cd1c-696f-4f45-83ba-8dec81dbc624\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a4240a-86f876f9cf-9bqvq" Nov 01 16:13:28.055042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:28.051509 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4sfq\" (UniqueName: \"kubernetes.io/projected/c8c062a2-3461-4bbe-ac11-81966a3cec16-kube-api-access-h4sfq\") pod \"payload-processing-e2e-inf-a-a4240a-745976f465-st69v\" (UID: \"c8c062a2-3461-4bbe-ac11-81966a3cec16\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a4240a-745976f465-st69v" Nov 01 16:13:28.066422 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:28.066335 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ftjc\" (UniqueName: \"kubernetes.io/projected/1964cd1c-696f-4f45-83ba-8dec81dbc624-kube-api-access-7ftjc\") pod \"payload-pre-processing-e2e-inf-a-a4240a-86f876f9cf-9bqvq\" (UID: \"1964cd1c-696f-4f45-83ba-8dec81dbc624\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a4240a-86f876f9cf-9bqvq" Nov 01 16:13:28.088328 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:28.088291 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8c062a2-3461-4bbe-ac11-81966a3cec16-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a4240a-745976f465-st69v\" (UID: \"c8c062a2-3461-4bbe-ac11-81966a3cec16\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a4240a-745976f465-st69v" Nov 01 16:13:28.193456 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:28.193411 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1964cd1c-696f-4f45-83ba-8dec81dbc624-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a4240a-86f876f9cf-9bqvq\" (UID: \"1964cd1c-696f-4f45-83ba-8dec81dbc624\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a4240a-86f876f9cf-9bqvq" Nov 01 16:13:28.193876 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:28.193850 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4sfq\" (UniqueName: \"kubernetes.io/projected/c8c062a2-3461-4bbe-ac11-81966a3cec16-kube-api-access-h4sfq\") pod \"payload-processing-e2e-inf-a-a4240a-745976f465-st69v\" (UID: \"c8c062a2-3461-4bbe-ac11-81966a3cec16\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a4240a-745976f465-st69v" Nov 01 16:13:28.194163 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:28.194145 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ftjc\" (UniqueName: \"kubernetes.io/projected/1964cd1c-696f-4f45-83ba-8dec81dbc624-kube-api-access-7ftjc\") pod \"payload-pre-processing-e2e-inf-a-a4240a-86f876f9cf-9bqvq\" (UID: \"1964cd1c-696f-4f45-83ba-8dec81dbc624\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a4240a-86f876f9cf-9bqvq" Nov 01 16:13:28.194491 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:28.194464 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8c062a2-3461-4bbe-ac11-81966a3cec16-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a4240a-745976f465-st69v\" (UID: \"c8c062a2-3461-4bbe-ac11-81966a3cec16\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a4240a-745976f465-st69v" Nov 01 16:13:28.211144 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:28.211113 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8c062a2-3461-4bbe-ac11-81966a3cec16-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a4240a-745976f465-st69v\" (UID: \"c8c062a2-3461-4bbe-ac11-81966a3cec16\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a4240a-745976f465-st69v" Nov 01 16:13:28.298085 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:28.281866 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4sfq\" (UniqueName: \"kubernetes.io/projected/c8c062a2-3461-4bbe-ac11-81966a3cec16-kube-api-access-h4sfq\") pod \"payload-processing-e2e-inf-a-a4240a-745976f465-st69v\" (UID: \"c8c062a2-3461-4bbe-ac11-81966a3cec16\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a4240a-745976f465-st69v" Nov 01 16:13:28.298085 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:28.283121 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1964cd1c-696f-4f45-83ba-8dec81dbc624-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a4240a-86f876f9cf-9bqvq\" (UID: \"1964cd1c-696f-4f45-83ba-8dec81dbc624\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a4240a-86f876f9cf-9bqvq" Nov 01 16:13:28.305950 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:28.305914 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ftjc\" (UniqueName: \"kubernetes.io/projected/1964cd1c-696f-4f45-83ba-8dec81dbc624-kube-api-access-7ftjc\") pod \"payload-pre-processing-e2e-inf-a-a4240a-86f876f9cf-9bqvq\" (UID: \"1964cd1c-696f-4f45-83ba-8dec81dbc624\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a4240a-86f876f9cf-9bqvq" Nov 01 16:13:28.437772 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:28.437739 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a4240a-745976f465-st69v" Nov 01 16:13:28.476563 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:28.476510 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a4240a-86f876f9cf-9bqvq" Nov 01 16:13:28.810532 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:28.810443 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="905f9285-a9c5-4a2a-abbf-8114e5ac9f26" path="/var/lib/kubelet/pods/905f9285-a9c5-4a2a-abbf-8114e5ac9f26/volumes" Nov 01 16:13:28.942401 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:13:28.942367 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8c062a2_3461_4bbe_ac11_81966a3cec16.slice/crio-a60b35030e1e00c63a411a9099bd566df633cfbcb23a61de9be2114388e4a81c WatchSource:0}: Error finding container a60b35030e1e00c63a411a9099bd566df633cfbcb23a61de9be2114388e4a81c: Status 404 returned error can't find the container with id a60b35030e1e00c63a411a9099bd566df633cfbcb23a61de9be2114388e4a81c Nov 01 16:13:28.942547 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:28.942524 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a4240a-745976f465-st69v"] Nov 01 16:13:28.998940 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:28.998913 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a4240a-86f876f9cf-9bqvq"] Nov 01 16:13:29.002813 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:13:29.002786 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1964cd1c_696f_4f45_83ba_8dec81dbc624.slice/crio-69ded54c1b5da5010566cff659b28495d5ab0a72febad51350d7ee04e8d5e8e6 WatchSource:0}: Error finding container 69ded54c1b5da5010566cff659b28495d5ab0a72febad51350d7ee04e8d5e8e6: Status 404 returned error can't find the container with id 69ded54c1b5da5010566cff659b28495d5ab0a72febad51350d7ee04e8d5e8e6 Nov 01 16:13:29.249519 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:29.249488 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a4240a-745976f465-st69v" event={"ID":"c8c062a2-3461-4bbe-ac11-81966a3cec16","Type":"ContainerStarted","Data":"c9eabc2a9000378c0a532d6557de1af55e08161638f9cb28f7c4fed6f308a9ba"} Nov 01 16:13:29.249706 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:29.249551 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a4240a-745976f465-st69v" event={"ID":"c8c062a2-3461-4bbe-ac11-81966a3cec16","Type":"ContainerStarted","Data":"a60b35030e1e00c63a411a9099bd566df633cfbcb23a61de9be2114388e4a81c"} Nov 01 16:13:29.249706 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:29.249580 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a4240a-745976f465-st69v" Nov 01 16:13:29.251300 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:29.251266 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a4240a-86f876f9cf-9bqvq" event={"ID":"1964cd1c-696f-4f45-83ba-8dec81dbc624","Type":"ContainerStarted","Data":"f45baca75c75cadf94e4a74a6f91e53a1304d6f802562fe14b7a39dd9e9b89ba"} Nov 01 16:13:29.251422 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:29.251312 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a4240a-86f876f9cf-9bqvq" event={"ID":"1964cd1c-696f-4f45-83ba-8dec81dbc624","Type":"ContainerStarted","Data":"69ded54c1b5da5010566cff659b28495d5ab0a72febad51350d7ee04e8d5e8e6"} Nov 01 16:13:29.251422 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:29.251352 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a4240a-86f876f9cf-9bqvq" Nov 01 16:13:29.268144 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:29.268105 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-a4240a-745976f465-st69v" podStartSLOduration=2.268093581 podStartE2EDuration="2.268093581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:13:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:13:29.26487334 +0000 UTC m=+1542.596610987" watchObservedRunningTime="2026-11-01 16:13:29.268093581 +0000 UTC m=+1542.599831230" Nov 01 16:13:29.288136 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:29.288080 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a4240a-86f876f9cf-9bqvq" podStartSLOduration=2.288006382 podStartE2EDuration="2.288006382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:13:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:13:29.283837789 +0000 UTC m=+1542.615575438" watchObservedRunningTime="2026-11-01 16:13:29.288006382 +0000 UTC m=+1542.619744032" Nov 01 16:13:35.407039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:35.399975 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-26310f-ccf8d9588-wczxf"] Nov 01 16:13:35.407039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:35.400375 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26310f-ccf8d9588-wczxf" podUID="42b30c2b-3415-45cb-9120-eefb427210d5" containerName="maas-api" containerID="cri-o://ac67bfe29eb9783505b353966080b1488f7f0003a5a9299c96e2e709e25f7821" gracePeriod=30 Nov 01 16:13:36.468037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:36.456205 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26310f-ccf8d9588-wczxf" Nov 01 16:13:36.468037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:36.457008 4363 generic.go:356] "Generic (PLEG): container finished" podID="42b30c2b-3415-45cb-9120-eefb427210d5" containerID="ac67bfe29eb9783505b353966080b1488f7f0003a5a9299c96e2e709e25f7821" exitCode=0 Nov 01 16:13:36.468037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:36.457301 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26310f-ccf8d9588-wczxf" event={"ID":"42b30c2b-3415-45cb-9120-eefb427210d5","Type":"ContainerDied","Data":"ac67bfe29eb9783505b353966080b1488f7f0003a5a9299c96e2e709e25f7821"} Nov 01 16:13:36.468037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:36.457366 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26310f-ccf8d9588-wczxf" event={"ID":"42b30c2b-3415-45cb-9120-eefb427210d5","Type":"ContainerDied","Data":"17303043d02f389f6928993cd257ad335e87e3b5883034b04096d1caed631f26"} Nov 01 16:13:36.468037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:36.457409 4363 scope.go:122] "RemoveContainer" containerID="ac67bfe29eb9783505b353966080b1488f7f0003a5a9299c96e2e709e25f7821" Nov 01 16:13:36.529104 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:36.529075 4363 scope.go:122] "RemoveContainer" containerID="ac67bfe29eb9783505b353966080b1488f7f0003a5a9299c96e2e709e25f7821" Nov 01 16:13:36.534249 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:13:36.534215 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac67bfe29eb9783505b353966080b1488f7f0003a5a9299c96e2e709e25f7821\": container with ID starting with ac67bfe29eb9783505b353966080b1488f7f0003a5a9299c96e2e709e25f7821 not found: ID does not exist" containerID="ac67bfe29eb9783505b353966080b1488f7f0003a5a9299c96e2e709e25f7821" Nov 01 16:13:36.534413 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:36.534386 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac67bfe29eb9783505b353966080b1488f7f0003a5a9299c96e2e709e25f7821"} err="failed to get container status \"ac67bfe29eb9783505b353966080b1488f7f0003a5a9299c96e2e709e25f7821\": rpc error: code = NotFound desc = could not find container \"ac67bfe29eb9783505b353966080b1488f7f0003a5a9299c96e2e709e25f7821\": container with ID starting with ac67bfe29eb9783505b353966080b1488f7f0003a5a9299c96e2e709e25f7821 not found: ID does not exist" Nov 01 16:13:36.641610 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:36.640686 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42b30c2b-3415-45cb-9120-eefb427210d5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/42b30c2b-3415-45cb-9120-eefb427210d5-metrics-certs\") pod \"42b30c2b-3415-45cb-9120-eefb427210d5\" (UID: \"42b30c2b-3415-45cb-9120-eefb427210d5\") " Nov 01 16:13:36.641610 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:36.640889 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42b30c2b-3415-45cb-9120-eefb427210d5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42b30c2b-3415-45cb-9120-eefb427210d5-openshift-service-ca\") pod \"42b30c2b-3415-45cb-9120-eefb427210d5\" (UID: \"42b30c2b-3415-45cb-9120-eefb427210d5\") " Nov 01 16:13:36.641610 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:36.641139 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42b30c2b-3415-45cb-9120-eefb427210d5-kube-api-access-qvtsw\" (UniqueName: \"kubernetes.io/projected/42b30c2b-3415-45cb-9120-eefb427210d5-kube-api-access-qvtsw\") pod \"42b30c2b-3415-45cb-9120-eefb427210d5\" (UID: \"42b30c2b-3415-45cb-9120-eefb427210d5\") " Nov 01 16:13:36.641610 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:36.641186 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42b30c2b-3415-45cb-9120-eefb427210d5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42b30c2b-3415-45cb-9120-eefb427210d5-maas-api-tls\") pod \"42b30c2b-3415-45cb-9120-eefb427210d5\" (UID: \"42b30c2b-3415-45cb-9120-eefb427210d5\") " Nov 01 16:13:36.642117 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:36.642087 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42b30c2b-3415-45cb-9120-eefb427210d5-openshift-service-ca" pod "42b30c2b-3415-45cb-9120-eefb427210d5" (UID: "42b30c2b-3415-45cb-9120-eefb427210d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:13:36.646771 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:36.643202 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42b30c2b-3415-45cb-9120-eefb427210d5-metrics-certs" pod "42b30c2b-3415-45cb-9120-eefb427210d5" (UID: "42b30c2b-3415-45cb-9120-eefb427210d5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:13:36.646771 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:36.644247 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42b30c2b-3415-45cb-9120-eefb427210d5-kube-api-access-qvtsw" pod "42b30c2b-3415-45cb-9120-eefb427210d5" (UID: "42b30c2b-3415-45cb-9120-eefb427210d5"). InnerVolumeSpecName "kube-api-access-qvtsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:13:36.646771 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:36.644471 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42b30c2b-3415-45cb-9120-eefb427210d5-maas-api-tls" pod "42b30c2b-3415-45cb-9120-eefb427210d5" (UID: "42b30c2b-3415-45cb-9120-eefb427210d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:13:36.744034 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:36.743981 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvtsw\" (UniqueName: \"kubernetes.io/projected/42b30c2b-3415-45cb-9120-eefb427210d5-kube-api-access-qvtsw\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:36.744034 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:36.744012 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42b30c2b-3415-45cb-9120-eefb427210d5-maas-api-tls\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:36.744034 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:36.744035 4363 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42b30c2b-3415-45cb-9120-eefb427210d5-metrics-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:36.744034 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:36.744044 4363 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42b30c2b-3415-45cb-9120-eefb427210d5-openshift-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:37.470200 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:37.470170 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26310f-ccf8d9588-wczxf" Nov 01 16:13:37.500387 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:37.500353 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-26310f-ccf8d9588-wczxf"] Nov 01 16:13:37.511406 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:37.511366 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-26310f-ccf8d9588-wczxf"] Nov 01 16:13:37.752632 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:37.752554 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb"] Nov 01 16:13:37.753886 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:37.753862 4363 cpu_manager.go:414] "containerMap: removing container" podUID="42b30c2b-3415-45cb-9120-eefb427210d5" containerName="maas-api" Nov 01 16:13:37.753945 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:37.753890 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42b30c2b-3415-45cb-9120-eefb427210d5" containerName="maas-api" Nov 01 16:13:37.754226 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:37.754206 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="42b30c2b-3415-45cb-9120-eefb427210d5" containerName="maas-api" Nov 01 16:13:37.764170 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:37.764142 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" Nov 01 16:13:37.765264 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:37.765232 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb"] Nov 01 16:13:37.774626 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:37.774606 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-784444\"/\"test-model-784444-kserve-self-signed-certs\"" Nov 01 16:13:37.777120 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:37.777096 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-784444\"/\"default-dockercfg-d5mx8\"" Nov 01 16:13:37.777823 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:37.777411 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-784444\"/\"openshift-service-ca.crt\"" Nov 01 16:13:37.777823 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:37.777588 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-784444\"/\"kube-root-ca.crt\"" Nov 01 16:13:37.865557 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:37.865527 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cbd41ee-3134-4b54-9ea1-0c13921269bc-tmp-dir\") pod \"test-model-784444-kserve-5679d76f49-vpwkb\" (UID: \"9cbd41ee-3134-4b54-9ea1-0c13921269bc\") " pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" Nov 01 16:13:37.865735 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:37.865648 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cbd41ee-3134-4b54-9ea1-0c13921269bc-dshm\") pod \"test-model-784444-kserve-5679d76f49-vpwkb\" (UID: \"9cbd41ee-3134-4b54-9ea1-0c13921269bc\") " pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" Nov 01 16:13:37.865848 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:37.865823 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cbd41ee-3134-4b54-9ea1-0c13921269bc-tls-certs\") pod \"test-model-784444-kserve-5679d76f49-vpwkb\" (UID: \"9cbd41ee-3134-4b54-9ea1-0c13921269bc\") " pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" Nov 01 16:13:37.865912 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:37.865873 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nj2f\" (UniqueName: \"kubernetes.io/projected/9cbd41ee-3134-4b54-9ea1-0c13921269bc-kube-api-access-7nj2f\") pod \"test-model-784444-kserve-5679d76f49-vpwkb\" (UID: \"9cbd41ee-3134-4b54-9ea1-0c13921269bc\") " pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" Nov 01 16:13:37.865989 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:37.865975 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cbd41ee-3134-4b54-9ea1-0c13921269bc-home\") pod \"test-model-784444-kserve-5679d76f49-vpwkb\" (UID: \"9cbd41ee-3134-4b54-9ea1-0c13921269bc\") " pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" Nov 01 16:13:37.866113 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:37.866093 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cbd41ee-3134-4b54-9ea1-0c13921269bc-model-cache\") pod \"test-model-784444-kserve-5679d76f49-vpwkb\" (UID: \"9cbd41ee-3134-4b54-9ea1-0c13921269bc\") " pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" Nov 01 16:13:37.971477 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:37.970562 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cbd41ee-3134-4b54-9ea1-0c13921269bc-home\") pod \"test-model-784444-kserve-5679d76f49-vpwkb\" (UID: \"9cbd41ee-3134-4b54-9ea1-0c13921269bc\") " pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" Nov 01 16:13:37.971477 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:37.970678 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cbd41ee-3134-4b54-9ea1-0c13921269bc-model-cache\") pod \"test-model-784444-kserve-5679d76f49-vpwkb\" (UID: \"9cbd41ee-3134-4b54-9ea1-0c13921269bc\") " pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" Nov 01 16:13:37.971477 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:37.970814 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cbd41ee-3134-4b54-9ea1-0c13921269bc-tmp-dir\") pod \"test-model-784444-kserve-5679d76f49-vpwkb\" (UID: \"9cbd41ee-3134-4b54-9ea1-0c13921269bc\") " pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" Nov 01 16:13:37.971477 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:37.970940 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cbd41ee-3134-4b54-9ea1-0c13921269bc-dshm\") pod \"test-model-784444-kserve-5679d76f49-vpwkb\" (UID: \"9cbd41ee-3134-4b54-9ea1-0c13921269bc\") " pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" Nov 01 16:13:37.972034 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:37.971992 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cbd41ee-3134-4b54-9ea1-0c13921269bc-tls-certs\") pod \"test-model-784444-kserve-5679d76f49-vpwkb\" (UID: \"9cbd41ee-3134-4b54-9ea1-0c13921269bc\") " pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" Nov 01 16:13:37.972242 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:37.972223 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nj2f\" (UniqueName: \"kubernetes.io/projected/9cbd41ee-3134-4b54-9ea1-0c13921269bc-kube-api-access-7nj2f\") pod \"test-model-784444-kserve-5679d76f49-vpwkb\" (UID: \"9cbd41ee-3134-4b54-9ea1-0c13921269bc\") " pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" Nov 01 16:13:37.974432 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:37.974404 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cbd41ee-3134-4b54-9ea1-0c13921269bc-model-cache\") pod \"test-model-784444-kserve-5679d76f49-vpwkb\" (UID: \"9cbd41ee-3134-4b54-9ea1-0c13921269bc\") " pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" Nov 01 16:13:37.974855 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:37.974832 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cbd41ee-3134-4b54-9ea1-0c13921269bc-home\") pod \"test-model-784444-kserve-5679d76f49-vpwkb\" (UID: \"9cbd41ee-3134-4b54-9ea1-0c13921269bc\") " pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" Nov 01 16:13:37.975259 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:37.975239 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cbd41ee-3134-4b54-9ea1-0c13921269bc-tmp-dir\") pod \"test-model-784444-kserve-5679d76f49-vpwkb\" (UID: \"9cbd41ee-3134-4b54-9ea1-0c13921269bc\") " pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" Nov 01 16:13:38.016916 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:38.016837 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nj2f\" (UniqueName: \"kubernetes.io/projected/9cbd41ee-3134-4b54-9ea1-0c13921269bc-kube-api-access-7nj2f\") pod \"test-model-784444-kserve-5679d76f49-vpwkb\" (UID: \"9cbd41ee-3134-4b54-9ea1-0c13921269bc\") " pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" Nov 01 16:13:38.037059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:38.017738 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cbd41ee-3134-4b54-9ea1-0c13921269bc-tls-certs\") pod \"test-model-784444-kserve-5679d76f49-vpwkb\" (UID: \"9cbd41ee-3134-4b54-9ea1-0c13921269bc\") " pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" Nov 01 16:13:38.037059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:38.020211 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cbd41ee-3134-4b54-9ea1-0c13921269bc-dshm\") pod \"test-model-784444-kserve-5679d76f49-vpwkb\" (UID: \"9cbd41ee-3134-4b54-9ea1-0c13921269bc\") " pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" Nov 01 16:13:38.090038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:38.086448 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" Nov 01 16:13:38.803516 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:38.803446 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42b30c2b-3415-45cb-9120-eefb427210d5" path="/var/lib/kubelet/pods/42b30c2b-3415-45cb-9120-eefb427210d5/volumes" Nov 01 16:13:38.855307 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:13:38.855273 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9cbd41ee_3134_4b54_9ea1_0c13921269bc.slice/crio-3b6ec5f3f1103f055790ef90c637e1be10fcad9933804d999c07e4e473a3ab7e WatchSource:0}: Error finding container 3b6ec5f3f1103f055790ef90c637e1be10fcad9933804d999c07e4e473a3ab7e: Status 404 returned error can't find the container with id 3b6ec5f3f1103f055790ef90c637e1be10fcad9933804d999c07e4e473a3ab7e Nov 01 16:13:38.855736 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:38.855718 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb"] Nov 01 16:13:39.572098 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:39.572064 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" event={"ID":"9cbd41ee-3134-4b54-9ea1-0c13921269bc","Type":"ContainerStarted","Data":"9419b2e7d4db15972f5be522fe1bb744d2d77077fe7e3246499c67ca7d04f926"} Nov 01 16:13:39.572296 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:39.572108 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" event={"ID":"9cbd41ee-3134-4b54-9ea1-0c13921269bc","Type":"ContainerStarted","Data":"3b6ec5f3f1103f055790ef90c637e1be10fcad9933804d999c07e4e473a3ab7e"} Nov 01 16:13:39.588666 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:39.588619 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" podStartSLOduration=2.588605717 podStartE2EDuration="2.588605717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:13:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:13:39.58741757 +0000 UTC m=+1552.919155221" watchObservedRunningTime="2026-11-01 16:13:39.588605717 +0000 UTC m=+1552.920343370" Nov 01 16:13:40.087202 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:40.087165 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" Nov 01 16:13:40.291088 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:40.291058 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a4240a-86f876f9cf-9bqvq" Nov 01 16:13:40.291261 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:40.291151 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a4240a-745976f465-st69v" Nov 01 16:13:44.017351 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:44.017305 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5"] Nov 01 16:13:44.017841 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:44.017578 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" podUID="2d1193ea-2099-4ce6-b417-82d1e8e051b8" containerName="istio-proxy" containerID="cri-o://7fd2600eea08863a59ea3299fe596b1fe3fea4ebf8abf813d7685d22a2a8192a" gracePeriod=30 Nov 01 16:13:45.211929 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:45.211881 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl"] Nov 01 16:13:45.219107 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:45.219077 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" Nov 01 16:13:45.225276 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:45.225241 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-9e044c-openshift-default-dockercfg-4tjb7\"" Nov 01 16:13:45.227908 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:45.227881 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl"] Nov 01 16:13:45.393042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:45.365420 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22eae6cc-4752-4174-a0f0-573adc82e32b-workload-certs\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl\" (UID: \"22eae6cc-4752-4174-a0f0-573adc82e32b\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" Nov 01 16:13:45.416043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:45.407614 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22eae6cc-4752-4174-a0f0-573adc82e32b-istiod-ca-cert\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl\" (UID: \"22eae6cc-4752-4174-a0f0-573adc82e32b\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" Nov 01 16:13:45.416043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:45.407805 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22eae6cc-4752-4174-a0f0-573adc82e32b-istio-token\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl\" (UID: \"22eae6cc-4752-4174-a0f0-573adc82e32b\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" Nov 01 16:13:45.416043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:45.407848 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22eae6cc-4752-4174-a0f0-573adc82e32b-istio-data\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl\" (UID: \"22eae6cc-4752-4174-a0f0-573adc82e32b\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" Nov 01 16:13:45.416043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:45.407882 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttxrr\" (UniqueName: \"kubernetes.io/projected/22eae6cc-4752-4174-a0f0-573adc82e32b-kube-api-access-ttxrr\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl\" (UID: \"22eae6cc-4752-4174-a0f0-573adc82e32b\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" Nov 01 16:13:45.416043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:45.407930 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22eae6cc-4752-4174-a0f0-573adc82e32b-workload-socket\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl\" (UID: \"22eae6cc-4752-4174-a0f0-573adc82e32b\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" Nov 01 16:13:45.416043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:45.408195 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22eae6cc-4752-4174-a0f0-573adc82e32b-istio-podinfo\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl\" (UID: \"22eae6cc-4752-4174-a0f0-573adc82e32b\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" Nov 01 16:13:45.416043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:45.408280 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22eae6cc-4752-4174-a0f0-573adc82e32b-credential-socket\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl\" (UID: \"22eae6cc-4752-4174-a0f0-573adc82e32b\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" Nov 01 16:13:45.416043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:45.408331 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22eae6cc-4752-4174-a0f0-573adc82e32b-istio-envoy\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl\" (UID: \"22eae6cc-4752-4174-a0f0-573adc82e32b\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" Nov 01 16:13:45.513098 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:45.512066 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttxrr\" (UniqueName: \"kubernetes.io/projected/22eae6cc-4752-4174-a0f0-573adc82e32b-kube-api-access-ttxrr\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl\" (UID: \"22eae6cc-4752-4174-a0f0-573adc82e32b\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" Nov 01 16:13:45.513098 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:45.512520 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22eae6cc-4752-4174-a0f0-573adc82e32b-workload-socket\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl\" (UID: \"22eae6cc-4752-4174-a0f0-573adc82e32b\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" Nov 01 16:13:45.513098 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:45.512687 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22eae6cc-4752-4174-a0f0-573adc82e32b-istio-podinfo\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl\" (UID: \"22eae6cc-4752-4174-a0f0-573adc82e32b\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" Nov 01 16:13:45.513098 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:45.512793 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22eae6cc-4752-4174-a0f0-573adc82e32b-credential-socket\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl\" (UID: \"22eae6cc-4752-4174-a0f0-573adc82e32b\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" Nov 01 16:13:45.513098 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:45.512854 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22eae6cc-4752-4174-a0f0-573adc82e32b-istio-envoy\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl\" (UID: \"22eae6cc-4752-4174-a0f0-573adc82e32b\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" Nov 01 16:13:45.513523 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:45.513116 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22eae6cc-4752-4174-a0f0-573adc82e32b-workload-certs\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl\" (UID: \"22eae6cc-4752-4174-a0f0-573adc82e32b\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" Nov 01 16:13:45.513523 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:45.513158 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22eae6cc-4752-4174-a0f0-573adc82e32b-istiod-ca-cert\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl\" (UID: \"22eae6cc-4752-4174-a0f0-573adc82e32b\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" Nov 01 16:13:45.513523 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:45.513340 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22eae6cc-4752-4174-a0f0-573adc82e32b-istio-token\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl\" (UID: \"22eae6cc-4752-4174-a0f0-573adc82e32b\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" Nov 01 16:13:45.513523 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:45.513386 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22eae6cc-4752-4174-a0f0-573adc82e32b-istio-data\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl\" (UID: \"22eae6cc-4752-4174-a0f0-573adc82e32b\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" Nov 01 16:13:45.517041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:45.513908 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22eae6cc-4752-4174-a0f0-573adc82e32b-istio-data\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl\" (UID: \"22eae6cc-4752-4174-a0f0-573adc82e32b\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" Nov 01 16:13:45.517041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:45.514204 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22eae6cc-4752-4174-a0f0-573adc82e32b-workload-socket\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl\" (UID: \"22eae6cc-4752-4174-a0f0-573adc82e32b\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" Nov 01 16:13:45.517041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:45.514469 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22eae6cc-4752-4174-a0f0-573adc82e32b-workload-certs\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl\" (UID: \"22eae6cc-4752-4174-a0f0-573adc82e32b\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" Nov 01 16:13:45.517041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:45.515138 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22eae6cc-4752-4174-a0f0-573adc82e32b-istiod-ca-cert\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl\" (UID: \"22eae6cc-4752-4174-a0f0-573adc82e32b\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" Nov 01 16:13:45.517041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:45.515399 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22eae6cc-4752-4174-a0f0-573adc82e32b-credential-socket\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl\" (UID: \"22eae6cc-4752-4174-a0f0-573adc82e32b\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" Nov 01 16:13:45.541042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:45.540652 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22eae6cc-4752-4174-a0f0-573adc82e32b-istio-envoy\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl\" (UID: \"22eae6cc-4752-4174-a0f0-573adc82e32b\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" Nov 01 16:13:45.607041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:45.606851 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttxrr\" (UniqueName: \"kubernetes.io/projected/22eae6cc-4752-4174-a0f0-573adc82e32b-kube-api-access-ttxrr\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl\" (UID: \"22eae6cc-4752-4174-a0f0-573adc82e32b\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" Nov 01 16:13:45.627042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:45.607363 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22eae6cc-4752-4174-a0f0-573adc82e32b-istio-token\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl\" (UID: \"22eae6cc-4752-4174-a0f0-573adc82e32b\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" Nov 01 16:13:45.662037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:45.629926 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22eae6cc-4752-4174-a0f0-573adc82e32b-istio-podinfo\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl\" (UID: \"22eae6cc-4752-4174-a0f0-573adc82e32b\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" Nov 01 16:13:45.858450 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:45.858416 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" Nov 01 16:13:46.410601 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:46.410564 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl"] Nov 01 16:13:46.424052 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:46.423121 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:13:46.424052 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:46.423215 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:13:46.424052 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:46.423257 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:13:46.877509 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:46.877466 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" event={"ID":"22eae6cc-4752-4174-a0f0-573adc82e32b","Type":"ContainerStarted","Data":"9b3bbf5f34103ac74f32d032cca159897c7f64f6dff7ccea3bd6567ccf289c19"} Nov 01 16:13:46.877717 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:46.877538 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" event={"ID":"22eae6cc-4752-4174-a0f0-573adc82e32b","Type":"ContainerStarted","Data":"71e289895986bc067ad57e4014d4df29022d4a01dd7ed50ca9629758786d03fd"} Nov 01 16:13:46.916450 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:46.915978 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" podStartSLOduration=1.915958701 podStartE2EDuration="1.915958701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:13:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:13:46.905216108 +0000 UTC m=+1560.236953759" watchObservedRunningTime="2026-11-01 16:13:46.915958701 +0000 UTC m=+1560.247696352" Nov 01 16:13:47.064648 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:47.064311 4363 patch_prober.go:29] interesting pod/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:13:47.064648 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:47.064380 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" podUID="2d1193ea-2099-4ce6-b417-82d1e8e051b8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:13:47.874045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:47.868861 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" Nov 01 16:13:47.889382 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:47.889349 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" Nov 01 16:13:47.952933 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:47.952902 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" Nov 01 16:13:47.957133 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:47.957104 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" Nov 01 16:13:48.108385 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:48.108346 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" Nov 01 16:13:48.127126 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:48.127056 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" Nov 01 16:13:48.348371 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:13:48.348276 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:13:48.972361 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:48.972333 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" Nov 01 16:13:49.945907 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:49.945456 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" Nov 01 16:13:50.182041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:50.181662 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d1193ea-2099-4ce6-b417-82d1e8e051b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d1193ea-2099-4ce6-b417-82d1e8e051b8-workload-certs\") pod \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\" (UID: \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\") " Nov 01 16:13:50.182041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:50.181822 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d1193ea-2099-4ce6-b417-82d1e8e051b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d1193ea-2099-4ce6-b417-82d1e8e051b8-istio-data\") pod \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\" (UID: \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\") " Nov 01 16:13:50.182041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:50.181899 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d1193ea-2099-4ce6-b417-82d1e8e051b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1193ea-2099-4ce6-b417-82d1e8e051b8-workload-socket\") pod \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\" (UID: \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\") " Nov 01 16:13:50.182041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:50.181933 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d1193ea-2099-4ce6-b417-82d1e8e051b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1193ea-2099-4ce6-b417-82d1e8e051b8-credential-socket\") pod \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\" (UID: \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\") " Nov 01 16:13:50.182711 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:50.182051 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d1193ea-2099-4ce6-b417-82d1e8e051b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d1193ea-2099-4ce6-b417-82d1e8e051b8-istiod-ca-cert\") pod \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\" (UID: \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\") " Nov 01 16:13:50.182711 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:50.182236 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d1193ea-2099-4ce6-b417-82d1e8e051b8-kube-api-access-fv5l6\" (UniqueName: \"kubernetes.io/projected/2d1193ea-2099-4ce6-b417-82d1e8e051b8-kube-api-access-fv5l6\") pod \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\" (UID: \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\") " Nov 01 16:13:50.182711 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:50.182570 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d1193ea-2099-4ce6-b417-82d1e8e051b8-workload-certs" pod "2d1193ea-2099-4ce6-b417-82d1e8e051b8" (UID: "2d1193ea-2099-4ce6-b417-82d1e8e051b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:13:50.229126 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:50.183046 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d1193ea-2099-4ce6-b417-82d1e8e051b8-workload-socket" pod "2d1193ea-2099-4ce6-b417-82d1e8e051b8" (UID: "2d1193ea-2099-4ce6-b417-82d1e8e051b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:13:50.229126 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:50.183326 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d1193ea-2099-4ce6-b417-82d1e8e051b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d1193ea-2099-4ce6-b417-82d1e8e051b8-istio-envoy\") pod \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\" (UID: \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\") " Nov 01 16:13:50.229126 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:50.183480 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d1193ea-2099-4ce6-b417-82d1e8e051b8-istio-token\" (UniqueName: \"kubernetes.io/projected/2d1193ea-2099-4ce6-b417-82d1e8e051b8-istio-token\") pod \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\" (UID: \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\") " Nov 01 16:13:50.229126 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:50.183528 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d1193ea-2099-4ce6-b417-82d1e8e051b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d1193ea-2099-4ce6-b417-82d1e8e051b8-istio-podinfo\") pod \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\" (UID: \"2d1193ea-2099-4ce6-b417-82d1e8e051b8\") " Nov 01 16:13:50.229126 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:50.189524 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d1193ea-2099-4ce6-b417-82d1e8e051b8-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:50.229126 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:50.189551 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1193ea-2099-4ce6-b417-82d1e8e051b8-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:50.254040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:50.252224 4363 generic.go:356] "Generic (PLEG): container finished" podID="2d1193ea-2099-4ce6-b417-82d1e8e051b8" containerID="7fd2600eea08863a59ea3299fe596b1fe3fea4ebf8abf813d7685d22a2a8192a" exitCode=0 Nov 01 16:13:50.254040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:50.253593 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" Nov 01 16:13:50.254344 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:50.254087 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" event={"ID":"2d1193ea-2099-4ce6-b417-82d1e8e051b8","Type":"ContainerDied","Data":"7fd2600eea08863a59ea3299fe596b1fe3fea4ebf8abf813d7685d22a2a8192a"} Nov 01 16:13:50.254344 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:50.254145 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5" event={"ID":"2d1193ea-2099-4ce6-b417-82d1e8e051b8","Type":"ContainerDied","Data":"cf4e824c29457c671795dd89448c932cb48bad86e88df6d523d62f1b4067c472"} Nov 01 16:13:50.254344 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:50.254178 4363 scope.go:122] "RemoveContainer" containerID="7fd2600eea08863a59ea3299fe596b1fe3fea4ebf8abf813d7685d22a2a8192a" Nov 01 16:13:50.363067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:50.358277 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d1193ea-2099-4ce6-b417-82d1e8e051b8-istiod-ca-cert" pod "2d1193ea-2099-4ce6-b417-82d1e8e051b8" (UID: "2d1193ea-2099-4ce6-b417-82d1e8e051b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:13:50.408497 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:50.408460 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d1193ea-2099-4ce6-b417-82d1e8e051b8-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:50.503163 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:50.494154 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d1193ea-2099-4ce6-b417-82d1e8e051b8-credential-socket" pod "2d1193ea-2099-4ce6-b417-82d1e8e051b8" (UID: "2d1193ea-2099-4ce6-b417-82d1e8e051b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:13:50.551492 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:50.494613 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d1193ea-2099-4ce6-b417-82d1e8e051b8-istio-data" pod "2d1193ea-2099-4ce6-b417-82d1e8e051b8" (UID: "2d1193ea-2099-4ce6-b417-82d1e8e051b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:13:50.592964 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:50.589235 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:50.592964 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:50.592949 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d1193ea-2099-4ce6-b417-82d1e8e051b8-kube-api-access-fv5l6" pod "2d1193ea-2099-4ce6-b417-82d1e8e051b8" (UID: "2d1193ea-2099-4ce6-b417-82d1e8e051b8"). InnerVolumeSpecName "kube-api-access-fv5l6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:13:50.612292 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:50.609088 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d1193ea-2099-4ce6-b417-82d1e8e051b8-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:50.612540 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:50.612517 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1193ea-2099-4ce6-b417-82d1e8e051b8-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:50.614959 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:50.612634 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv5l6\" (UniqueName: \"kubernetes.io/projected/2d1193ea-2099-4ce6-b417-82d1e8e051b8-kube-api-access-fv5l6\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:50.634038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:50.633199 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" podUID="a15aa191-e376-490a-8763-16cdd7685b5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": context deadline exceeded" Nov 01 16:13:50.736368 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:50.736331 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d1193ea-2099-4ce6-b417-82d1e8e051b8-istio-envoy" pod "2d1193ea-2099-4ce6-b417-82d1e8e051b8" (UID: "2d1193ea-2099-4ce6-b417-82d1e8e051b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:13:50.756363 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:50.756333 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d1193ea-2099-4ce6-b417-82d1e8e051b8-istio-podinfo" pod "2d1193ea-2099-4ce6-b417-82d1e8e051b8" (UID: "2d1193ea-2099-4ce6-b417-82d1e8e051b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:13:50.802046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:50.797413 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d1193ea-2099-4ce6-b417-82d1e8e051b8-istio-token" pod "2d1193ea-2099-4ce6-b417-82d1e8e051b8" (UID: "2d1193ea-2099-4ce6-b417-82d1e8e051b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:13:50.845296 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:50.828274 4363 scope.go:122] "RemoveContainer" containerID="f2ce23c7cc7f14058bb2d2a272324caa25278b384841729f423191a077b7e365" Nov 01 16:13:50.897932 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:50.847314 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d1193ea-2099-4ce6-b417-82d1e8e051b8-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:50.897932 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:50.847351 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d1193ea-2099-4ce6-b417-82d1e8e051b8-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:50.897932 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:50.847372 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d1193ea-2099-4ce6-b417-82d1e8e051b8-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:50.971736 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:50.971454 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:51.171312 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:51.171267 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" podUID="d92f7196-153d-46b0-9e5b-51667329cd02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.169:8000/ready\": context deadline exceeded" Nov 01 16:13:51.235048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:51.216997 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:51.395044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:51.393923 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:51.407843 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:51.397995 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" podUID="37773d39-9d52-4b1d-94c7-68e02714a476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.171:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:51.683165 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:51.683117 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" podUID="a15aa191-e376-490a-8763-16cdd7685b5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": context deadline exceeded" Nov 01 16:13:51.746316 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:51.734614 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": context deadline exceeded" Nov 01 16:13:51.763104 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:51.751194 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 01 16:13:51.786672 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:51.743387 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a4240a-5f8bf4f8dd-4fpd6"] Nov 01 16:13:51.963577 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:51.943588 4363 cpu_manager.go:414] "containerMap: removing container" podUID="2d1193ea-2099-4ce6-b417-82d1e8e051b8" containerName="istio-proxy" Nov 01 16:13:51.964718 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:51.964684 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d1193ea-2099-4ce6-b417-82d1e8e051b8" containerName="istio-proxy" Nov 01 16:13:51.965216 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:51.965036 4363 cpu_manager.go:414] "containerMap: removing container" podUID="2d1193ea-2099-4ce6-b417-82d1e8e051b8" containerName="istio-proxy" Nov 01 16:13:52.033156 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:52.033126 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d1193ea-2099-4ce6-b417-82d1e8e051b8" containerName="istio-proxy" Nov 01 16:13:52.034649 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:52.034626 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d1193ea-2099-4ce6-b417-82d1e8e051b8" containerName="istio-proxy" Nov 01 16:13:52.100042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:52.099849 4363 scope.go:122] "RemoveContainer" containerID="7fd2600eea08863a59ea3299fe596b1fe3fea4ebf8abf813d7685d22a2a8192a" Nov 01 16:13:52.100042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:52.099909 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k" podUID="336c4cd8-6126-4431-930e-57f9b7ccf09d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:52.184500 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:52.183109 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" podUID="d92f7196-153d-46b0-9e5b-51667329cd02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.169:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:52.276736 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:52.269537 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:52.503038 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:13:52.502189 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fd2600eea08863a59ea3299fe596b1fe3fea4ebf8abf813d7685d22a2a8192a\": container with ID starting with 7fd2600eea08863a59ea3299fe596b1fe3fea4ebf8abf813d7685d22a2a8192a not found: ID does not exist" containerID="7fd2600eea08863a59ea3299fe596b1fe3fea4ebf8abf813d7685d22a2a8192a" Nov 01 16:13:52.503038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:52.502242 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fd2600eea08863a59ea3299fe596b1fe3fea4ebf8abf813d7685d22a2a8192a"} err="failed to get container status \"7fd2600eea08863a59ea3299fe596b1fe3fea4ebf8abf813d7685d22a2a8192a\": rpc error: code = NotFound desc = could not find container \"7fd2600eea08863a59ea3299fe596b1fe3fea4ebf8abf813d7685d22a2a8192a\": container with ID starting with 7fd2600eea08863a59ea3299fe596b1fe3fea4ebf8abf813d7685d22a2a8192a not found: ID does not exist" Nov 01 16:13:52.503038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:52.502269 4363 scope.go:122] "RemoveContainer" containerID="f2ce23c7cc7f14058bb2d2a272324caa25278b384841729f423191a077b7e365" Nov 01 16:13:52.511785 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:13:52.511746 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2ce23c7cc7f14058bb2d2a272324caa25278b384841729f423191a077b7e365\": container with ID starting with f2ce23c7cc7f14058bb2d2a272324caa25278b384841729f423191a077b7e365 not found: ID does not exist" containerID="f2ce23c7cc7f14058bb2d2a272324caa25278b384841729f423191a077b7e365" Nov 01 16:13:52.512047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:52.511997 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2ce23c7cc7f14058bb2d2a272324caa25278b384841729f423191a077b7e365"} err="failed to get container status \"f2ce23c7cc7f14058bb2d2a272324caa25278b384841729f423191a077b7e365\": rpc error: code = NotFound desc = could not find container \"f2ce23c7cc7f14058bb2d2a272324caa25278b384841729f423191a077b7e365\": container with ID starting with f2ce23c7cc7f14058bb2d2a272324caa25278b384841729f423191a077b7e365 not found: ID does not exist" Nov 01 16:13:52.512397 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:52.512319 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:52.535854 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:52.535734 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:13:52.575038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:52.563351 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8h5j2" podUID="14cafac8-b130-4780-aa9e-e60215541376" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": context deadline exceeded" Nov 01 16:13:52.607117 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:52.606996 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" Nov 01 16:13:52.607381 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:52.607363 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" Nov 01 16:13:52.607461 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:52.607398 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a4240a-5f8bf4f8dd-4fpd6"] Nov 01 16:13:52.607522 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:52.607480 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:13:52.653084 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:52.648698 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4240a-5f8bf4f8dd-4fpd6" Nov 01 16:13:52.685453 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:52.685414 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" podUID="a15aa191-e376-490a-8763-16cdd7685b5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:52.740192 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:52.740141 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" podUID="37773d39-9d52-4b1d-94c7-68e02714a476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.171:8000/ready\": context deadline exceeded" Nov 01 16:13:52.741319 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:52.741288 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" podUID="9cbd41ee-3134-4b54-9ea1-0c13921269bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.181:8000/ready\": context deadline exceeded" Nov 01 16:13:52.750340 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:52.750312 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" Nov 01 16:13:52.750583 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:52.750568 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" Nov 01 16:13:52.787101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:52.768332 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 01 16:13:52.788779 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:52.788739 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c-maas-api-tls\") pod \"maas-api-e2e-inf-a-a4240a-5f8bf4f8dd-4fpd6\" (UID: \"9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4240a-5f8bf4f8dd-4fpd6" Nov 01 16:13:52.805431 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:52.805393 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bllk9\" (UniqueName: \"kubernetes.io/projected/9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c-kube-api-access-bllk9\") pod \"maas-api-e2e-inf-a-a4240a-5f8bf4f8dd-4fpd6\" (UID: \"9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4240a-5f8bf4f8dd-4fpd6" Nov 01 16:13:52.805757 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:52.805736 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a4240a-5f8bf4f8dd-4fpd6\" (UID: \"9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4240a-5f8bf4f8dd-4fpd6" Nov 01 16:13:52.805978 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:52.805942 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c-metrics-certs\") pod \"maas-api-e2e-inf-a-a4240a-5f8bf4f8dd-4fpd6\" (UID: \"9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4240a-5f8bf4f8dd-4fpd6" Nov 01 16:13:52.806584 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:52.800212 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 16:13:52.835071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:52.833079 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:13:52.969206 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:52.969166 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a4240a-serving-cert\"" Nov 01 16:13:53.019094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:52.985195 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c-maas-api-tls\") pod \"maas-api-e2e-inf-a-a4240a-5f8bf4f8dd-4fpd6\" (UID: \"9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4240a-5f8bf4f8dd-4fpd6" Nov 01 16:13:53.042182 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:53.031686 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bllk9\" (UniqueName: \"kubernetes.io/projected/9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c-kube-api-access-bllk9\") pod \"maas-api-e2e-inf-a-a4240a-5f8bf4f8dd-4fpd6\" (UID: \"9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4240a-5f8bf4f8dd-4fpd6" Nov 01 16:13:53.042182 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:53.031888 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a4240a-5f8bf4f8dd-4fpd6\" (UID: \"9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4240a-5f8bf4f8dd-4fpd6" Nov 01 16:13:53.124189 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:53.067926 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a4240a-5f8bf4f8dd-4fpd6\" (UID: \"9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4240a-5f8bf4f8dd-4fpd6" Nov 01 16:13:53.157037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:53.069673 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c-metrics-certs\") pod \"maas-api-e2e-inf-a-a4240a-5f8bf4f8dd-4fpd6\" (UID: \"9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4240a-5f8bf4f8dd-4fpd6" Nov 01 16:13:53.321164 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:53.315968 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:13:53.321164 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:53.316350 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:13:53.362060 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:53.355599 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9e044c-668c79d887-78jg4"] Nov 01 16:13:53.496632 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:53.486675 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c-maas-api-tls\") pod \"maas-api-e2e-inf-a-a4240a-5f8bf4f8dd-4fpd6\" (UID: \"9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4240a-5f8bf4f8dd-4fpd6" Nov 01 16:13:53.566277 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:53.566225 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:53.639130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:53.602296 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bllk9\" (UniqueName: \"kubernetes.io/projected/9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c-kube-api-access-bllk9\") pod \"maas-api-e2e-inf-a-a4240a-5f8bf4f8dd-4fpd6\" (UID: \"9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4240a-5f8bf4f8dd-4fpd6" Nov 01 16:13:53.713448 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:53.692104 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" podUID="a15aa191-e376-490a-8763-16cdd7685b5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:53.757673 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:53.749601 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c-metrics-certs\") pod \"maas-api-e2e-inf-a-a4240a-5f8bf4f8dd-4fpd6\" (UID: \"9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4240a-5f8bf4f8dd-4fpd6" Nov 01 16:13:53.829127 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:53.761148 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" podUID="9cbd41ee-3134-4b54-9ea1-0c13921269bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.181:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:53.829127 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:53.761937 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" podUID="37773d39-9d52-4b1d-94c7-68e02714a476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.171:8000/ready\": context deadline exceeded" Nov 01 16:13:53.840039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:53.831904 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:53.933274 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:53.932203 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4240a-5f8bf4f8dd-4fpd6" Nov 01 16:13:54.011819 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:54.011783 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:13:54.060237 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:54.060208 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d1193ea-2099-4ce6-b417-82d1e8e051b8" containerName="istio-proxy" Nov 01 16:13:54.118668 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:54.099284 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" podUID="d92f7196-153d-46b0-9e5b-51667329cd02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.169:8000/ready\": context deadline exceeded" Nov 01 16:13:54.145119 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:54.143320 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 16:13:54.145119 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:54.143446 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9e044c-668c79d887-78jg4"] Nov 01 16:13:54.145119 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:54.143488 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5"] Nov 01 16:13:54.145119 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:54.143512 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-26310f-openshift-default-85bb76567f-kw9m5"] Nov 01 16:13:54.145119 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:54.143573 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:13:54.145119 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:54.143622 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" Nov 01 16:13:54.145119 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:54.143657 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:13:54.145119 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:54.144997 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e044c-668c79d887-78jg4" Nov 01 16:13:54.161040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:54.150914 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-9e044c-dockercfg-b44jc\"" Nov 01 16:13:54.242286 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:54.242262 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-9e044c\"" Nov 01 16:13:54.358042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:54.329315 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7lj4\" (UniqueName: \"kubernetes.io/projected/79acdb79-6330-4958-a12d-d42d9d5c8d4e-kube-api-access-x7lj4\") pod \"payload-pre-processing-e2e-inf-a-9e044c-668c79d887-78jg4\" (UID: \"79acdb79-6330-4958-a12d-d42d9d5c8d4e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e044c-668c79d887-78jg4" Nov 01 16:13:54.373592 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:54.373558 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79acdb79-6330-4958-a12d-d42d9d5c8d4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9e044c-668c79d887-78jg4\" (UID: \"79acdb79-6330-4958-a12d-d42d9d5c8d4e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e044c-668c79d887-78jg4" Nov 01 16:13:54.374099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:54.363590 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:13:54.374863 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:54.363557 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded" Nov 01 16:13:54.375578 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:54.363629 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:13:54.375998 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:54.363611 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:13:54.386903 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:54.386879 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:13:54.657295 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:54.657262 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 16:13:54.763192 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:54.763143 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" podUID="37773d39-9d52-4b1d-94c7-68e02714a476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.171:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:54.817045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:54.788426 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" podUID="a15aa191-e376-490a-8763-16cdd7685b5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": context deadline exceeded" Nov 01 16:13:54.822259 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:54.822226 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7lj4\" (UniqueName: \"kubernetes.io/projected/79acdb79-6330-4958-a12d-d42d9d5c8d4e-kube-api-access-x7lj4\") pod \"payload-pre-processing-e2e-inf-a-9e044c-668c79d887-78jg4\" (UID: \"79acdb79-6330-4958-a12d-d42d9d5c8d4e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e044c-668c79d887-78jg4" Nov 01 16:13:54.826255 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:54.826213 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79acdb79-6330-4958-a12d-d42d9d5c8d4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9e044c-668c79d887-78jg4\" (UID: \"79acdb79-6330-4958-a12d-d42d9d5c8d4e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e044c-668c79d887-78jg4" Nov 01 16:13:54.860043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:54.831208 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79acdb79-6330-4958-a12d-d42d9d5c8d4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9e044c-668c79d887-78jg4\" (UID: \"79acdb79-6330-4958-a12d-d42d9d5c8d4e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e044c-668c79d887-78jg4" Nov 01 16:13:54.887884 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:54.887742 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" podUID="9cbd41ee-3134-4b54-9ea1-0c13921269bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.181:8000/ready\": context deadline exceeded" Nov 01 16:13:55.066582 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:55.041249 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": context deadline exceeded" Nov 01 16:13:55.201095 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:55.066864 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:13:55.201861 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:55.201766 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7lj4\" (UniqueName: \"kubernetes.io/projected/79acdb79-6330-4958-a12d-d42d9d5c8d4e-kube-api-access-x7lj4\") pod \"payload-pre-processing-e2e-inf-a-9e044c-668c79d887-78jg4\" (UID: \"79acdb79-6330-4958-a12d-d42d9d5c8d4e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e044c-668c79d887-78jg4" Nov 01 16:13:55.376040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:55.369295 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e044c-668c79d887-78jg4" Nov 01 16:13:55.405039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:55.402955 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:13:55.430037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:55.373116 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:13:55.530041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:55.377207 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:13:55.559485 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:55.559435 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:55.670164 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:55.667772 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 01 16:13:55.879047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:55.867448 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" podUID="a15aa191-e376-490a-8763-16cdd7685b5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": context deadline exceeded" Nov 01 16:13:55.941094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:55.916393 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" podUID="9cbd41ee-3134-4b54-9ea1-0c13921269bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.181:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:55.941094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:55.923363 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-frphn" podUID="fa3886b7-e5e1-4f64-93d4-a097f446a835" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:55.956107 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:55.956071 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d1193ea-2099-4ce6-b417-82d1e8e051b8" path="/var/lib/kubelet/pods/2d1193ea-2099-4ce6-b417-82d1e8e051b8/volumes" Nov 01 16:13:55.956992 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:13:55.956965 4363 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.161s" Nov 01 16:13:55.957344 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:55.957327 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:13:56.050043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:56.023577 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:13:56.134679 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:56.134626 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" podUID="37773d39-9d52-4b1d-94c7-68e02714a476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.171:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:56.231099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:56.215232 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:56.331044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:56.291784 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" podUID="d92f7196-153d-46b0-9e5b-51667329cd02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.169:8000/ready\": context deadline exceeded" Nov 01 16:13:56.467345 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:56.467304 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:56.474074 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:56.474039 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:13:56.865059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:56.840114 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": context deadline exceeded" Nov 01 16:13:56.917042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:56.881931 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:13:56.961365 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:56.961255 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:13:57.024042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:56.966636 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:57.165650 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:56.967793 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" Nov 01 16:13:57.165979 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:57.165956 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 16:13:57.289692 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:57.165810 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" podUID="9cbd41ee-3134-4b54-9ea1-0c13921269bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.181:8000/ready\": context deadline exceeded" Nov 01 16:13:57.410398 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:57.410358 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" podUID="d92f7196-153d-46b0-9e5b-51667329cd02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.169:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:57.734673 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:57.734606 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" podUID="37773d39-9d52-4b1d-94c7-68e02714a476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.171:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:58.012801 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:58.012723 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:58.013406 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:58.013376 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:13:58.146088 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:58.138643 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" podUID="a15aa191-e376-490a-8763-16cdd7685b5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": context deadline exceeded" Nov 01 16:13:58.151173 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:58.151130 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:58.179041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:58.178675 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 01 16:13:58.399215 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:57.472787 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:13:58.413065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:57.472718 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:58.505036 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:58.504158 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-55fbbbf9dc-l5qnm" podUID="dae26a45-6a59-4c32-a565-704e50c5f271" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Nov 01 16:13:58.585039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:58.581499 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr" podUID="305de5fb-52ed-401a-89fe-b7d802472f5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:58.585039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:58.582575 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8h5j2" podUID="14cafac8-b130-4780-aa9e-e60215541376" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded" Nov 01 16:13:58.585039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:58.582711 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k" podUID="336c4cd8-6126-4431-930e-57f9b7ccf09d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": context deadline exceeded" Nov 01 16:13:58.702124 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:58.689628 4363 patch_prober.go:29] interesting pod/thanos-querier-777b758c86-h4jsj container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.14:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:13:58.702124 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:58.689695 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-777b758c86-h4jsj" podUID="88d3d141-d4fa-4033-a3cb-372752fa2c67" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.14:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:58.822161 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:58.822111 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c65e3-77686b98cf-wzm9h" podUID="06fcb8d1-bff1-4182-92d2-98a57314f6fd" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Nov 01 16:13:58.903095 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:58.858101 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" podUID="d92f7196-153d-46b0-9e5b-51667329cd02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.169:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:59.021281 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:59.021178 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:59.074042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:59.038380 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:13:59.198683 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:59.185183 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:59.213180 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:59.213135 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" podUID="37773d39-9d52-4b1d-94c7-68e02714a476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.171:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:59.257080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:59.231658 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" podUID="a15aa191-e376-490a-8763-16cdd7685b5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:59.299897 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:59.299766 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:13:59.331045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:59.326210 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" podUID="9cbd41ee-3134-4b54-9ea1-0c13921269bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.181:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:59.394040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:59.362098 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" podUID="9cbd41ee-3134-4b54-9ea1-0c13921269bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.181:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:59.410969 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:59.410921 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:59.434906 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:59.411563 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:59.588039 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:13:59.587930 4363 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.422s" Nov 01 16:13:59.614030 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:13:59.613987 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" Nov 01 16:14:00.059139 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:00.043612 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:14:00.165124 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:00.156946 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:14:00.259451 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:00.259401 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" podUID="37773d39-9d52-4b1d-94c7-68e02714a476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.171:8000/ready\": context deadline exceeded" Nov 01 16:14:00.334906 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:00.334805 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4240a-5f8bf4f8dd-4fpd6" event={"ID":"9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c","Type":"ContainerStarted","Data":"89e6b8165b0ba1e523a6dfb8f13a0633fb921f2ad392fb5287f5fcbf17c05c54"} Nov 01 16:14:00.335126 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:00.334951 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" Nov 01 16:14:00.335126 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:00.334987 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e044c-668c79d887-78jg4" event={"ID":"79acdb79-6330-4958-a12d-d42d9d5c8d4e","Type":"ContainerStarted","Data":"c5723460f035dcd5c55693c981dd5f32923a1fa1150ec2c4a9e6516510286742"} Nov 01 16:14:00.335276 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:00.335168 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:14:00.335276 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:00.335213 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" Nov 01 16:14:00.335276 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:00.335246 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 16:14:00.335432 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:00.335296 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" Nov 01 16:14:00.335432 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:00.335330 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:14:00.335432 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:00.335391 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:14:00.335581 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:00.335441 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:14:00.335581 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:00.335474 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" Nov 01 16:14:00.335581 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:00.335525 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 16:14:00.335581 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:00.335558 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:14:00.335773 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:00.335602 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a4240a-5f8bf4f8dd-4fpd6"] Nov 01 16:14:00.335773 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:00.335639 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9e044c-668c79d887-78jg4"] Nov 01 16:14:00.344295 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:00.344263 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" Nov 01 16:14:00.659293 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:00.658062 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4240a-5f8bf4f8dd-4fpd6" event={"ID":"9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c","Type":"ContainerStarted","Data":"10599dd59e2a6cff1983ba4b08d7492202c82e14221657e10559900129454bef"} Nov 01 16:14:00.659293 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:00.659174 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4240a-5f8bf4f8dd-4fpd6" Nov 01 16:14:00.667316 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:00.665590 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e044c-668c79d887-78jg4" event={"ID":"79acdb79-6330-4958-a12d-d42d9d5c8d4e","Type":"ContainerStarted","Data":"95e8035c38cc902452b8d1cf36881d5003e298ef2e9fc05a55fe25f66db2073e"} Nov 01 16:14:00.667550 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:00.667531 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e044c-668c79d887-78jg4" Nov 01 16:14:00.736303 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:00.736192 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4240a-5f8bf4f8dd-4fpd6" podStartSLOduration=10.861806542 podStartE2EDuration="11.736174564s" totalImagesPullingTime="874.368022ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:13:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:14:00.731685705 +0000 UTC m=+1574.063423357" watchObservedRunningTime="2026-11-01 16:14:00.736174564 +0000 UTC m=+1574.067912219" Nov 01 16:14:00.761591 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:00.761475 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e044c-668c79d887-78jg4" podStartSLOduration=10.761456173 podStartE2EDuration="10.761456173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:13:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:14:00.75854856 +0000 UTC m=+1574.090286209" watchObservedRunningTime="2026-11-01 16:14:00.761456173 +0000 UTC m=+1574.093193815" Nov 01 16:14:05.406224 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:05.406181 4363 patch_prober.go:29] interesting pod/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:14:05.406843 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:05.406798 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" podUID="b6716c2c-b539-4e3c-ac09-7f2d1426af74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:14:06.415176 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:06.415143 4363 patch_prober.go:29] interesting pod/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:14:08.233229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:06.415203 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" podUID="ebf773ee-489a-418d-a031-327a467839c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:14:08.233229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:07.320875 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-99e837/test-model-99e837-kserve-68448d5ff7-hfsbc"] Nov 01 16:14:08.233229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:07.339299 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-99e837/test-model-99e837-kserve-68448d5ff7-hfsbc"] Nov 01 16:14:08.233229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:07.339406 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-99e837/test-model-99e837-kserve-68448d5ff7-hfsbc" Nov 01 16:14:08.233229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:07.349661 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-99e837\"/\"default-dockercfg-dhfqg\"" Nov 01 16:14:08.233229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:07.350364 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-99e837\"/\"kube-root-ca.crt\"" Nov 01 16:14:08.233229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:07.350430 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-99e837\"/\"test-model-99e837-kserve-self-signed-certs\"" Nov 01 16:14:08.233229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:07.350456 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-99e837\"/\"openshift-service-ca.crt\"" Nov 01 16:14:08.233229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:07.409131 4363 patch_prober.go:29] interesting pod/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:14:08.233229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:07.409174 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" podUID="388de32a-ab53-48d9-983d-ce302916e7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:14:08.233229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:07.504400 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62fde750-084e-488b-9d65-24b38a323cb0-dshm\") pod \"test-model-99e837-kserve-68448d5ff7-hfsbc\" (UID: \"62fde750-084e-488b-9d65-24b38a323cb0\") " pod="ai-tenant-e2e-inf-b-99e837/test-model-99e837-kserve-68448d5ff7-hfsbc" Nov 01 16:14:08.233229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:07.504487 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62fde750-084e-488b-9d65-24b38a323cb0-model-cache\") pod \"test-model-99e837-kserve-68448d5ff7-hfsbc\" (UID: \"62fde750-084e-488b-9d65-24b38a323cb0\") " pod="ai-tenant-e2e-inf-b-99e837/test-model-99e837-kserve-68448d5ff7-hfsbc" Nov 01 16:14:08.233229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:07.504525 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktnn4\" (UniqueName: \"kubernetes.io/projected/62fde750-084e-488b-9d65-24b38a323cb0-kube-api-access-ktnn4\") pod \"test-model-99e837-kserve-68448d5ff7-hfsbc\" (UID: \"62fde750-084e-488b-9d65-24b38a323cb0\") " pod="ai-tenant-e2e-inf-b-99e837/test-model-99e837-kserve-68448d5ff7-hfsbc" Nov 01 16:14:08.233229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:07.504717 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62fde750-084e-488b-9d65-24b38a323cb0-tls-certs\") pod \"test-model-99e837-kserve-68448d5ff7-hfsbc\" (UID: \"62fde750-084e-488b-9d65-24b38a323cb0\") " pod="ai-tenant-e2e-inf-b-99e837/test-model-99e837-kserve-68448d5ff7-hfsbc" Nov 01 16:14:08.233229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:07.504886 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62fde750-084e-488b-9d65-24b38a323cb0-home\") pod \"test-model-99e837-kserve-68448d5ff7-hfsbc\" (UID: \"62fde750-084e-488b-9d65-24b38a323cb0\") " pod="ai-tenant-e2e-inf-b-99e837/test-model-99e837-kserve-68448d5ff7-hfsbc" Nov 01 16:14:08.233229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:07.504942 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62fde750-084e-488b-9d65-24b38a323cb0-tmp-dir\") pod \"test-model-99e837-kserve-68448d5ff7-hfsbc\" (UID: \"62fde750-084e-488b-9d65-24b38a323cb0\") " pod="ai-tenant-e2e-inf-b-99e837/test-model-99e837-kserve-68448d5ff7-hfsbc" Nov 01 16:14:08.233229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:07.633437 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62fde750-084e-488b-9d65-24b38a323cb0-dshm\") pod \"test-model-99e837-kserve-68448d5ff7-hfsbc\" (UID: \"62fde750-084e-488b-9d65-24b38a323cb0\") " pod="ai-tenant-e2e-inf-b-99e837/test-model-99e837-kserve-68448d5ff7-hfsbc" Nov 01 16:14:08.233229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:07.633530 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62fde750-084e-488b-9d65-24b38a323cb0-model-cache\") pod \"test-model-99e837-kserve-68448d5ff7-hfsbc\" (UID: \"62fde750-084e-488b-9d65-24b38a323cb0\") " pod="ai-tenant-e2e-inf-b-99e837/test-model-99e837-kserve-68448d5ff7-hfsbc" Nov 01 16:14:08.233229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:07.633570 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktnn4\" (UniqueName: \"kubernetes.io/projected/62fde750-084e-488b-9d65-24b38a323cb0-kube-api-access-ktnn4\") pod \"test-model-99e837-kserve-68448d5ff7-hfsbc\" (UID: \"62fde750-084e-488b-9d65-24b38a323cb0\") " pod="ai-tenant-e2e-inf-b-99e837/test-model-99e837-kserve-68448d5ff7-hfsbc" Nov 01 16:14:08.233229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:07.633786 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62fde750-084e-488b-9d65-24b38a323cb0-tls-certs\") pod \"test-model-99e837-kserve-68448d5ff7-hfsbc\" (UID: \"62fde750-084e-488b-9d65-24b38a323cb0\") " pod="ai-tenant-e2e-inf-b-99e837/test-model-99e837-kserve-68448d5ff7-hfsbc" Nov 01 16:14:08.233229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:07.633971 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62fde750-084e-488b-9d65-24b38a323cb0-home\") pod \"test-model-99e837-kserve-68448d5ff7-hfsbc\" (UID: \"62fde750-084e-488b-9d65-24b38a323cb0\") " pod="ai-tenant-e2e-inf-b-99e837/test-model-99e837-kserve-68448d5ff7-hfsbc" Nov 01 16:14:08.233229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:07.634142 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62fde750-084e-488b-9d65-24b38a323cb0-tmp-dir\") pod \"test-model-99e837-kserve-68448d5ff7-hfsbc\" (UID: \"62fde750-084e-488b-9d65-24b38a323cb0\") " pod="ai-tenant-e2e-inf-b-99e837/test-model-99e837-kserve-68448d5ff7-hfsbc" Nov 01 16:14:08.233229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:07.634774 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62fde750-084e-488b-9d65-24b38a323cb0-tmp-dir\") pod \"test-model-99e837-kserve-68448d5ff7-hfsbc\" (UID: \"62fde750-084e-488b-9d65-24b38a323cb0\") " pod="ai-tenant-e2e-inf-b-99e837/test-model-99e837-kserve-68448d5ff7-hfsbc" Nov 01 16:14:08.233229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:07.635581 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62fde750-084e-488b-9d65-24b38a323cb0-model-cache\") pod \"test-model-99e837-kserve-68448d5ff7-hfsbc\" (UID: \"62fde750-084e-488b-9d65-24b38a323cb0\") " pod="ai-tenant-e2e-inf-b-99e837/test-model-99e837-kserve-68448d5ff7-hfsbc" Nov 01 16:14:08.233229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:07.636269 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62fde750-084e-488b-9d65-24b38a323cb0-home\") pod \"test-model-99e837-kserve-68448d5ff7-hfsbc\" (UID: \"62fde750-084e-488b-9d65-24b38a323cb0\") " pod="ai-tenant-e2e-inf-b-99e837/test-model-99e837-kserve-68448d5ff7-hfsbc" Nov 01 16:14:08.233229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:07.686730 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62fde750-084e-488b-9d65-24b38a323cb0-tls-certs\") pod \"test-model-99e837-kserve-68448d5ff7-hfsbc\" (UID: \"62fde750-084e-488b-9d65-24b38a323cb0\") " pod="ai-tenant-e2e-inf-b-99e837/test-model-99e837-kserve-68448d5ff7-hfsbc" Nov 01 16:14:08.233229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:07.688791 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62fde750-084e-488b-9d65-24b38a323cb0-dshm\") pod \"test-model-99e837-kserve-68448d5ff7-hfsbc\" (UID: \"62fde750-084e-488b-9d65-24b38a323cb0\") " pod="ai-tenant-e2e-inf-b-99e837/test-model-99e837-kserve-68448d5ff7-hfsbc" Nov 01 16:14:08.233229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:07.688989 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktnn4\" (UniqueName: \"kubernetes.io/projected/62fde750-084e-488b-9d65-24b38a323cb0-kube-api-access-ktnn4\") pod \"test-model-99e837-kserve-68448d5ff7-hfsbc\" (UID: \"62fde750-084e-488b-9d65-24b38a323cb0\") " pod="ai-tenant-e2e-inf-b-99e837/test-model-99e837-kserve-68448d5ff7-hfsbc" Nov 01 16:14:08.233229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:07.689148 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-99e837/test-model-99e837-kserve-68448d5ff7-hfsbc" Nov 01 16:14:08.233229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:07.708339 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4240a-5f8bf4f8dd-4fpd6" Nov 01 16:14:08.322288 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:08.322187 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-99e837/test-model-99e837-kserve-68448d5ff7-hfsbc"] Nov 01 16:14:08.338165 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:14:08.338135 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62fde750_084e_488b_9d65_24b38a323cb0.slice/crio-dbcd73ccc58809ce57365a88b9581ab3ab459a046206dfbf5b167f3afcfa2b32 WatchSource:0}: Error finding container dbcd73ccc58809ce57365a88b9581ab3ab459a046206dfbf5b167f3afcfa2b32: Status 404 returned error can't find the container with id dbcd73ccc58809ce57365a88b9581ab3ab459a046206dfbf5b167f3afcfa2b32 Nov 01 16:14:08.762384 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:08.762345 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-99e837/test-model-99e837-kserve-68448d5ff7-hfsbc" event={"ID":"62fde750-084e-488b-9d65-24b38a323cb0","Type":"ContainerStarted","Data":"6de671ef5553e04c15a2ef15397dadd2c9b689d54a98fc913b176186551708e3"} Nov 01 16:14:08.762580 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:08.762399 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-99e837/test-model-99e837-kserve-68448d5ff7-hfsbc" event={"ID":"62fde750-084e-488b-9d65-24b38a323cb0","Type":"ContainerStarted","Data":"dbcd73ccc58809ce57365a88b9581ab3ab459a046206dfbf5b167f3afcfa2b32"} Nov 01 16:14:08.764377 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:08.764354 4363 generic.go:356] "Generic (PLEG): container finished" podID="388de32a-ab53-48d9-983d-ce302916e7a8" containerID="4781f143604d4c1c79729186e3dd8913f8dd19537443e0fa4c9bba5a1c4a0081" exitCode=0 Nov 01 16:14:08.764498 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:08.764431 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" event={"ID":"388de32a-ab53-48d9-983d-ce302916e7a8","Type":"ContainerDied","Data":"4781f143604d4c1c79729186e3dd8913f8dd19537443e0fa4c9bba5a1c4a0081"} Nov 01 16:14:08.765071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:08.765010 4363 scope.go:122] "RemoveContainer" containerID="4781f143604d4c1c79729186e3dd8913f8dd19537443e0fa4c9bba5a1c4a0081" Nov 01 16:14:08.766869 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:08.766849 4363 generic.go:356] "Generic (PLEG): container finished" podID="b6716c2c-b539-4e3c-ac09-7f2d1426af74" containerID="95dd1549927c9b22958db180914ee1cfa32ed7449cd317694cbacfd7ff364525" exitCode=0 Nov 01 16:14:08.766965 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:08.766932 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" event={"ID":"b6716c2c-b539-4e3c-ac09-7f2d1426af74","Type":"ContainerDied","Data":"95dd1549927c9b22958db180914ee1cfa32ed7449cd317694cbacfd7ff364525"} Nov 01 16:14:08.767436 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:08.767407 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:14:08.767534 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:08.767445 4363 scope.go:122] "RemoveContainer" containerID="95dd1549927c9b22958db180914ee1cfa32ed7449cd317694cbacfd7ff364525" Nov 01 16:14:08.767534 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:08.767490 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:14:08.767625 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:08.767539 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:14:08.768760 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:08.768741 4363 generic.go:356] "Generic (PLEG): container finished" podID="ebf773ee-489a-418d-a031-327a467839c4" containerID="1fb51714c82041bd241aadebc9f1707c9825ad779c38ebfec5ffb437a84e719c" exitCode=0 Nov 01 16:14:08.768839 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:08.768772 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" event={"ID":"ebf773ee-489a-418d-a031-327a467839c4","Type":"ContainerDied","Data":"1fb51714c82041bd241aadebc9f1707c9825ad779c38ebfec5ffb437a84e719c"} Nov 01 16:14:08.769196 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:08.769178 4363 scope.go:122] "RemoveContainer" containerID="1fb51714c82041bd241aadebc9f1707c9825ad779c38ebfec5ffb437a84e719c" Nov 01 16:14:08.769580 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:08.769551 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:14:08.769656 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:08.769633 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:14:08.769718 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:08.769683 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:14:08.771709 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:08.771683 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:14:08.771794 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:08.771761 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:14:08.771857 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:08.771810 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:14:08.783289 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:08.783238 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-99e837/test-model-99e837-kserve-68448d5ff7-hfsbc" podStartSLOduration=1.78322212 podStartE2EDuration="1.78322212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:14:08.779590099 +0000 UTC m=+1582.111327741" watchObservedRunningTime="2026-11-01 16:14:08.78322212 +0000 UTC m=+1582.114959774" Nov 01 16:14:09.696128 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:09.690110 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-99e837/test-model-99e837-kserve-68448d5ff7-hfsbc" Nov 01 16:14:09.783382 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:09.783306 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" event={"ID":"ebf773ee-489a-418d-a031-327a467839c4","Type":"ContainerStarted","Data":"2fcdcc811dd8df484b2271e68fae197a6f7050a95ace4fb255c1c5ff0216ad64"} Nov 01 16:14:09.785673 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:09.785636 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" event={"ID":"388de32a-ab53-48d9-983d-ce302916e7a8","Type":"ContainerStarted","Data":"d46c601161a716fbd254be500daa4c9ce6687747c8f52b2941278d1d8c6dae07"} Nov 01 16:14:09.788232 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:09.788202 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" event={"ID":"b6716c2c-b539-4e3c-ac09-7f2d1426af74","Type":"ContainerStarted","Data":"992c76fb0b84fab448f1f2f44a73f0b15590dd6a7359a3a7dd6c038e52feacce"} Nov 01 16:14:10.274113 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:10.261154 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" Nov 01 16:14:10.274113 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:10.261281 4363 patch_prober.go:29] interesting pod/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 01 16:14:10.274113 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:10.261328 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" podUID="388de32a-ab53-48d9-983d-ce302916e7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 01 16:14:10.765371 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:10.765334 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" Nov 01 16:14:10.781177 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:10.769205 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" Nov 01 16:14:10.781465 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:10.775140 4363 patch_prober.go:29] interesting pod/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Nov 01 16:14:10.781648 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:10.781623 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" podUID="ebf773ee-489a-418d-a031-327a467839c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Nov 01 16:14:11.513046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:10.885115 4363 patch_prober.go:29] interesting pod/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Nov 01 16:14:11.520169 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:11.520109 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" podUID="b6716c2c-b539-4e3c-ac09-7f2d1426af74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Nov 01 16:14:11.520992 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:11.236843 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" podUID="d92f7196-153d-46b0-9e5b-51667329cd02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.169:8000/ready\": context deadline exceeded" Nov 01 16:14:11.552680 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:11.259467 4363 patch_prober.go:29] interesting pod/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 01 16:14:11.552933 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:11.552903 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" podUID="388de32a-ab53-48d9-983d-ce302916e7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 01 16:14:11.553062 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:11.379117 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:14:11.554481 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:11.278664 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 01 16:14:11.555707 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:11.334072 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:14:11.562441 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:11.461224 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" podUID="a15aa191-e376-490a-8763-16cdd7685b5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:14:11.562625 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:11.397095 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:14:11.562753 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:11.308358 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:14:11.784869 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:11.784773 4363 patch_prober.go:29] interesting pod/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Nov 01 16:14:11.785657 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:11.785625 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" podUID="ebf773ee-489a-418d-a031-327a467839c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Nov 01 16:14:11.786226 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:11.784978 4363 patch_prober.go:29] interesting pod/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Nov 01 16:14:11.786385 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:11.786365 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" podUID="b6716c2c-b539-4e3c-ac09-7f2d1426af74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Nov 01 16:14:11.879220 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:14:11.840511 4363 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.027s" Nov 01 16:14:11.879220 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:11.840676 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e044c-668c79d887-78jg4" Nov 01 16:14:11.879220 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:11.840707 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-04f248/test-model-04f248-kserve-7cf9d4bf7f-kd4rz"] Nov 01 16:14:12.052101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:12.051153 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:14:12.091292 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:12.091259 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-04f248/test-model-04f248-kserve-7cf9d4bf7f-kd4rz"] Nov 01 16:14:12.092308 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:12.092283 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-04f248/test-model-04f248-kserve-7cf9d4bf7f-kd4rz" Nov 01 16:14:12.118544 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:12.118513 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-04f248\"/\"test-model-04f248-kserve-self-signed-certs\"" Nov 01 16:14:12.118896 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:12.118874 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-04f248\"/\"openshift-service-ca.crt\"" Nov 01 16:14:12.119160 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:12.119144 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-04f248\"/\"kube-root-ca.crt\"" Nov 01 16:14:12.119470 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:12.119454 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-04f248\"/\"default-dockercfg-rpbf9\"" Nov 01 16:14:12.136231 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:12.136191 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:14:12.142040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:12.139286 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" podUID="a15aa191-e376-490a-8763-16cdd7685b5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:14:12.186045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:12.171861 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3860672d-226a-47fd-a812-cca575bea518-dshm\") pod \"test-model-04f248-kserve-7cf9d4bf7f-kd4rz\" (UID: \"3860672d-226a-47fd-a812-cca575bea518\") " pod="ai-tenant-e2e-inf-a-04f248/test-model-04f248-kserve-7cf9d4bf7f-kd4rz" Nov 01 16:14:12.186045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:12.173061 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3860672d-226a-47fd-a812-cca575bea518-tmp-dir\") pod \"test-model-04f248-kserve-7cf9d4bf7f-kd4rz\" (UID: \"3860672d-226a-47fd-a812-cca575bea518\") " pod="ai-tenant-e2e-inf-a-04f248/test-model-04f248-kserve-7cf9d4bf7f-kd4rz" Nov 01 16:14:12.210043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:12.200220 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3860672d-226a-47fd-a812-cca575bea518-tls-certs\") pod \"test-model-04f248-kserve-7cf9d4bf7f-kd4rz\" (UID: \"3860672d-226a-47fd-a812-cca575bea518\") " pod="ai-tenant-e2e-inf-a-04f248/test-model-04f248-kserve-7cf9d4bf7f-kd4rz" Nov 01 16:14:12.210043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:12.200575 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3860672d-226a-47fd-a812-cca575bea518-model-cache\") pod \"test-model-04f248-kserve-7cf9d4bf7f-kd4rz\" (UID: \"3860672d-226a-47fd-a812-cca575bea518\") " pod="ai-tenant-e2e-inf-a-04f248/test-model-04f248-kserve-7cf9d4bf7f-kd4rz" Nov 01 16:14:12.210043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:12.200652 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3860672d-226a-47fd-a812-cca575bea518-home\") pod \"test-model-04f248-kserve-7cf9d4bf7f-kd4rz\" (UID: \"3860672d-226a-47fd-a812-cca575bea518\") " pod="ai-tenant-e2e-inf-a-04f248/test-model-04f248-kserve-7cf9d4bf7f-kd4rz" Nov 01 16:14:12.210043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:12.200886 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5x9z\" (UniqueName: \"kubernetes.io/projected/3860672d-226a-47fd-a812-cca575bea518-kube-api-access-t5x9z\") pod \"test-model-04f248-kserve-7cf9d4bf7f-kd4rz\" (UID: \"3860672d-226a-47fd-a812-cca575bea518\") " pod="ai-tenant-e2e-inf-a-04f248/test-model-04f248-kserve-7cf9d4bf7f-kd4rz" Nov 01 16:14:12.285248 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:12.261873 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" podUID="9cbd41ee-3134-4b54-9ea1-0c13921269bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.181:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:14:12.374080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:12.357853 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5x9z\" (UniqueName: \"kubernetes.io/projected/3860672d-226a-47fd-a812-cca575bea518-kube-api-access-t5x9z\") pod \"test-model-04f248-kserve-7cf9d4bf7f-kd4rz\" (UID: \"3860672d-226a-47fd-a812-cca575bea518\") " pod="ai-tenant-e2e-inf-a-04f248/test-model-04f248-kserve-7cf9d4bf7f-kd4rz" Nov 01 16:14:12.374080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:12.358127 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3860672d-226a-47fd-a812-cca575bea518-dshm\") pod \"test-model-04f248-kserve-7cf9d4bf7f-kd4rz\" (UID: \"3860672d-226a-47fd-a812-cca575bea518\") " pod="ai-tenant-e2e-inf-a-04f248/test-model-04f248-kserve-7cf9d4bf7f-kd4rz" Nov 01 16:14:12.374080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:12.358178 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3860672d-226a-47fd-a812-cca575bea518-tmp-dir\") pod \"test-model-04f248-kserve-7cf9d4bf7f-kd4rz\" (UID: \"3860672d-226a-47fd-a812-cca575bea518\") " pod="ai-tenant-e2e-inf-a-04f248/test-model-04f248-kserve-7cf9d4bf7f-kd4rz" Nov 01 16:14:12.374080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:12.358286 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3860672d-226a-47fd-a812-cca575bea518-tls-certs\") pod \"test-model-04f248-kserve-7cf9d4bf7f-kd4rz\" (UID: \"3860672d-226a-47fd-a812-cca575bea518\") " pod="ai-tenant-e2e-inf-a-04f248/test-model-04f248-kserve-7cf9d4bf7f-kd4rz" Nov 01 16:14:12.374080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:12.358459 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3860672d-226a-47fd-a812-cca575bea518-model-cache\") pod \"test-model-04f248-kserve-7cf9d4bf7f-kd4rz\" (UID: \"3860672d-226a-47fd-a812-cca575bea518\") " pod="ai-tenant-e2e-inf-a-04f248/test-model-04f248-kserve-7cf9d4bf7f-kd4rz" Nov 01 16:14:12.374080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:12.358507 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3860672d-226a-47fd-a812-cca575bea518-home\") pod \"test-model-04f248-kserve-7cf9d4bf7f-kd4rz\" (UID: \"3860672d-226a-47fd-a812-cca575bea518\") " pod="ai-tenant-e2e-inf-a-04f248/test-model-04f248-kserve-7cf9d4bf7f-kd4rz" Nov 01 16:14:12.374080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:12.359240 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3860672d-226a-47fd-a812-cca575bea518-home\") pod \"test-model-04f248-kserve-7cf9d4bf7f-kd4rz\" (UID: \"3860672d-226a-47fd-a812-cca575bea518\") " pod="ai-tenant-e2e-inf-a-04f248/test-model-04f248-kserve-7cf9d4bf7f-kd4rz" Nov 01 16:14:12.379038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:12.377478 4363 patch_prober.go:29] interesting pod/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 01 16:14:12.379038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:12.377549 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" podUID="388de32a-ab53-48d9-983d-ce302916e7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 01 16:14:12.403101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:12.388476 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3860672d-226a-47fd-a812-cca575bea518-model-cache\") pod \"test-model-04f248-kserve-7cf9d4bf7f-kd4rz\" (UID: \"3860672d-226a-47fd-a812-cca575bea518\") " pod="ai-tenant-e2e-inf-a-04f248/test-model-04f248-kserve-7cf9d4bf7f-kd4rz" Nov 01 16:14:12.427042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:12.405712 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3860672d-226a-47fd-a812-cca575bea518-tmp-dir\") pod \"test-model-04f248-kserve-7cf9d4bf7f-kd4rz\" (UID: \"3860672d-226a-47fd-a812-cca575bea518\") " pod="ai-tenant-e2e-inf-a-04f248/test-model-04f248-kserve-7cf9d4bf7f-kd4rz" Nov 01 16:14:12.533218 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:12.532947 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3860672d-226a-47fd-a812-cca575bea518-tls-certs\") pod \"test-model-04f248-kserve-7cf9d4bf7f-kd4rz\" (UID: \"3860672d-226a-47fd-a812-cca575bea518\") " pod="ai-tenant-e2e-inf-a-04f248/test-model-04f248-kserve-7cf9d4bf7f-kd4rz" Nov 01 16:14:12.557038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:12.532491 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" podUID="d92f7196-153d-46b0-9e5b-51667329cd02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.169:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:14:12.572084 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:12.559525 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" Nov 01 16:14:12.607055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:12.578406 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3860672d-226a-47fd-a812-cca575bea518-dshm\") pod \"test-model-04f248-kserve-7cf9d4bf7f-kd4rz\" (UID: \"3860672d-226a-47fd-a812-cca575bea518\") " pod="ai-tenant-e2e-inf-a-04f248/test-model-04f248-kserve-7cf9d4bf7f-kd4rz" Nov 01 16:14:12.621207 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:12.607724 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:14:12.621207 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:12.608112 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:14:12.621207 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:12.608383 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:14:12.621207 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:12.608666 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:14:12.621207 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:12.609390 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:14:12.621207 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:12.609451 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 16:14:12.621207 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:12.609490 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:14:12.621207 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:12.609525 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 16:14:12.818632 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:12.818535 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" podUID="37773d39-9d52-4b1d-94c7-68e02714a476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.171:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:14:12.915062 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:12.905952 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5x9z\" (UniqueName: \"kubernetes.io/projected/3860672d-226a-47fd-a812-cca575bea518-kube-api-access-t5x9z\") pod \"test-model-04f248-kserve-7cf9d4bf7f-kd4rz\" (UID: \"3860672d-226a-47fd-a812-cca575bea518\") " pod="ai-tenant-e2e-inf-a-04f248/test-model-04f248-kserve-7cf9d4bf7f-kd4rz" Nov 01 16:14:12.975077 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:12.964956 4363 patch_prober.go:29] interesting pod/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Nov 01 16:14:12.975773 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:12.975730 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" podUID="b6716c2c-b539-4e3c-ac09-7f2d1426af74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Nov 01 16:14:13.005060 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:13.002170 4363 patch_prober.go:29] interesting pod/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Nov 01 16:14:13.005060 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:13.002233 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" podUID="ebf773ee-489a-418d-a031-327a467839c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Nov 01 16:14:13.203065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:13.179868 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:14:13.215243 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:13.181847 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" podUID="a15aa191-e376-490a-8763-16cdd7685b5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": context deadline exceeded" Nov 01 16:14:13.249276 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:13.191679 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:14:13.250867 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:13.193864 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-04f248/test-model-04f248-kserve-7cf9d4bf7f-kd4rz" Nov 01 16:14:13.275179 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:13.275133 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" podUID="9cbd41ee-3134-4b54-9ea1-0c13921269bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.181:8000/ready\": context deadline exceeded" Nov 01 16:14:13.284255 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:13.284229 4363 trace.go:236] Trace[67775726]: "Calculate volume metrics of ca-trust-extracted for pod openshift-image-registry/image-registry-85755df759-sswt8" (01-Nov-2026 16:14:12.164) (total time: 1119ms): Nov 01 16:14:13.284255 ip-10-0-128-238 kubenswrapper[4363]: Trace[67775726]: [1.119169035s] [1.119169035s] END Nov 01 16:14:13.312597 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:13.312555 4363 patch_prober.go:29] interesting pod/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 01 16:14:13.312900 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:13.312858 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" podUID="388de32a-ab53-48d9-983d-ce302916e7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 01 16:14:13.973173 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:13.830433 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:14:13.973173 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:13.830721 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 01 16:14:13.973173 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:13.830912 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:14:14.084060 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:14.007482 4363 patch_prober.go:29] interesting pod/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Nov 01 16:14:14.084060 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:14.007584 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" podUID="ebf773ee-489a-418d-a031-327a467839c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Nov 01 16:14:14.084060 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:14.027187 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:14:14.179046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:14.161812 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" podUID="d92f7196-153d-46b0-9e5b-51667329cd02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.169:8000/ready\": context deadline exceeded" Nov 01 16:14:14.308670 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:14.280417 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" podUID="9cbd41ee-3134-4b54-9ea1-0c13921269bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.181:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:14:14.340718 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:14.340658 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" podUID="37773d39-9d52-4b1d-94c7-68e02714a476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.171:8000/ready\": context deadline exceeded" Nov 01 16:14:14.442265 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:14.359568 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" podUID="a15aa191-e376-490a-8763-16cdd7685b5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": context deadline exceeded" Nov 01 16:14:14.442877 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:14.366773 4363 patch_prober.go:29] interesting pod/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 01 16:14:14.444127 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:14.444090 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" podUID="388de32a-ab53-48d9-983d-ce302916e7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 01 16:14:14.444291 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:14.301999 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:14:14.448037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:14.374824 4363 patch_prober.go:29] interesting pod/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Nov 01 16:14:14.448242 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:14.448215 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" podUID="b6716c2c-b539-4e3c-ac09-7f2d1426af74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Nov 01 16:14:14.500110 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:14.500058 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:14:14.525636 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:14.513225 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded" Nov 01 16:14:14.705542 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:14:14.705499 4363 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.911s" Nov 01 16:14:14.711197 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:14.711175 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:14:14.869217 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:14.727695 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" Nov 01 16:14:14.894487 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:14.871747 4363 patch_prober.go:29] interesting pod/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Nov 01 16:14:14.894487 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:14.871833 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" podUID="ebf773ee-489a-418d-a031-327a467839c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Nov 01 16:14:14.911354 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:14.911295 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:14:14.945189 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:14.928423 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 01 16:14:15.109042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:15.075519 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": context deadline exceeded" Nov 01 16:14:15.241893 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:15.215370 4363 patch_prober.go:29] interesting pod/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Nov 01 16:14:15.242176 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:15.242135 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" podUID="b6716c2c-b539-4e3c-ac09-7f2d1426af74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Nov 01 16:14:15.317330 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:15.314918 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" podUID="d92f7196-153d-46b0-9e5b-51667329cd02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.169:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:14:15.389038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:15.369810 4363 patch_prober.go:29] interesting pod/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 01 16:14:15.389038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:15.369905 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" podUID="388de32a-ab53-48d9-983d-ce302916e7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 01 16:14:15.491044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:15.470102 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:14:15.560098 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:15.544587 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:14:15.560098 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:15.545157 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:14:15.660552 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:15.660150 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:14:15.685645 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:15.685536 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:14:15.930065 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:14:15.930033 4363 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.112s" Nov 01 16:14:15.930368 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:15.930352 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:14:15.930563 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:15.930538 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" Nov 01 16:14:15.930765 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:15.930747 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" Nov 01 16:14:15.930902 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:15.930891 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:14:15.931009 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:15.931000 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" Nov 01 16:14:15.931151 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:15.931141 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" Nov 01 16:14:15.931277 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:15.931268 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" Nov 01 16:14:15.931406 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:15.931397 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:14:15.936055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:15.935712 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 16:14:15.936251 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:15.936221 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:14:16.015044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:15.971975 4363 patch_prober.go:29] interesting pod/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Nov 01 16:14:16.015044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:15.972073 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" podUID="b6716c2c-b539-4e3c-ac09-7f2d1426af74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Nov 01 16:14:16.015044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:15.990287 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" podUID="9cbd41ee-3134-4b54-9ea1-0c13921269bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.181:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:14:16.015044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:15.990996 4363 patch_prober.go:29] interesting pod/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Nov 01 16:14:16.015044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:15.991061 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" podUID="ebf773ee-489a-418d-a031-327a467839c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Nov 01 16:14:16.153094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:16.130579 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:14:16.328429 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:16.280983 4363 patch_prober.go:29] interesting pod/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 01 16:14:16.328743 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:16.328708 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" podUID="388de32a-ab53-48d9-983d-ce302916e7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 01 16:14:16.334130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:16.329946 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" podUID="d92f7196-153d-46b0-9e5b-51667329cd02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.169:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:14:16.404868 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:16.393590 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-04f248/test-model-04f248-kserve-7cf9d4bf7f-kd4rz"] Nov 01 16:14:16.470737 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:16.468589 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 16:14:16.470737 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:16.470234 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:14:16.652073 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:16.652030 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" podUID="d92f7196-153d-46b0-9e5b-51667329cd02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.169:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:14:16.761694 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:16.761600 4363 patch_prober.go:29] interesting pod/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Nov 01 16:14:16.761950 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:16.761919 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" podUID="b6716c2c-b539-4e3c-ac09-7f2d1426af74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Nov 01 16:14:16.763297 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:16.763272 4363 patch_prober.go:29] interesting pod/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Nov 01 16:14:16.763480 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:16.763459 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" podUID="ebf773ee-489a-418d-a031-327a467839c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Nov 01 16:14:17.159763 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:17.155563 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" podUID="9cbd41ee-3134-4b54-9ea1-0c13921269bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.181:8000/ready\": context deadline exceeded" Nov 01 16:14:17.225697 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:17.225650 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" Nov 01 16:14:17.267038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:17.265369 4363 patch_prober.go:29] interesting pod/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 01 16:14:17.267038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:17.265422 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" podUID="388de32a-ab53-48d9-983d-ce302916e7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 01 16:14:17.422783 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:17.422733 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" podUID="d92f7196-153d-46b0-9e5b-51667329cd02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.169:8000/ready\": context deadline exceeded" Nov 01 16:14:17.524044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:17.501434 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:14:17.643053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:17.627075 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-04f248/test-model-04f248-kserve-7cf9d4bf7f-kd4rz" event={"ID":"3860672d-226a-47fd-a812-cca575bea518","Type":"ContainerStarted","Data":"59f87ca26ae0f29809650979c5187a46246473e166dec09de104bc253ab4c4c0"} Nov 01 16:14:17.716138 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:17.694860 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-99e837/test-model-99e837-kserve-68448d5ff7-hfsbc" Nov 01 16:14:17.766208 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:17.766160 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" podUID="37773d39-9d52-4b1d-94c7-68e02714a476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.171:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:14:17.770523 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:17.770491 4363 patch_prober.go:29] interesting pod/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Nov 01 16:14:17.771486 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:17.771449 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" podUID="b6716c2c-b539-4e3c-ac09-7f2d1426af74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Nov 01 16:14:17.772616 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:17.772591 4363 patch_prober.go:29] interesting pod/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Nov 01 16:14:17.772785 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:17.772766 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" podUID="ebf773ee-489a-418d-a031-327a467839c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Nov 01 16:14:18.011874 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:18.011761 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:14:18.160174 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:18.160115 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" podUID="9cbd41ee-3134-4b54-9ea1-0c13921269bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.181:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:14:18.254053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:18.244910 4363 patch_prober.go:29] interesting pod/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 01 16:14:18.254053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:18.244966 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" podUID="388de32a-ab53-48d9-983d-ce302916e7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 01 16:14:18.530044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:18.518175 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" podUID="d92f7196-153d-46b0-9e5b-51667329cd02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.169:8000/ready\": context deadline exceeded" Nov 01 16:14:18.693938 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:18.693621 4363 prober.go:122] "Probe failed" probeType="Startup" pod="ai-tenant-e2e-inf-b-99e837/test-model-99e837-kserve-68448d5ff7-hfsbc" podUID="62fde750-084e-488b-9d65-24b38a323cb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.185:8000/health\": context deadline exceeded" Nov 01 16:14:18.699455 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:18.699400 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-04f248/test-model-04f248-kserve-7cf9d4bf7f-kd4rz" event={"ID":"3860672d-226a-47fd-a812-cca575bea518","Type":"ContainerStarted","Data":"deae462eb6fc6d13fb764813e44bcf0465484eb57a238e2456ea84b8991599b2"} Nov 01 16:14:18.726039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:18.722689 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-04f248/test-model-04f248-kserve-7cf9d4bf7f-kd4rz" podStartSLOduration=8.722667369 podStartE2EDuration="8.722667369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:14:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:14:18.721514036 +0000 UTC m=+1592.053251689" watchObservedRunningTime="2026-11-01 16:14:18.722667369 +0000 UTC m=+1592.054405011" Nov 01 16:14:18.769406 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:18.768698 4363 patch_prober.go:29] interesting pod/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Nov 01 16:14:18.769406 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:18.768756 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" podUID="ebf773ee-489a-418d-a031-327a467839c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Nov 01 16:14:18.769406 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:18.769312 4363 patch_prober.go:29] interesting pod/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Nov 01 16:14:18.769406 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:18.769353 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" podUID="b6716c2c-b539-4e3c-ac09-7f2d1426af74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Nov 01 16:14:19.162425 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:19.149062 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" podUID="9cbd41ee-3134-4b54-9ea1-0c13921269bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.181:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:14:19.301769 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:19.172783 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" podUID="9cbd41ee-3134-4b54-9ea1-0c13921269bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.181:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:14:19.301769 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:19.240723 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:14:19.301769 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:19.249936 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:14:19.517103 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:19.431217 4363 patch_prober.go:29] interesting pod/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 01 16:14:19.517103 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:19.431280 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" podUID="388de32a-ab53-48d9-983d-ce302916e7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 01 16:14:19.517103 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:19.454043 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:14:19.517103 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:19.454091 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-04f248/test-model-04f248-kserve-7cf9d4bf7f-kd4rz" Nov 01 16:14:19.517103 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:19.454118 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" Nov 01 16:14:19.517103 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:19.481473 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:14:19.517103 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:19.481562 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" Nov 01 16:14:19.618805 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:14:19.618747 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:14:19.693706 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:19.693663 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:14:19.741305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:19.741272 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" Nov 01 16:14:19.853690 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:19.853659 4363 patch_prober.go:29] interesting pod/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Nov 01 16:14:19.912414 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:19.890564 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" podUID="b6716c2c-b539-4e3c-ac09-7f2d1426af74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Nov 01 16:14:20.001140 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:20.001103 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" Nov 01 16:14:20.013335 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:20.013306 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" Nov 01 16:14:20.032173 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:20.032147 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" Nov 01 16:14:20.266554 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:20.266469 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" Nov 01 16:14:20.268126 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:20.268105 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" Nov 01 16:14:20.268848 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:20.268824 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" Nov 01 16:14:20.767476 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:20.767449 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" Nov 01 16:14:20.768403 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:20.768380 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" Nov 01 16:14:23.194358 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:23.194316 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-04f248/test-model-04f248-kserve-7cf9d4bf7f-kd4rz" Nov 01 16:14:23.212909 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:23.212876 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-04f248/test-model-04f248-kserve-7cf9d4bf7f-kd4rz" Nov 01 16:14:23.805580 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:23.805550 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-04f248/test-model-04f248-kserve-7cf9d4bf7f-kd4rz" Nov 01 16:14:27.701738 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:27.701707 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-99e837/test-model-99e837-kserve-68448d5ff7-hfsbc" Nov 01 16:14:27.713222 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:27.713194 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-99e837/test-model-99e837-kserve-68448d5ff7-hfsbc" Nov 01 16:14:48.416060 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:14:48.413765 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:14:55.610581 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:55.610521 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:14:55.888145 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:14:55.876908 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" podUID="a15aa191-e376-490a-8763-16cdd7685b5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": context deadline exceeded" Nov 01 16:15:00.017108 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:00.017064 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv"] Nov 01 16:15:00.017977 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:00.017938 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" podUID="74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c" containerName="istio-proxy" containerID="cri-o://9a7c872d94e083fdc805f44d27c1cc61c2bd1b5828fc585d17f010d0168a821f" gracePeriod=30 Nov 01 16:15:00.383045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:00.375940 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892495-btfk6"] Nov 01 16:15:00.627577 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:00.624622 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892495-btfk6"] Nov 01 16:15:00.627577 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:00.624680 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr"] Nov 01 16:15:00.627577 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:00.625704 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892495-btfk6" Nov 01 16:15:00.627577 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:00.626768 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" podUID="af073176-cb1d-4eaf-9b6c-43787f6a7227" containerName="istio-proxy" containerID="cri-o://caf00e813eeb9ce335df97a58f7d406b7312c819f436d8d93cb7da42bcd6817d" gracePeriod=30 Nov 01 16:15:00.731840 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:00.731806 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:15:00.756276 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:00.756213 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bf581a3-128c-4b8a-a004-e61af24326c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892495-btfk6\" (UID: \"0bf581a3-128c-4b8a-a004-e61af24326c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892495-btfk6" Nov 01 16:15:00.756670 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:00.756646 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf6nr\" (UniqueName: \"kubernetes.io/projected/0bf581a3-128c-4b8a-a004-e61af24326c2-kube-api-access-pf6nr\") pod \"maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892495-btfk6\" (UID: \"0bf581a3-128c-4b8a-a004-e61af24326c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892495-btfk6" Nov 01 16:15:00.871195 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:00.871151 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf6nr\" (UniqueName: \"kubernetes.io/projected/0bf581a3-128c-4b8a-a004-e61af24326c2-kube-api-access-pf6nr\") pod \"maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892495-btfk6\" (UID: \"0bf581a3-128c-4b8a-a004-e61af24326c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892495-btfk6" Nov 01 16:15:00.871486 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:00.871465 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bf581a3-128c-4b8a-a004-e61af24326c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892495-btfk6\" (UID: \"0bf581a3-128c-4b8a-a004-e61af24326c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892495-btfk6" Nov 01 16:15:00.895731 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:00.872331 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bf581a3-128c-4b8a-a004-e61af24326c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892495-btfk6\" (UID: \"0bf581a3-128c-4b8a-a004-e61af24326c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892495-btfk6" Nov 01 16:15:01.174853 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:01.174819 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf6nr\" (UniqueName: \"kubernetes.io/projected/0bf581a3-128c-4b8a-a004-e61af24326c2-kube-api-access-pf6nr\") pod \"maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892495-btfk6\" (UID: \"0bf581a3-128c-4b8a-a004-e61af24326c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892495-btfk6" Nov 01 16:15:01.324140 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:01.324103 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892495-btfk6" Nov 01 16:15:01.662977 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:01.662718 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892495-btfk6"] Nov 01 16:15:01.963137 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:01.963071 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892495-btfk6" event={"ID":"0bf581a3-128c-4b8a-a004-e61af24326c2","Type":"ContainerStarted","Data":"f5020bb807b588ad57b66067a3c905254dd4d7cece50bf946593acff5a6db0e0"} Nov 01 16:15:02.977232 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:02.977186 4363 generic.go:356] "Generic (PLEG): container finished" podID="0bf581a3-128c-4b8a-a004-e61af24326c2" containerID="f3297c53e8e5e52d0efa5a44d528e0dce27b3f948db1fa35446643cd00353b76" exitCode=60 Nov 01 16:15:02.977687 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:02.977274 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892495-btfk6" event={"ID":"0bf581a3-128c-4b8a-a004-e61af24326c2","Type":"ContainerDied","Data":"f3297c53e8e5e52d0efa5a44d528e0dce27b3f948db1fa35446643cd00353b76"} Nov 01 16:15:02.978096 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:02.978008 4363 scope.go:122] "RemoveContainer" containerID="f3297c53e8e5e52d0efa5a44d528e0dce27b3f948db1fa35446643cd00353b76" Nov 01 16:15:03.993177 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:03.993140 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892495-btfk6" event={"ID":"0bf581a3-128c-4b8a-a004-e61af24326c2","Type":"ContainerStarted","Data":"e198ab11a3398898ec2ccb46ee7e9388622146ee2f8bc47245514796f818cc34"} Nov 01 16:15:03.994185 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:03.994145 4363 patch_prober.go:29] interesting pod/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:15:03.994327 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:03.994194 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" podUID="22eae6cc-4752-4174-a0f0-573adc82e32b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:15:04.017344 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:04.017282 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892495-btfk6" podStartSLOduration=4.017263834 podStartE2EDuration="4.017263834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:15:04.009853236 +0000 UTC m=+1637.341590887" watchObservedRunningTime="2026-11-01 16:15:04.017263834 +0000 UTC m=+1637.349001485" Nov 01 16:15:05.138583 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.135138 4363 generic.go:356] "Generic (PLEG): container finished" podID="0bf581a3-128c-4b8a-a004-e61af24326c2" containerID="e198ab11a3398898ec2ccb46ee7e9388622146ee2f8bc47245514796f818cc34" exitCode=60 Nov 01 16:15:05.138583 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.135311 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892495-btfk6" event={"ID":"0bf581a3-128c-4b8a-a004-e61af24326c2","Type":"ContainerDied","Data":"e198ab11a3398898ec2ccb46ee7e9388622146ee2f8bc47245514796f818cc34"} Nov 01 16:15:05.138583 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.135372 4363 scope.go:122] "RemoveContainer" containerID="f3297c53e8e5e52d0efa5a44d528e0dce27b3f948db1fa35446643cd00353b76" Nov 01 16:15:05.138583 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.135609 4363 scope.go:122] "RemoveContainer" containerID="e198ab11a3398898ec2ccb46ee7e9388622146ee2f8bc47245514796f818cc34" Nov 01 16:15:05.138583 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:15:05.136185 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892495-btfk6_odh-ai-gateway-infra(0bf581a3-128c-4b8a-a004-e61af24326c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892495-btfk6" podUID="0bf581a3-128c-4b8a-a004-e61af24326c2" Nov 01 16:15:05.507663 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.507625 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" Nov 01 16:15:05.520527 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.520487 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-istio-podinfo\") pod \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\" (UID: \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\") " Nov 01 16:15:05.520784 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.520636 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-credential-socket\") pod \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\" (UID: \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\") " Nov 01 16:15:05.520784 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.520739 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-istiod-ca-cert\") pod \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\" (UID: \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\") " Nov 01 16:15:05.520901 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.520797 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-kube-api-access-hszw2\" (UniqueName: \"kubernetes.io/projected/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-kube-api-access-hszw2\") pod \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\" (UID: \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\") " Nov 01 16:15:05.520901 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.520841 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-workload-certs\") pod \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\" (UID: \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\") " Nov 01 16:15:05.520901 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.520887 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-istio-envoy\") pod \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\" (UID: \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\") " Nov 01 16:15:05.521065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.520998 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-istio-token\" (UniqueName: \"kubernetes.io/projected/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-istio-token\") pod \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\" (UID: \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\") " Nov 01 16:15:05.521310 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.521286 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-workload-certs" pod "74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c" (UID: "74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:15:05.521382 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.521316 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-istiod-ca-cert" pod "74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c" (UID: "74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:15:05.521596 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.521563 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-credential-socket" pod "74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c" (UID: "74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:15:05.521716 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.521695 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-istio-data\") pod \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\" (UID: \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\") " Nov 01 16:15:05.521787 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.521764 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-workload-socket\") pod \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\" (UID: \"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c\") " Nov 01 16:15:05.522060 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.522009 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-istio-data" pod "74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c" (UID: "74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:15:05.522273 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.522246 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-workload-socket" pod "74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c" (UID: "74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:15:05.522930 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.522900 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:05.523054 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.522937 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:05.523054 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.522952 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:05.523054 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.522965 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:05.523054 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.522977 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:05.524743 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.524058 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-istio-podinfo" pod "74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c" (UID: "74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:15:05.525763 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.525144 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-istio-envoy" pod "74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c" (UID: "74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:15:05.534846 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.531474 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-kube-api-access-hszw2" pod "74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c" (UID: "74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c"). InnerVolumeSpecName "kube-api-access-hszw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:15:05.534846 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.531518 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-istio-token" pod "74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c" (UID: "74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:15:05.573629 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.571062 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv"] Nov 01 16:15:05.575635 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.575239 4363 cpu_manager.go:414] "containerMap: removing container" podUID="74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c" containerName="istio-proxy" Nov 01 16:15:05.575635 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.575264 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c" containerName="istio-proxy" Nov 01 16:15:05.575635 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.575564 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c" containerName="istio-proxy" Nov 01 16:15:05.608512 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.608478 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv"] Nov 01 16:15:05.608701 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.608613 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" Nov 01 16:15:05.628456 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.624295 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-workload-socket\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv\" (UID: \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" Nov 01 16:15:05.628456 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.624364 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-istiod-ca-cert\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv\" (UID: \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" Nov 01 16:15:05.628456 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.624454 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-istio-token\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv\" (UID: \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" Nov 01 16:15:05.628456 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.624541 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-workload-certs\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv\" (UID: \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" Nov 01 16:15:05.628456 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.624586 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-istio-envoy\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv\" (UID: \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" Nov 01 16:15:05.628456 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.624628 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-istio-data\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv\" (UID: \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" Nov 01 16:15:05.628456 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.624688 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-credential-socket\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv\" (UID: \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" Nov 01 16:15:05.628456 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.624781 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-istio-podinfo\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv\" (UID: \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" Nov 01 16:15:05.628456 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.624829 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnk27\" (UniqueName: \"kubernetes.io/projected/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-kube-api-access-cnk27\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv\" (UID: \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" Nov 01 16:15:05.628456 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.625054 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:05.628456 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.625076 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hszw2\" (UniqueName: \"kubernetes.io/projected/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-kube-api-access-hszw2\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:05.628456 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.625093 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:05.628456 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.625109 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:05.726406 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.726369 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-istiod-ca-cert\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv\" (UID: \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" Nov 01 16:15:05.726614 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.726483 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-istio-token\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv\" (UID: \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" Nov 01 16:15:05.726614 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.726575 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-workload-certs\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv\" (UID: \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" Nov 01 16:15:05.726765 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.726611 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-istio-envoy\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv\" (UID: \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" Nov 01 16:15:05.726765 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.726655 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-istio-data\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv\" (UID: \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" Nov 01 16:15:05.726765 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.726721 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-credential-socket\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv\" (UID: \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" Nov 01 16:15:05.726932 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.726814 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-istio-podinfo\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv\" (UID: \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" Nov 01 16:15:05.727212 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.727183 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-istiod-ca-cert\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv\" (UID: \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" Nov 01 16:15:05.727854 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.727194 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnk27\" (UniqueName: \"kubernetes.io/projected/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-kube-api-access-cnk27\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv\" (UID: \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" Nov 01 16:15:05.727854 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.727628 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-workload-socket\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv\" (UID: \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" Nov 01 16:15:05.727854 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.727767 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-workload-certs\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv\" (UID: \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" Nov 01 16:15:05.728114 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.727983 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-workload-socket\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv\" (UID: \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" Nov 01 16:15:05.728507 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.728486 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-credential-socket\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv\" (UID: \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" Nov 01 16:15:05.728733 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.728713 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-istio-data\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv\" (UID: \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" Nov 01 16:15:05.743984 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.730547 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-istio-envoy\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv\" (UID: \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" Nov 01 16:15:05.769363 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.752656 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-istio-token\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv\" (UID: \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" Nov 01 16:15:05.769363 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.752717 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnk27\" (UniqueName: \"kubernetes.io/projected/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-kube-api-access-cnk27\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv\" (UID: \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" Nov 01 16:15:05.769363 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.753124 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-istio-podinfo\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv\" (UID: \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" Nov 01 16:15:05.978114 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:05.967924 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" Nov 01 16:15:06.296823 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.296750 4363 scope.go:122] "RemoveContainer" containerID="e198ab11a3398898ec2ccb46ee7e9388622146ee2f8bc47245514796f818cc34" Nov 01 16:15:06.297779 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:15:06.297519 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892495-btfk6_odh-ai-gateway-infra(0bf581a3-128c-4b8a-a004-e61af24326c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892495-btfk6" podUID="0bf581a3-128c-4b8a-a004-e61af24326c2" Nov 01 16:15:06.305145 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.300874 4363 generic.go:356] "Generic (PLEG): container finished" podID="74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c" containerID="9a7c872d94e083fdc805f44d27c1cc61c2bd1b5828fc585d17f010d0168a821f" exitCode=0 Nov 01 16:15:06.305145 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.301149 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" event={"ID":"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c","Type":"ContainerDied","Data":"9a7c872d94e083fdc805f44d27c1cc61c2bd1b5828fc585d17f010d0168a821f"} Nov 01 16:15:06.305145 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.301240 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" event={"ID":"74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c","Type":"ContainerDied","Data":"7fd1e38f496d126d8c338a22a677f3571b153c21461881c4c59e00e1889d5b85"} Nov 01 16:15:06.305145 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.301277 4363 scope.go:122] "RemoveContainer" containerID="9a7c872d94e083fdc805f44d27c1cc61c2bd1b5828fc585d17f010d0168a821f" Nov 01 16:15:06.306724 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.306294 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv" Nov 01 16:15:06.316270 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.316237 4363 generic.go:356] "Generic (PLEG): container finished" podID="af073176-cb1d-4eaf-9b6c-43787f6a7227" containerID="caf00e813eeb9ce335df97a58f7d406b7312c819f436d8d93cb7da42bcd6817d" exitCode=0 Nov 01 16:15:06.316596 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.316569 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" event={"ID":"af073176-cb1d-4eaf-9b6c-43787f6a7227","Type":"ContainerDied","Data":"caf00e813eeb9ce335df97a58f7d406b7312c819f436d8d93cb7da42bcd6817d"} Nov 01 16:15:06.355416 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.355269 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" Nov 01 16:15:06.360435 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.360368 4363 scope.go:122] "RemoveContainer" containerID="9a7c872d94e083fdc805f44d27c1cc61c2bd1b5828fc585d17f010d0168a821f" Nov 01 16:15:06.361121 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:15:06.361004 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a7c872d94e083fdc805f44d27c1cc61c2bd1b5828fc585d17f010d0168a821f\": container with ID starting with 9a7c872d94e083fdc805f44d27c1cc61c2bd1b5828fc585d17f010d0168a821f not found: ID does not exist" containerID="9a7c872d94e083fdc805f44d27c1cc61c2bd1b5828fc585d17f010d0168a821f" Nov 01 16:15:06.361121 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.361067 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a7c872d94e083fdc805f44d27c1cc61c2bd1b5828fc585d17f010d0168a821f"} err="failed to get container status \"9a7c872d94e083fdc805f44d27c1cc61c2bd1b5828fc585d17f010d0168a821f\": rpc error: code = NotFound desc = could not find container \"9a7c872d94e083fdc805f44d27c1cc61c2bd1b5828fc585d17f010d0168a821f\": container with ID starting with 9a7c872d94e083fdc805f44d27c1cc61c2bd1b5828fc585d17f010d0168a821f not found: ID does not exist" Nov 01 16:15:06.423148 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.422081 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e044c-29892495-pvbf4"] Nov 01 16:15:06.425179 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.423794 4363 cpu_manager.go:414] "containerMap: removing container" podUID="af073176-cb1d-4eaf-9b6c-43787f6a7227" containerName="istio-proxy" Nov 01 16:15:06.425179 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.423819 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af073176-cb1d-4eaf-9b6c-43787f6a7227" containerName="istio-proxy" Nov 01 16:15:06.425179 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.424164 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="af073176-cb1d-4eaf-9b6c-43787f6a7227" containerName="istio-proxy" Nov 01 16:15:06.446396 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.431733 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af073176-cb1d-4eaf-9b6c-43787f6a7227-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af073176-cb1d-4eaf-9b6c-43787f6a7227-istiod-ca-cert\") pod \"af073176-cb1d-4eaf-9b6c-43787f6a7227\" (UID: \"af073176-cb1d-4eaf-9b6c-43787f6a7227\") " Nov 01 16:15:06.446396 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.431885 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af073176-cb1d-4eaf-9b6c-43787f6a7227-istio-token\" (UniqueName: \"kubernetes.io/projected/af073176-cb1d-4eaf-9b6c-43787f6a7227-istio-token\") pod \"af073176-cb1d-4eaf-9b6c-43787f6a7227\" (UID: \"af073176-cb1d-4eaf-9b6c-43787f6a7227\") " Nov 01 16:15:06.446396 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.431941 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af073176-cb1d-4eaf-9b6c-43787f6a7227-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af073176-cb1d-4eaf-9b6c-43787f6a7227-workload-socket\") pod \"af073176-cb1d-4eaf-9b6c-43787f6a7227\" (UID: \"af073176-cb1d-4eaf-9b6c-43787f6a7227\") " Nov 01 16:15:06.446396 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.432061 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af073176-cb1d-4eaf-9b6c-43787f6a7227-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af073176-cb1d-4eaf-9b6c-43787f6a7227-istio-envoy\") pod \"af073176-cb1d-4eaf-9b6c-43787f6a7227\" (UID: \"af073176-cb1d-4eaf-9b6c-43787f6a7227\") " Nov 01 16:15:06.446396 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.432142 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af073176-cb1d-4eaf-9b6c-43787f6a7227-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af073176-cb1d-4eaf-9b6c-43787f6a7227-istio-data\") pod \"af073176-cb1d-4eaf-9b6c-43787f6a7227\" (UID: \"af073176-cb1d-4eaf-9b6c-43787f6a7227\") " Nov 01 16:15:06.446396 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.432236 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af073176-cb1d-4eaf-9b6c-43787f6a7227-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af073176-cb1d-4eaf-9b6c-43787f6a7227-workload-certs\") pod \"af073176-cb1d-4eaf-9b6c-43787f6a7227\" (UID: \"af073176-cb1d-4eaf-9b6c-43787f6a7227\") " Nov 01 16:15:06.446396 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.432275 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af073176-cb1d-4eaf-9b6c-43787f6a7227-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af073176-cb1d-4eaf-9b6c-43787f6a7227-credential-socket\") pod \"af073176-cb1d-4eaf-9b6c-43787f6a7227\" (UID: \"af073176-cb1d-4eaf-9b6c-43787f6a7227\") " Nov 01 16:15:06.446396 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.432341 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af073176-cb1d-4eaf-9b6c-43787f6a7227-kube-api-access-v5mv9\" (UniqueName: \"kubernetes.io/projected/af073176-cb1d-4eaf-9b6c-43787f6a7227-kube-api-access-v5mv9\") pod \"af073176-cb1d-4eaf-9b6c-43787f6a7227\" (UID: \"af073176-cb1d-4eaf-9b6c-43787f6a7227\") " Nov 01 16:15:06.446396 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.432496 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af073176-cb1d-4eaf-9b6c-43787f6a7227-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af073176-cb1d-4eaf-9b6c-43787f6a7227-istio-podinfo\") pod \"af073176-cb1d-4eaf-9b6c-43787f6a7227\" (UID: \"af073176-cb1d-4eaf-9b6c-43787f6a7227\") " Nov 01 16:15:06.446396 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.436540 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af073176-cb1d-4eaf-9b6c-43787f6a7227-istiod-ca-cert" pod "af073176-cb1d-4eaf-9b6c-43787f6a7227" (UID: "af073176-cb1d-4eaf-9b6c-43787f6a7227"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:15:06.446396 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.437144 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af073176-cb1d-4eaf-9b6c-43787f6a7227-workload-certs" pod "af073176-cb1d-4eaf-9b6c-43787f6a7227" (UID: "af073176-cb1d-4eaf-9b6c-43787f6a7227"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:15:06.446396 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.438451 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af073176-cb1d-4eaf-9b6c-43787f6a7227-workload-socket" pod "af073176-cb1d-4eaf-9b6c-43787f6a7227" (UID: "af073176-cb1d-4eaf-9b6c-43787f6a7227"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:15:06.446396 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.441906 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af073176-cb1d-4eaf-9b6c-43787f6a7227-credential-socket" pod "af073176-cb1d-4eaf-9b6c-43787f6a7227" (UID: "af073176-cb1d-4eaf-9b6c-43787f6a7227"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:15:06.446396 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.444561 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af073176-cb1d-4eaf-9b6c-43787f6a7227-istio-envoy" pod "af073176-cb1d-4eaf-9b6c-43787f6a7227" (UID: "af073176-cb1d-4eaf-9b6c-43787f6a7227"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:15:06.446396 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.444779 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af073176-cb1d-4eaf-9b6c-43787f6a7227-istio-data" pod "af073176-cb1d-4eaf-9b6c-43787f6a7227" (UID: "af073176-cb1d-4eaf-9b6c-43787f6a7227"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:15:06.446396 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.444905 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af073176-cb1d-4eaf-9b6c-43787f6a7227-istio-token" pod "af073176-cb1d-4eaf-9b6c-43787f6a7227" (UID: "af073176-cb1d-4eaf-9b6c-43787f6a7227"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:15:06.447416 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.447192 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af073176-cb1d-4eaf-9b6c-43787f6a7227-kube-api-access-v5mv9" pod "af073176-cb1d-4eaf-9b6c-43787f6a7227" (UID: "af073176-cb1d-4eaf-9b6c-43787f6a7227"). InnerVolumeSpecName "kube-api-access-v5mv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:15:06.451450 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.451409 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af073176-cb1d-4eaf-9b6c-43787f6a7227-istio-podinfo" pod "af073176-cb1d-4eaf-9b6c-43787f6a7227" (UID: "af073176-cb1d-4eaf-9b6c-43787f6a7227"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:15:06.458052 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.457727 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e044c-29892495-pvbf4" Nov 01 16:15:06.556298 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.477491 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99e837-29892495-lnsd2"] Nov 01 16:15:06.556298 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.536941 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892495-2nh9k"] Nov 01 16:15:06.556298 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.539767 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af073176-cb1d-4eaf-9b6c-43787f6a7227-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:06.556298 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.539796 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af073176-cb1d-4eaf-9b6c-43787f6a7227-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:06.556298 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.539814 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af073176-cb1d-4eaf-9b6c-43787f6a7227-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:06.556298 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.539829 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af073176-cb1d-4eaf-9b6c-43787f6a7227-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:06.556298 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.539843 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5mv9\" (UniqueName: \"kubernetes.io/projected/af073176-cb1d-4eaf-9b6c-43787f6a7227-kube-api-access-v5mv9\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:06.556298 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.539858 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af073176-cb1d-4eaf-9b6c-43787f6a7227-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:06.556298 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.539872 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af073176-cb1d-4eaf-9b6c-43787f6a7227-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:06.556298 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.539887 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af073176-cb1d-4eaf-9b6c-43787f6a7227-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:06.556298 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.539901 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af073176-cb1d-4eaf-9b6c-43787f6a7227-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:06.556298 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.543137 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99e837-29892495-lnsd2" Nov 01 16:15:06.591644 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.591002 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892495-rwvhk"] Nov 01 16:15:06.591644 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.591103 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892495-2nh9k" Nov 01 16:15:06.597180 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.596715 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892495-xxk9n"] Nov 01 16:15:06.601694 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.600046 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892495-rwvhk" Nov 01 16:15:06.602561 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.602539 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892495-v75rj"] Nov 01 16:15:06.605166 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.605105 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892495-xxk9n" Nov 01 16:15:06.612789 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.612767 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892495-wgnf7"] Nov 01 16:15:06.613157 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.613134 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892495-v75rj" Nov 01 16:15:06.620223 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.620204 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2ce08a-29892495-znkts"] Nov 01 16:15:06.620571 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.620319 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892495-wgnf7" Nov 01 16:15:06.627736 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.627520 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892495-wfnfr"] Nov 01 16:15:06.627736 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.627602 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2ce08a-29892495-znkts" Nov 01 16:15:06.634363 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.634296 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892495-wfnfr" Nov 01 16:15:06.643374 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.641733 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj2xc\" (UniqueName: \"kubernetes.io/projected/1191cd6c-176c-4f44-97c3-06942c21b3d9-kube-api-access-qj2xc\") pod \"maas-api-key-cleanup-e2e-inf-a-9e044c-29892495-pvbf4\" (UID: \"1191cd6c-176c-4f44-97c3-06942c21b3d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e044c-29892495-pvbf4" Nov 01 16:15:06.643374 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.641982 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhr9v\" (UniqueName: \"kubernetes.io/projected/a4377fa0-6280-4da7-b589-2afd643d3bce-kube-api-access-vhr9v\") pod \"maas-api-key-cleanup-e2e-inf-b-99e837-29892495-lnsd2\" (UID: \"a4377fa0-6280-4da7-b589-2afd643d3bce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99e837-29892495-lnsd2" Nov 01 16:15:06.643374 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.642060 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4377fa0-6280-4da7-b589-2afd643d3bce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-99e837-29892495-lnsd2\" (UID: \"a4377fa0-6280-4da7-b589-2afd643d3bce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99e837-29892495-lnsd2" Nov 01 16:15:06.643374 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.642146 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1191cd6c-176c-4f44-97c3-06942c21b3d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-9e044c-29892495-pvbf4\" (UID: \"1191cd6c-176c-4f44-97c3-06942c21b3d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e044c-29892495-pvbf4" Nov 01 16:15:06.649075 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.648100 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4240a-29892495-z49zr"] Nov 01 16:15:06.668760 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.668737 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4240a-29892495-z49zr" Nov 01 16:15:06.685625 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.685515 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e044c-29892495-pvbf4"] Nov 01 16:15:06.695508 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.695427 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892495-2nh9k"] Nov 01 16:15:06.713329 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.713303 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892495-rwvhk"] Nov 01 16:15:06.719194 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.719159 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892495-xxk9n"] Nov 01 16:15:06.726211 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.726185 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892495-v75rj"] Nov 01 16:15:06.735309 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.735279 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2ce08a-29892495-znkts"] Nov 01 16:15:06.744124 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.744103 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892495-wgnf7"] Nov 01 16:15:06.744904 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.744881 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/896ad84c-84b6-4f21-b2b8-9689578e43c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892495-2nh9k\" (UID: \"896ad84c-84b6-4f21-b2b8-9689578e43c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892495-2nh9k" Nov 01 16:15:06.745128 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.745109 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/689843e8-64df-4f9c-9718-d121ba50a8cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892495-rwvhk\" (UID: \"689843e8-64df-4f9c-9718-d121ba50a8cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892495-rwvhk" Nov 01 16:15:06.745327 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.745307 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cecab247-7b67-4a90-a3fd-99dfde6c8994-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892495-v75rj\" (UID: \"cecab247-7b67-4a90-a3fd-99dfde6c8994\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892495-v75rj" Nov 01 16:15:06.745506 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.745486 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1191cd6c-176c-4f44-97c3-06942c21b3d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-9e044c-29892495-pvbf4\" (UID: \"1191cd6c-176c-4f44-97c3-06942c21b3d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e044c-29892495-pvbf4" Nov 01 16:15:06.745673 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.745646 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0918fbb-97c7-4653-ae3d-3de28dc050bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bd831a-29892495-xxk9n\" (UID: \"a0918fbb-97c7-4653-ae3d-3de28dc050bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892495-xxk9n" Nov 01 16:15:06.745748 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.745718 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tblg5\" (UniqueName: \"kubernetes.io/projected/689843e8-64df-4f9c-9718-d121ba50a8cb-kube-api-access-tblg5\") pod \"maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892495-rwvhk\" (UID: \"689843e8-64df-4f9c-9718-d121ba50a8cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892495-rwvhk" Nov 01 16:15:06.746244 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.746215 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1191cd6c-176c-4f44-97c3-06942c21b3d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-9e044c-29892495-pvbf4\" (UID: \"1191cd6c-176c-4f44-97c3-06942c21b3d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e044c-29892495-pvbf4" Nov 01 16:15:06.746357 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.746330 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/381f38b2-e936-4b33-8cf1-121a2607045a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-43fac8-29892495-wgnf7\" (UID: \"381f38b2-e936-4b33-8cf1-121a2607045a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892495-wgnf7" Nov 01 16:15:06.746430 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.746416 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5pf2\" (UniqueName: \"kubernetes.io/projected/381f38b2-e936-4b33-8cf1-121a2607045a-kube-api-access-b5pf2\") pod \"maas-api-key-cleanup-e2e-worker-w6-43fac8-29892495-wgnf7\" (UID: \"381f38b2-e936-4b33-8cf1-121a2607045a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892495-wgnf7" Nov 01 16:15:06.746487 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.746440 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj2xc\" (UniqueName: \"kubernetes.io/projected/1191cd6c-176c-4f44-97c3-06942c21b3d9-kube-api-access-qj2xc\") pod \"maas-api-key-cleanup-e2e-inf-a-9e044c-29892495-pvbf4\" (UID: \"1191cd6c-176c-4f44-97c3-06942c21b3d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e044c-29892495-pvbf4" Nov 01 16:15:06.746487 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.746470 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff730377-456e-4937-9f9e-f3102f6c3f6a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-2ce08a-29892495-znkts\" (UID: \"ff730377-456e-4937-9f9e-f3102f6c3f6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2ce08a-29892495-znkts" Nov 01 16:15:06.746601 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.746496 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hgtp\" (UniqueName: \"kubernetes.io/projected/876c5b6a-7cab-4f19-8300-0fb7624d3e8f-kube-api-access-6hgtp\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfff59-29892495-wfnfr\" (UID: \"876c5b6a-7cab-4f19-8300-0fb7624d3e8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892495-wfnfr" Nov 01 16:15:06.746601 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.746529 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs6zn\" (UniqueName: \"kubernetes.io/projected/896ad84c-84b6-4f21-b2b8-9689578e43c9-kube-api-access-vs6zn\") pod \"maas-api-key-cleanup-29892495-2nh9k\" (UID: \"896ad84c-84b6-4f21-b2b8-9689578e43c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892495-2nh9k" Nov 01 16:15:06.746707 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.746601 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/876c5b6a-7cab-4f19-8300-0fb7624d3e8f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfff59-29892495-wfnfr\" (UID: \"876c5b6a-7cab-4f19-8300-0fb7624d3e8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892495-wfnfr" Nov 01 16:15:06.746707 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.746626 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn97q\" (UniqueName: \"kubernetes.io/projected/ff730377-456e-4937-9f9e-f3102f6c3f6a-kube-api-access-fn97q\") pod \"maas-api-key-cleanup-e2e-inf-b-2ce08a-29892495-znkts\" (UID: \"ff730377-456e-4937-9f9e-f3102f6c3f6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2ce08a-29892495-znkts" Nov 01 16:15:06.746707 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.746653 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w4cw\" (UniqueName: \"kubernetes.io/projected/cecab247-7b67-4a90-a3fd-99dfde6c8994-kube-api-access-6w4cw\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892495-v75rj\" (UID: \"cecab247-7b67-4a90-a3fd-99dfde6c8994\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892495-v75rj" Nov 01 16:15:06.746707 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.746683 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdl46\" (UniqueName: \"kubernetes.io/projected/a0918fbb-97c7-4653-ae3d-3de28dc050bf-kube-api-access-kdl46\") pod \"maas-api-key-cleanup-e2e-worker-w3-bd831a-29892495-xxk9n\" (UID: \"a0918fbb-97c7-4653-ae3d-3de28dc050bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892495-xxk9n" Nov 01 16:15:06.746707 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.746705 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhr9v\" (UniqueName: \"kubernetes.io/projected/a4377fa0-6280-4da7-b589-2afd643d3bce-kube-api-access-vhr9v\") pod \"maas-api-key-cleanup-e2e-inf-b-99e837-29892495-lnsd2\" (UID: \"a4377fa0-6280-4da7-b589-2afd643d3bce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99e837-29892495-lnsd2" Nov 01 16:15:06.746981 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.746742 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4377fa0-6280-4da7-b589-2afd643d3bce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-99e837-29892495-lnsd2\" (UID: \"a4377fa0-6280-4da7-b589-2afd643d3bce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99e837-29892495-lnsd2" Nov 01 16:15:06.747327 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.747306 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4377fa0-6280-4da7-b589-2afd643d3bce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-99e837-29892495-lnsd2\" (UID: \"a4377fa0-6280-4da7-b589-2afd643d3bce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99e837-29892495-lnsd2" Nov 01 16:15:06.752955 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.752937 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892495-wfnfr"] Nov 01 16:15:06.759062 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.759041 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv"] Nov 01 16:15:06.765344 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.765066 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-lgmlv"] Nov 01 16:15:06.768777 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.768747 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhr9v\" (UniqueName: \"kubernetes.io/projected/a4377fa0-6280-4da7-b589-2afd643d3bce-kube-api-access-vhr9v\") pod \"maas-api-key-cleanup-e2e-inf-b-99e837-29892495-lnsd2\" (UID: \"a4377fa0-6280-4da7-b589-2afd643d3bce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99e837-29892495-lnsd2" Nov 01 16:15:06.770496 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.770374 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj2xc\" (UniqueName: \"kubernetes.io/projected/1191cd6c-176c-4f44-97c3-06942c21b3d9-kube-api-access-qj2xc\") pod \"maas-api-key-cleanup-e2e-inf-a-9e044c-29892495-pvbf4\" (UID: \"1191cd6c-176c-4f44-97c3-06942c21b3d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e044c-29892495-pvbf4" Nov 01 16:15:06.773332 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.773257 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4240a-29892495-z49zr"] Nov 01 16:15:06.783890 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.783798 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99e837-29892495-lnsd2"] Nov 01 16:15:06.914039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.897633 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e044c-29892495-pvbf4" Nov 01 16:15:06.914039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.904582 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99e837-29892495-lnsd2" Nov 01 16:15:06.941102 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.940152 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn97q\" (UniqueName: \"kubernetes.io/projected/ff730377-456e-4937-9f9e-f3102f6c3f6a-kube-api-access-fn97q\") pod \"maas-api-key-cleanup-e2e-inf-b-2ce08a-29892495-znkts\" (UID: \"ff730377-456e-4937-9f9e-f3102f6c3f6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2ce08a-29892495-znkts" Nov 01 16:15:06.941363 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.941344 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w4cw\" (UniqueName: \"kubernetes.io/projected/cecab247-7b67-4a90-a3fd-99dfde6c8994-kube-api-access-6w4cw\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892495-v75rj\" (UID: \"cecab247-7b67-4a90-a3fd-99dfde6c8994\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892495-v75rj" Nov 01 16:15:06.941518 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.941503 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdl46\" (UniqueName: \"kubernetes.io/projected/a0918fbb-97c7-4653-ae3d-3de28dc050bf-kube-api-access-kdl46\") pod \"maas-api-key-cleanup-e2e-worker-w3-bd831a-29892495-xxk9n\" (UID: \"a0918fbb-97c7-4653-ae3d-3de28dc050bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892495-xxk9n" Nov 01 16:15:06.941918 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.941898 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/896ad84c-84b6-4f21-b2b8-9689578e43c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892495-2nh9k\" (UID: \"896ad84c-84b6-4f21-b2b8-9689578e43c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892495-2nh9k" Nov 01 16:15:06.942163 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.942143 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/689843e8-64df-4f9c-9718-d121ba50a8cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892495-rwvhk\" (UID: \"689843e8-64df-4f9c-9718-d121ba50a8cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892495-rwvhk" Nov 01 16:15:06.942292 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.942277 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cecab247-7b67-4a90-a3fd-99dfde6c8994-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892495-v75rj\" (UID: \"cecab247-7b67-4a90-a3fd-99dfde6c8994\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892495-v75rj" Nov 01 16:15:06.942415 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.942400 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsrtr\" (UniqueName: \"kubernetes.io/projected/2a839fcd-b668-4ae2-912c-fc2c812c4a55-kube-api-access-tsrtr\") pod \"maas-api-key-cleanup-e2e-inf-a-a4240a-29892495-z49zr\" (UID: \"2a839fcd-b668-4ae2-912c-fc2c812c4a55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4240a-29892495-z49zr" Nov 01 16:15:06.942553 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.942535 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a839fcd-b668-4ae2-912c-fc2c812c4a55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a4240a-29892495-z49zr\" (UID: \"2a839fcd-b668-4ae2-912c-fc2c812c4a55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4240a-29892495-z49zr" Nov 01 16:15:06.942692 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.942675 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0918fbb-97c7-4653-ae3d-3de28dc050bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bd831a-29892495-xxk9n\" (UID: \"a0918fbb-97c7-4653-ae3d-3de28dc050bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892495-xxk9n" Nov 01 16:15:06.942893 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.942872 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tblg5\" (UniqueName: \"kubernetes.io/projected/689843e8-64df-4f9c-9718-d121ba50a8cb-kube-api-access-tblg5\") pod \"maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892495-rwvhk\" (UID: \"689843e8-64df-4f9c-9718-d121ba50a8cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892495-rwvhk" Nov 01 16:15:06.943098 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.943078 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/381f38b2-e936-4b33-8cf1-121a2607045a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-43fac8-29892495-wgnf7\" (UID: \"381f38b2-e936-4b33-8cf1-121a2607045a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892495-wgnf7" Nov 01 16:15:06.943354 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.943332 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5pf2\" (UniqueName: \"kubernetes.io/projected/381f38b2-e936-4b33-8cf1-121a2607045a-kube-api-access-b5pf2\") pod \"maas-api-key-cleanup-e2e-worker-w6-43fac8-29892495-wgnf7\" (UID: \"381f38b2-e936-4b33-8cf1-121a2607045a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892495-wgnf7" Nov 01 16:15:06.943519 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.943500 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff730377-456e-4937-9f9e-f3102f6c3f6a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-2ce08a-29892495-znkts\" (UID: \"ff730377-456e-4937-9f9e-f3102f6c3f6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2ce08a-29892495-znkts" Nov 01 16:15:06.943678 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.943659 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hgtp\" (UniqueName: \"kubernetes.io/projected/876c5b6a-7cab-4f19-8300-0fb7624d3e8f-kube-api-access-6hgtp\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfff59-29892495-wfnfr\" (UID: \"876c5b6a-7cab-4f19-8300-0fb7624d3e8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892495-wfnfr" Nov 01 16:15:06.943851 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.943834 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs6zn\" (UniqueName: \"kubernetes.io/projected/896ad84c-84b6-4f21-b2b8-9689578e43c9-kube-api-access-vs6zn\") pod \"maas-api-key-cleanup-29892495-2nh9k\" (UID: \"896ad84c-84b6-4f21-b2b8-9689578e43c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892495-2nh9k" Nov 01 16:15:06.944139 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.944119 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/876c5b6a-7cab-4f19-8300-0fb7624d3e8f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfff59-29892495-wfnfr\" (UID: \"876c5b6a-7cab-4f19-8300-0fb7624d3e8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892495-wfnfr" Nov 01 16:15:06.944606 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.944577 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/689843e8-64df-4f9c-9718-d121ba50a8cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892495-rwvhk\" (UID: \"689843e8-64df-4f9c-9718-d121ba50a8cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892495-rwvhk" Nov 01 16:15:06.946077 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.942260 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:15:06.946334 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.946301 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:15:06.946500 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.943857 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/896ad84c-84b6-4f21-b2b8-9689578e43c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892495-2nh9k\" (UID: \"896ad84c-84b6-4f21-b2b8-9689578e43c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892495-2nh9k" Nov 01 16:15:06.946578 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.946066 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff730377-456e-4937-9f9e-f3102f6c3f6a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-2ce08a-29892495-znkts\" (UID: \"ff730377-456e-4937-9f9e-f3102f6c3f6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2ce08a-29892495-znkts" Nov 01 16:15:06.946714 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.946690 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:15:06.947577 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.945158 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/876c5b6a-7cab-4f19-8300-0fb7624d3e8f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfff59-29892495-wfnfr\" (UID: \"876c5b6a-7cab-4f19-8300-0fb7624d3e8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892495-wfnfr" Nov 01 16:15:06.948664 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.947919 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cecab247-7b67-4a90-a3fd-99dfde6c8994-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892495-v75rj\" (UID: \"cecab247-7b67-4a90-a3fd-99dfde6c8994\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892495-v75rj" Nov 01 16:15:06.948784 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.947964 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/381f38b2-e936-4b33-8cf1-121a2607045a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-43fac8-29892495-wgnf7\" (UID: \"381f38b2-e936-4b33-8cf1-121a2607045a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892495-wgnf7" Nov 01 16:15:06.948895 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.945418 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0918fbb-97c7-4653-ae3d-3de28dc050bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bd831a-29892495-xxk9n\" (UID: \"a0918fbb-97c7-4653-ae3d-3de28dc050bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892495-xxk9n" Nov 01 16:15:06.953699 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.953670 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdl46\" (UniqueName: \"kubernetes.io/projected/a0918fbb-97c7-4653-ae3d-3de28dc050bf-kube-api-access-kdl46\") pod \"maas-api-key-cleanup-e2e-worker-w3-bd831a-29892495-xxk9n\" (UID: \"a0918fbb-97c7-4653-ae3d-3de28dc050bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892495-xxk9n" Nov 01 16:15:06.971274 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.971235 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn97q\" (UniqueName: \"kubernetes.io/projected/ff730377-456e-4937-9f9e-f3102f6c3f6a-kube-api-access-fn97q\") pod \"maas-api-key-cleanup-e2e-inf-b-2ce08a-29892495-znkts\" (UID: \"ff730377-456e-4937-9f9e-f3102f6c3f6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2ce08a-29892495-znkts" Nov 01 16:15:06.971978 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.971751 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c" path="/var/lib/kubelet/pods/74a60e78-8fd2-43a1-9a8d-a1d13ee76a5c/volumes" Nov 01 16:15:06.973069 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.972819 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5pf2\" (UniqueName: \"kubernetes.io/projected/381f38b2-e936-4b33-8cf1-121a2607045a-kube-api-access-b5pf2\") pod \"maas-api-key-cleanup-e2e-worker-w6-43fac8-29892495-wgnf7\" (UID: \"381f38b2-e936-4b33-8cf1-121a2607045a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892495-wgnf7" Nov 01 16:15:06.974417 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.973591 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w4cw\" (UniqueName: \"kubernetes.io/projected/cecab247-7b67-4a90-a3fd-99dfde6c8994-kube-api-access-6w4cw\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892495-v75rj\" (UID: \"cecab247-7b67-4a90-a3fd-99dfde6c8994\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892495-v75rj" Nov 01 16:15:06.986397 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.984717 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hgtp\" (UniqueName: \"kubernetes.io/projected/876c5b6a-7cab-4f19-8300-0fb7624d3e8f-kube-api-access-6hgtp\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfff59-29892495-wfnfr\" (UID: \"876c5b6a-7cab-4f19-8300-0fb7624d3e8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892495-wfnfr" Nov 01 16:15:06.987443 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.987423 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892495-xxk9n" Nov 01 16:15:06.988509 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.988411 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs6zn\" (UniqueName: \"kubernetes.io/projected/896ad84c-84b6-4f21-b2b8-9689578e43c9-kube-api-access-vs6zn\") pod \"maas-api-key-cleanup-29892495-2nh9k\" (UID: \"896ad84c-84b6-4f21-b2b8-9689578e43c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892495-2nh9k" Nov 01 16:15:06.989203 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.988856 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892495-v75rj" Nov 01 16:15:06.990457 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:06.990431 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tblg5\" (UniqueName: \"kubernetes.io/projected/689843e8-64df-4f9c-9718-d121ba50a8cb-kube-api-access-tblg5\") pod \"maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892495-rwvhk\" (UID: \"689843e8-64df-4f9c-9718-d121ba50a8cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892495-rwvhk" Nov 01 16:15:07.010548 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:07.010522 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892495-wgnf7" Nov 01 16:15:07.024183 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:07.024152 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2ce08a-29892495-znkts" Nov 01 16:15:07.044152 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:07.043916 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892495-wfnfr" Nov 01 16:15:07.051102 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:07.047791 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsrtr\" (UniqueName: \"kubernetes.io/projected/2a839fcd-b668-4ae2-912c-fc2c812c4a55-kube-api-access-tsrtr\") pod \"maas-api-key-cleanup-e2e-inf-a-a4240a-29892495-z49zr\" (UID: \"2a839fcd-b668-4ae2-912c-fc2c812c4a55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4240a-29892495-z49zr" Nov 01 16:15:07.051102 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:07.047846 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a839fcd-b668-4ae2-912c-fc2c812c4a55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a4240a-29892495-z49zr\" (UID: \"2a839fcd-b668-4ae2-912c-fc2c812c4a55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4240a-29892495-z49zr" Nov 01 16:15:07.051452 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:07.051339 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a839fcd-b668-4ae2-912c-fc2c812c4a55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a4240a-29892495-z49zr\" (UID: \"2a839fcd-b668-4ae2-912c-fc2c812c4a55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4240a-29892495-z49zr" Nov 01 16:15:07.066363 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:07.063260 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv"] Nov 01 16:15:07.066363 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:07.066052 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsrtr\" (UniqueName: \"kubernetes.io/projected/2a839fcd-b668-4ae2-912c-fc2c812c4a55-kube-api-access-tsrtr\") pod \"maas-api-key-cleanup-e2e-inf-a-a4240a-29892495-z49zr\" (UID: \"2a839fcd-b668-4ae2-912c-fc2c812c4a55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4240a-29892495-z49zr" Nov 01 16:15:07.220990 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:07.220735 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892495-2nh9k" Nov 01 16:15:07.246637 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:07.245513 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892495-rwvhk" Nov 01 16:15:07.344815 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:07.344789 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" event={"ID":"af073176-cb1d-4eaf-9b6c-43787f6a7227","Type":"ContainerDied","Data":"a0c01501bd0056ad8e8e03a3cc55857877ed2861cd18040cc55a7076adafe40b"} Nov 01 16:15:07.345674 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:07.345658 4363 scope.go:122] "RemoveContainer" containerID="caf00e813eeb9ce335df97a58f7d406b7312c819f436d8d93cb7da42bcd6817d" Nov 01 16:15:07.345852 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:07.345333 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr" Nov 01 16:15:07.346080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:07.345599 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4240a-29892495-z49zr" Nov 01 16:15:07.375208 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:07.375179 4363 generic.go:356] "Generic (PLEG): container finished" podID="23798655-3db1-4155-9f07-7c46e4fc7a14" containerID="a0c6a86522b2aa01f5b55176dbc467ba847f72036ff9ec5cacac183d6d7faef8" exitCode=0 Nov 01 16:15:07.377792 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:07.375432 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" event={"ID":"23798655-3db1-4155-9f07-7c46e4fc7a14","Type":"ContainerDied","Data":"a0c6a86522b2aa01f5b55176dbc467ba847f72036ff9ec5cacac183d6d7faef8"} Nov 01 16:15:07.377792 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:07.376093 4363 scope.go:122] "RemoveContainer" containerID="a0c6a86522b2aa01f5b55176dbc467ba847f72036ff9ec5cacac183d6d7faef8" Nov 01 16:15:07.387857 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:07.386741 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" event={"ID":"768d7927-13bd-4fe9-984e-c5ac56bc4fdb","Type":"ContainerStarted","Data":"2e2c499dc24759b887be8c6339f77a616ee4e1ef1bc78d5eba9416ecbc58fe5d"} Nov 01 16:15:07.387857 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:07.386784 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" event={"ID":"768d7927-13bd-4fe9-984e-c5ac56bc4fdb","Type":"ContainerStarted","Data":"14f9d3c30258b49d3ca08dea9ba7e9292dd015dfe36fac4ec79307f2be674647"} Nov 01 16:15:07.387857 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:07.387752 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr"] Nov 01 16:15:07.389927 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:07.389898 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:15:07.390171 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:07.390148 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:15:07.390457 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:07.390342 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:15:07.407346 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:07.407318 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-wwkmr"] Nov 01 16:15:07.433338 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:07.430290 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" podStartSLOduration=66.430270774 podStartE2EDuration="1m6.430270774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:15:07.428008409 +0000 UTC m=+1640.759746057" watchObservedRunningTime="2026-11-01 16:15:07.430270774 +0000 UTC m=+1640.762008426" Nov 01 16:15:07.992167 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:15:07.992141 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1191cd6c_176c_4f44_97c3_06942c21b3d9.slice/crio-71ffedb77443120cc778e5822ddf13378e72525b9d00defc1b8bbd06a835c05f WatchSource:0}: Error finding container 71ffedb77443120cc778e5822ddf13378e72525b9d00defc1b8bbd06a835c05f: Status 404 returned error can't find the container with id 71ffedb77443120cc778e5822ddf13378e72525b9d00defc1b8bbd06a835c05f Nov 01 16:15:08.017626 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:08.014573 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e044c-29892495-pvbf4"] Nov 01 16:15:08.026711 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:08.026681 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" Nov 01 16:15:08.028495 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:08.027510 4363 patch_prober.go:29] interesting pod/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Nov 01 16:15:08.028644 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:08.028536 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" podUID="768d7927-13bd-4fe9-984e-c5ac56bc4fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Nov 01 16:15:08.170595 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:08.170535 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2ce08a-29892495-znkts"] Nov 01 16:15:08.267450 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:08.267423 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892495-xxk9n"] Nov 01 16:15:08.267932 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:15:08.267875 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4377fa0_6280_4da7_b589_2afd643d3bce.slice/crio-70de9a563cca52ff157fdcf8da2e820b45f5c9e63afa1166666962b5ebd711a9 WatchSource:0}: Error finding container 70de9a563cca52ff157fdcf8da2e820b45f5c9e63afa1166666962b5ebd711a9: Status 404 returned error can't find the container with id 70de9a563cca52ff157fdcf8da2e820b45f5c9e63afa1166666962b5ebd711a9 Nov 01 16:15:08.274819 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:08.274133 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99e837-29892495-lnsd2"] Nov 01 16:15:08.345611 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:08.345581 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892495-v75rj"] Nov 01 16:15:08.396330 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:08.380218 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892495-rwvhk"] Nov 01 16:15:08.527866 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:08.513085 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892495-wfnfr"] Nov 01 16:15:08.630884 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:08.585741 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892495-wgnf7"] Nov 01 16:15:08.630884 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:08.593776 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4240a-29892495-z49zr"] Nov 01 16:15:08.630884 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:08.600195 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892495-2nh9k"] Nov 01 16:15:08.645990 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:08.645653 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99e837-29892495-lnsd2" event={"ID":"a4377fa0-6280-4da7-b589-2afd643d3bce","Type":"ContainerStarted","Data":"70de9a563cca52ff157fdcf8da2e820b45f5c9e63afa1166666962b5ebd711a9"} Nov 01 16:15:08.647592 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:08.647266 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892495-rwvhk" event={"ID":"689843e8-64df-4f9c-9718-d121ba50a8cb","Type":"ContainerStarted","Data":"4f4580b112ba3659e0f6a1c03e79da755924d0d2a48d326d2893fb615ae6df73"} Nov 01 16:15:08.648900 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:08.648876 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892495-wfnfr" event={"ID":"876c5b6a-7cab-4f19-8300-0fb7624d3e8f","Type":"ContainerStarted","Data":"d0e9f18d7080262ecd64e1c649b7b49c255e17f09e8b6a5de4960de91da92ac7"} Nov 01 16:15:08.651670 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:08.651349 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" event={"ID":"23798655-3db1-4155-9f07-7c46e4fc7a14","Type":"ContainerStarted","Data":"a100941674dd5b6bdc0a4cc87c9dd372d7403e1afe33e84345f6c1b79f5acbdc"} Nov 01 16:15:08.653552 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:08.653519 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892495-v75rj" event={"ID":"cecab247-7b67-4a90-a3fd-99dfde6c8994","Type":"ContainerStarted","Data":"8f3be7c9b2c7a2a2539cb9d0c9ba7a8dd216d965216f6bf969224b8afb5c54dd"} Nov 01 16:15:08.655645 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:08.655062 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2ce08a-29892495-znkts" event={"ID":"ff730377-456e-4937-9f9e-f3102f6c3f6a","Type":"ContainerStarted","Data":"38c3cab7d6f29ba2c55cf130b206d6e78f67e5449893eae74665859d25db9871"} Nov 01 16:15:08.659375 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:08.659348 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e044c-29892495-pvbf4" event={"ID":"1191cd6c-176c-4f44-97c3-06942c21b3d9","Type":"ContainerStarted","Data":"71ffedb77443120cc778e5822ddf13378e72525b9d00defc1b8bbd06a835c05f"} Nov 01 16:15:08.665057 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:08.665032 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892495-xxk9n" event={"ID":"a0918fbb-97c7-4653-ae3d-3de28dc050bf","Type":"ContainerStarted","Data":"dcd9c42c934528824b00698cd420d4afd85676dcd985af2990980df9540c1015"} Nov 01 16:15:08.739046 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:15:08.738993 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod896ad84c_84b6_4f21_b2b8_9689578e43c9.slice/crio-47fe2cc5043e0c866887fb95b3bd471e692dbac64680b32d9391f0b7df4340eb WatchSource:0}: Error finding container 47fe2cc5043e0c866887fb95b3bd471e692dbac64680b32d9391f0b7df4340eb: Status 404 returned error can't find the container with id 47fe2cc5043e0c866887fb95b3bd471e692dbac64680b32d9391f0b7df4340eb Nov 01 16:15:08.836242 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:15:08.834584 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a839fcd_b668_4ae2_912c_fc2c812c4a55.slice/crio-15d6ad71ac6165bcc9af44e0eb0a96101281fdab808c6717767fefa3299b16f5 WatchSource:0}: Error finding container 15d6ad71ac6165bcc9af44e0eb0a96101281fdab808c6717767fefa3299b16f5: Status 404 returned error can't find the container with id 15d6ad71ac6165bcc9af44e0eb0a96101281fdab808c6717767fefa3299b16f5 Nov 01 16:15:08.866157 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:08.866088 4363 patch_prober.go:29] interesting pod/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 01 16:15:08.866295 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:08.866219 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" podUID="23798655-3db1-4155-9f07-7c46e4fc7a14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 01 16:15:09.028172 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:09.027769 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af073176-cb1d-4eaf-9b6c-43787f6a7227" path="/var/lib/kubelet/pods/af073176-cb1d-4eaf-9b6c-43787f6a7227/volumes" Nov 01 16:15:09.069660 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:09.028408 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" Nov 01 16:15:09.069660 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:09.041581 4363 patch_prober.go:29] interesting pod/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Nov 01 16:15:09.069660 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:09.063326 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" podUID="768d7927-13bd-4fe9-984e-c5ac56bc4fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Nov 01 16:15:09.676067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:09.676033 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2ce08a-29892495-znkts" event={"ID":"ff730377-456e-4937-9f9e-f3102f6c3f6a","Type":"ContainerStarted","Data":"01a7a44b32ed46964a61730b2c9f6e08e56a49cf5c950659796ec7725dd84aa7"} Nov 01 16:15:09.763177 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:09.754800 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2ce08a-29892495-znkts" podStartSLOduration=9.754778674 podStartE2EDuration="9.754778674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:15:09.715924342 +0000 UTC m=+1643.047661993" watchObservedRunningTime="2026-11-01 16:15:09.754778674 +0000 UTC m=+1643.086516325" Nov 01 16:15:09.763177 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:09.759885 4363 generic.go:356] "Generic (PLEG): container finished" podID="1191cd6c-176c-4f44-97c3-06942c21b3d9" containerID="6dd1acfdf17d4d378fd49e5391172dcd76f879400d6f198f6b75e1be3b78dd73" exitCode=7 Nov 01 16:15:09.763177 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:09.759998 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e044c-29892495-pvbf4" event={"ID":"1191cd6c-176c-4f44-97c3-06942c21b3d9","Type":"ContainerDied","Data":"6dd1acfdf17d4d378fd49e5391172dcd76f879400d6f198f6b75e1be3b78dd73"} Nov 01 16:15:09.763177 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:09.760785 4363 scope.go:122] "RemoveContainer" containerID="6dd1acfdf17d4d378fd49e5391172dcd76f879400d6f198f6b75e1be3b78dd73" Nov 01 16:15:09.763792 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:09.763240 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892495-2nh9k" event={"ID":"896ad84c-84b6-4f21-b2b8-9689578e43c9","Type":"ContainerStarted","Data":"47fe2cc5043e0c866887fb95b3bd471e692dbac64680b32d9391f0b7df4340eb"} Nov 01 16:15:09.766576 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:09.766555 4363 generic.go:356] "Generic (PLEG): container finished" podID="22eae6cc-4752-4174-a0f0-573adc82e32b" containerID="9b3bbf5f34103ac74f32d032cca159897c7f64f6dff7ccea3bd6567ccf289c19" exitCode=0 Nov 01 16:15:09.766738 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:09.766587 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" event={"ID":"22eae6cc-4752-4174-a0f0-573adc82e32b","Type":"ContainerDied","Data":"9b3bbf5f34103ac74f32d032cca159897c7f64f6dff7ccea3bd6567ccf289c19"} Nov 01 16:15:09.767164 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:09.767091 4363 scope.go:122] "RemoveContainer" containerID="9b3bbf5f34103ac74f32d032cca159897c7f64f6dff7ccea3bd6567ccf289c19" Nov 01 16:15:09.768244 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:09.768191 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892495-wgnf7" event={"ID":"381f38b2-e936-4b33-8cf1-121a2607045a","Type":"ContainerStarted","Data":"7892cff2a4a5d7d823033914cb2607b616ce52cfb68447c2cc0827611fafd7c1"} Nov 01 16:15:09.769763 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:09.769743 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4240a-29892495-z49zr" event={"ID":"2a839fcd-b668-4ae2-912c-fc2c812c4a55","Type":"ContainerStarted","Data":"15d6ad71ac6165bcc9af44e0eb0a96101281fdab808c6717767fefa3299b16f5"} Nov 01 16:15:09.850085 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:09.845262 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:15:09.850085 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:09.845364 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:15:09.850085 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:09.845416 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:15:09.861855 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:09.861828 4363 patch_prober.go:29] interesting pod/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 01 16:15:09.861998 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:09.861877 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" podUID="23798655-3db1-4155-9f07-7c46e4fc7a14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 01 16:15:10.037695 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:10.027776 4363 patch_prober.go:29] interesting pod/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Nov 01 16:15:10.037695 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:10.027833 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" podUID="768d7927-13bd-4fe9-984e-c5ac56bc4fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Nov 01 16:15:10.136241 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:10.134119 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" Nov 01 16:15:10.876360 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:10.876319 4363 patch_prober.go:29] interesting pod/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 01 16:15:10.876860 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:10.876825 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" podUID="23798655-3db1-4155-9f07-7c46e4fc7a14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 01 16:15:10.956468 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:10.956372 4363 generic.go:356] "Generic (PLEG): container finished" podID="ff730377-456e-4937-9f9e-f3102f6c3f6a" containerID="01a7a44b32ed46964a61730b2c9f6e08e56a49cf5c950659796ec7725dd84aa7" exitCode=60 Nov 01 16:15:10.957338 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:10.957312 4363 scope.go:122] "RemoveContainer" containerID="01a7a44b32ed46964a61730b2c9f6e08e56a49cf5c950659796ec7725dd84aa7" Nov 01 16:15:11.001684 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:10.968622 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2ce08a-29892495-znkts" event={"ID":"ff730377-456e-4937-9f9e-f3102f6c3f6a","Type":"ContainerDied","Data":"01a7a44b32ed46964a61730b2c9f6e08e56a49cf5c950659796ec7725dd84aa7"} Nov 01 16:15:11.075041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:11.059656 4363 patch_prober.go:29] interesting pod/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Nov 01 16:15:11.075041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:11.059717 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" podUID="768d7927-13bd-4fe9-984e-c5ac56bc4fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Nov 01 16:15:11.145038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:11.133690 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892495-xxk9n" event={"ID":"a0918fbb-97c7-4653-ae3d-3de28dc050bf","Type":"ContainerStarted","Data":"f305f7d4caec8fa8c9dc3504dc2b51cdb079d3a8a5a3c83d6928d89987863e1d"} Nov 01 16:15:11.161436 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:11.161333 4363 generic.go:356] "Generic (PLEG): container finished" podID="a4377fa0-6280-4da7-b589-2afd643d3bce" containerID="a5cbbca8a335687db39b7a8c61462f619b9c327d981f91905bcec6f57e7058ba" exitCode=60 Nov 01 16:15:11.161521 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:11.161482 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99e837-29892495-lnsd2" event={"ID":"a4377fa0-6280-4da7-b589-2afd643d3bce","Type":"ContainerDied","Data":"a5cbbca8a335687db39b7a8c61462f619b9c327d981f91905bcec6f57e7058ba"} Nov 01 16:15:11.163543 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:11.163520 4363 scope.go:122] "RemoveContainer" containerID="a5cbbca8a335687db39b7a8c61462f619b9c327d981f91905bcec6f57e7058ba" Nov 01 16:15:11.172659 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:11.172627 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892495-rwvhk" event={"ID":"689843e8-64df-4f9c-9718-d121ba50a8cb","Type":"ContainerStarted","Data":"1701f18e4dd329c42f1818a138c56981685203602aec2c61fcb635ed1b27a8f1"} Nov 01 16:15:11.183876 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:11.181640 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" event={"ID":"22eae6cc-4752-4174-a0f0-573adc82e32b","Type":"ContainerStarted","Data":"931a83ee2d5ef1d73bdeec852b45c33a754b9fe3ca32df298f4970f18e95938c"} Nov 01 16:15:11.186849 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:11.186820 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892495-wfnfr" event={"ID":"876c5b6a-7cab-4f19-8300-0fb7624d3e8f","Type":"ContainerStarted","Data":"972d69bdc1778973c86691f0379d0e14e924479c70a2e3fa8295b31f8830278d"} Nov 01 16:15:11.202790 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:11.202735 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892495-xxk9n" podStartSLOduration=11.202715044 podStartE2EDuration="11.202715044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:15:11.179563079 +0000 UTC m=+1644.511300731" watchObservedRunningTime="2026-11-01 16:15:11.202715044 +0000 UTC m=+1644.534452694" Nov 01 16:15:11.275212 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:11.274083 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4240a-29892495-z49zr" event={"ID":"2a839fcd-b668-4ae2-912c-fc2c812c4a55","Type":"ContainerStarted","Data":"4210cb5d9ff1cdf6c4763debde79b570d914b7db4ec6d82c730976157648bd82"} Nov 01 16:15:11.319102 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:11.297996 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892495-v75rj" event={"ID":"cecab247-7b67-4a90-a3fd-99dfde6c8994","Type":"ContainerStarted","Data":"a50da91d9eae6bd9d78be9b6a4243ecb856f89e4892b708a3b1149e2c42cb5b6"} Nov 01 16:15:11.319102 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:11.314494 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892495-rwvhk" podStartSLOduration=11.314473191 podStartE2EDuration="11.314473191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:15:11.225407983 +0000 UTC m=+1644.557145634" watchObservedRunningTime="2026-11-01 16:15:11.314473191 +0000 UTC m=+1644.646210842" Nov 01 16:15:11.378931 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:11.378854 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892495-wfnfr" podStartSLOduration=11.378836037 podStartE2EDuration="11.378836037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:15:11.304470703 +0000 UTC m=+1644.636208355" watchObservedRunningTime="2026-11-01 16:15:11.378836037 +0000 UTC m=+1644.710573687" Nov 01 16:15:11.899287 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:11.869379 4363 patch_prober.go:29] interesting pod/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 01 16:15:11.899287 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:11.869435 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" podUID="23798655-3db1-4155-9f07-7c46e4fc7a14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 01 16:15:11.899287 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:11.869812 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" Nov 01 16:15:11.899287 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:11.870921 4363 patch_prober.go:29] interesting pod/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Nov 01 16:15:11.899287 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:11.870964 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" podUID="22eae6cc-4752-4174-a0f0-573adc82e32b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Nov 01 16:15:12.080170 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:12.071896 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4240a-29892495-z49zr" podStartSLOduration=12.071873736 podStartE2EDuration="12.071873736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:15:11.332484363 +0000 UTC m=+1644.664222014" watchObservedRunningTime="2026-11-01 16:15:12.071873736 +0000 UTC m=+1645.403611389" Nov 01 16:15:12.173038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:12.170067 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892495-v75rj" podStartSLOduration=12.170048419 podStartE2EDuration="12.170048419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:15:11.61938978 +0000 UTC m=+1644.951127420" watchObservedRunningTime="2026-11-01 16:15:12.170048419 +0000 UTC m=+1645.501786070" Nov 01 16:15:12.189057 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:12.186226 4363 patch_prober.go:29] interesting pod/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Nov 01 16:15:12.189057 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:12.186301 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" podUID="768d7927-13bd-4fe9-984e-c5ac56bc4fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Nov 01 16:15:12.504383 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:12.504332 4363 generic.go:356] "Generic (PLEG): container finished" podID="381f38b2-e936-4b33-8cf1-121a2607045a" containerID="e8e5cc6140c89df9f578f45977a7d8d0fadeebf5e0584b5daa81035b4a697e08" exitCode=60 Nov 01 16:15:12.521961 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:12.521930 4363 generic.go:356] "Generic (PLEG): container finished" podID="2a839fcd-b668-4ae2-912c-fc2c812c4a55" containerID="4210cb5d9ff1cdf6c4763debde79b570d914b7db4ec6d82c730976157648bd82" exitCode=60 Nov 01 16:15:12.523140 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:12.523119 4363 scope.go:122] "RemoveContainer" containerID="4210cb5d9ff1cdf6c4763debde79b570d914b7db4ec6d82c730976157648bd82" Nov 01 16:15:12.524456 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:12.524425 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892495-wgnf7" event={"ID":"381f38b2-e936-4b33-8cf1-121a2607045a","Type":"ContainerDied","Data":"e8e5cc6140c89df9f578f45977a7d8d0fadeebf5e0584b5daa81035b4a697e08"} Nov 01 16:15:12.524759 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:12.524732 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4240a-29892495-z49zr" event={"ID":"2a839fcd-b668-4ae2-912c-fc2c812c4a55","Type":"ContainerDied","Data":"4210cb5d9ff1cdf6c4763debde79b570d914b7db4ec6d82c730976157648bd82"} Nov 01 16:15:12.525461 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:12.525434 4363 scope.go:122] "RemoveContainer" containerID="e8e5cc6140c89df9f578f45977a7d8d0fadeebf5e0584b5daa81035b4a697e08" Nov 01 16:15:12.528373 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:12.528342 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:15:12.564012 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:12.563962 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:15:12.580558 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:12.564592 4363 generic.go:356] "Generic (PLEG): container finished" podID="cecab247-7b67-4a90-a3fd-99dfde6c8994" containerID="a50da91d9eae6bd9d78be9b6a4243ecb856f89e4892b708a3b1149e2c42cb5b6" exitCode=60 Nov 01 16:15:12.583263 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:12.583222 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:15:12.594308 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:12.594270 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892495-v75rj" event={"ID":"cecab247-7b67-4a90-a3fd-99dfde6c8994","Type":"ContainerDied","Data":"a50da91d9eae6bd9d78be9b6a4243ecb856f89e4892b708a3b1149e2c42cb5b6"} Nov 01 16:15:12.663199 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:12.663167 4363 scope.go:122] "RemoveContainer" containerID="a50da91d9eae6bd9d78be9b6a4243ecb856f89e4892b708a3b1149e2c42cb5b6" Nov 01 16:15:12.668145 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:12.668109 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:15:12.732817 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:12.732765 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-99e837/test-model-99e837-kserve-68448d5ff7-hfsbc" podUID="62fde750-084e-488b-9d65-24b38a323cb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.185:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:15:12.747280 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:12.747241 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" podUID="9cbd41ee-3134-4b54-9ea1-0c13921269bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.181:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:15:12.868406 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:12.848429 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" podUID="a15aa191-e376-490a-8763-16cdd7685b5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": context deadline exceeded" Nov 01 16:15:12.938996 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:12.938955 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" podUID="37773d39-9d52-4b1d-94c7-68e02714a476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.171:8000/ready\": context deadline exceeded" Nov 01 16:15:12.943207 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:12.943178 4363 patch_prober.go:29] interesting pod/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 01 16:15:13.029047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:13.027488 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" podUID="23798655-3db1-4155-9f07-7c46e4fc7a14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 01 16:15:13.036048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:12.944138 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:15:13.055121 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:13.012962 4363 patch_prober.go:29] interesting pod/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Nov 01 16:15:13.055396 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:13.055363 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" podUID="22eae6cc-4752-4174-a0f0-573adc82e32b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Nov 01 16:15:13.092045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:13.088282 4363 patch_prober.go:29] interesting pod/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Nov 01 16:15:13.092045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:13.088344 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" podUID="768d7927-13bd-4fe9-984e-c5ac56bc4fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Nov 01 16:15:13.134042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:13.114722 4363 scope.go:122] "RemoveContainer" containerID="9fef4b9c73cecb711d5b26591c6433c78f30344096871596ea5ed4c43a89af82" Nov 01 16:15:13.134042 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:15:13.115306 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-9e044c-29892495-pvbf4_odh-ai-gateway-infra(1191cd6c-176c-4f44-97c3-06942c21b3d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e044c-29892495-pvbf4" podUID="1191cd6c-176c-4f44-97c3-06942c21b3d9" Nov 01 16:15:13.356794 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:13.356752 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e044c-29892495-pvbf4" event={"ID":"1191cd6c-176c-4f44-97c3-06942c21b3d9","Type":"ContainerStarted","Data":"9fef4b9c73cecb711d5b26591c6433c78f30344096871596ea5ed4c43a89af82"} Nov 01 16:15:13.385158 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:13.385124 4363 generic.go:356] "Generic (PLEG): container finished" podID="a0918fbb-97c7-4653-ae3d-3de28dc050bf" containerID="f305f7d4caec8fa8c9dc3504dc2b51cdb079d3a8a5a3c83d6928d89987863e1d" exitCode=60 Nov 01 16:15:13.385445 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:13.385422 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892495-xxk9n" event={"ID":"a0918fbb-97c7-4653-ae3d-3de28dc050bf","Type":"ContainerDied","Data":"f305f7d4caec8fa8c9dc3504dc2b51cdb079d3a8a5a3c83d6928d89987863e1d"} Nov 01 16:15:13.584641 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:13.583094 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" podUID="d92f7196-153d-46b0-9e5b-51667329cd02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.169:8000/ready\": context deadline exceeded" Nov 01 16:15:13.651037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:13.650690 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:15:13.659238 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:13.651369 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:15:13.659238 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:13.655363 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:15:13.659238 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:13.656686 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:15:13.659238 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:13.656742 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:15:13.659238 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:13.656775 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:15:13.662620 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:13.661927 4363 scope.go:122] "RemoveContainer" containerID="f305f7d4caec8fa8c9dc3504dc2b51cdb079d3a8a5a3c83d6928d89987863e1d" Nov 01 16:15:13.724088 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:13.704100 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:15:13.724088 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:13.721417 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:15:13.754876 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:13.753134 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-99e837/test-model-99e837-kserve-68448d5ff7-hfsbc" podUID="62fde750-084e-488b-9d65-24b38a323cb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.185:8000/ready\": context deadline exceeded" Nov 01 16:15:13.754876 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:13.753797 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-99e837/test-model-99e837-kserve-68448d5ff7-hfsbc" Nov 01 16:15:13.754876 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:13.753087 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" podUID="9cbd41ee-3134-4b54-9ea1-0c13921269bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.181:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:15:13.754876 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:13.754524 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" Nov 01 16:15:13.825345 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:13.810134 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892495-2nh9k" event={"ID":"896ad84c-84b6-4f21-b2b8-9689578e43c9","Type":"ContainerStarted","Data":"938f223c05db37d35963947683c72196aa9cef190863d34e8e39fc7cb8600447"} Nov 01 16:15:13.865069 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:13.839148 4363 scope.go:122] "RemoveContainer" containerID="938f223c05db37d35963947683c72196aa9cef190863d34e8e39fc7cb8600447" Nov 01 16:15:13.880117 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:13.874234 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-04f248/test-model-04f248-kserve-7cf9d4bf7f-kd4rz" podUID="3860672d-226a-47fd-a812-cca575bea518" containerName="main" probeResult="failure" output="Get \"https://10.134.0.186:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:15:13.880117 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:13.874281 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" podUID="a15aa191-e376-490a-8763-16cdd7685b5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:15:13.880117 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:13.874683 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" Nov 01 16:15:13.922079 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:13.883995 4363 patch_prober.go:29] interesting pod/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 01 16:15:13.922079 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:13.884073 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" podUID="23798655-3db1-4155-9f07-7c46e4fc7a14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 01 16:15:13.922079 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:13.885003 4363 patch_prober.go:29] interesting pod/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Nov 01 16:15:13.922079 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:13.885055 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" podUID="22eae6cc-4752-4174-a0f0-573adc82e32b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Nov 01 16:15:13.953088 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:13.938681 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:15:13.954680 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:13.954630 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" podUID="37773d39-9d52-4b1d-94c7-68e02714a476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.171:8000/ready\": context deadline exceeded" Nov 01 16:15:13.962062 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:13.955282 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" Nov 01 16:15:14.116155 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:14.084962 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:15:14.116155 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:14.085274 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:15:14.116155 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:14.085662 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 16:15:14.171064 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:14.126463 4363 patch_prober.go:29] interesting pod/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Nov 01 16:15:14.275188 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:14.275146 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" podUID="768d7927-13bd-4fe9-984e-c5ac56bc4fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Nov 01 16:15:14.452757 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:14.451536 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" Nov 01 16:15:14.507870 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:14.481552 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-04f248/test-model-04f248-kserve-7cf9d4bf7f-kd4rz" podUID="3860672d-226a-47fd-a812-cca575bea518" containerName="main" probeResult="failure" output="Get \"https://10.134.0.186:8000/health\": context deadline exceeded" Nov 01 16:15:14.594949 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:14.567262 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded" Nov 01 16:15:14.622346 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:14.621347 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" podUID="d92f7196-153d-46b0-9e5b-51667329cd02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.169:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:15:14.622346 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:14.621758 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" Nov 01 16:15:14.664283 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:14.664240 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:15:14.665846 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:14.665794 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:15:14.687199 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:14.687140 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:15:14.769649 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:14.753613 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-99e837/test-model-99e837-kserve-68448d5ff7-hfsbc" Nov 01 16:15:14.949651 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:14.946693 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" podUID="a15aa191-e376-490a-8763-16cdd7685b5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:15:15.052844 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:15.052799 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" podUID="37773d39-9d52-4b1d-94c7-68e02714a476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.171:8000/ready\": context deadline exceeded" Nov 01 16:15:15.067148 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:15.067113 4363 patch_prober.go:29] interesting pod/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 01 16:15:15.067381 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:15.067350 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" podUID="23798655-3db1-4155-9f07-7c46e4fc7a14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 01 16:15:15.067751 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:15.067726 4363 patch_prober.go:29] interesting pod/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Nov 01 16:15:15.074050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:15.067949 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" podUID="22eae6cc-4752-4174-a0f0-573adc82e32b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Nov 01 16:15:15.091203 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:15.091161 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": context deadline exceeded" Nov 01 16:15:15.121539 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:15.121494 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded" Nov 01 16:15:15.129186 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:15.129145 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:15:15.129706 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:15.129671 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:15:15.182396 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:15.182348 4363 patch_prober.go:29] interesting pod/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Nov 01 16:15:15.182690 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:15.182645 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" podUID="768d7927-13bd-4fe9-984e-c5ac56bc4fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Nov 01 16:15:15.461187 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:15.461140 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" podUID="9cbd41ee-3134-4b54-9ea1-0c13921269bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.181:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:15:15.745099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:15.691509 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" podUID="d92f7196-153d-46b0-9e5b-51667329cd02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.169:8000/ready\": context deadline exceeded" Nov 01 16:15:15.745099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:15.692119 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:15:15.745099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:15.692435 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:15:15.745099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:15.692820 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:15:15.777232 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:15.753781 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-99e837/test-model-99e837-kserve-68448d5ff7-hfsbc" podUID="62fde750-084e-488b-9d65-24b38a323cb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.185:8000/ready\": context deadline exceeded" Nov 01 16:15:15.903077 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:15.881902 4363 patch_prober.go:29] interesting pod/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 01 16:15:15.903077 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:15.881964 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" podUID="23798655-3db1-4155-9f07-7c46e4fc7a14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 01 16:15:15.903077 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:15.882397 4363 patch_prober.go:29] interesting pod/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Nov 01 16:15:15.903077 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:15.882452 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" podUID="22eae6cc-4752-4174-a0f0-573adc82e32b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Nov 01 16:15:16.027347 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:16.017111 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" podUID="a15aa191-e376-490a-8763-16cdd7685b5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": context deadline exceeded" Nov 01 16:15:16.043248 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:16.037743 4363 patch_prober.go:29] interesting pod/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Nov 01 16:15:16.043248 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:16.037799 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" podUID="768d7927-13bd-4fe9-984e-c5ac56bc4fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Nov 01 16:15:16.098103 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:16.076177 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" podUID="37773d39-9d52-4b1d-94c7-68e02714a476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.171:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:15:16.098103 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:16.077392 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-frphn" podUID="fa3886b7-e5e1-4f64-93d4-a097f446a835" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:15:16.138329 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:16.138271 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 01 16:15:16.142299 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:16.142267 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 01 16:15:16.202932 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:16.202889 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" podUID="37773d39-9d52-4b1d-94c7-68e02714a476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.171:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:15:16.474642 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:16.469142 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" podUID="9cbd41ee-3134-4b54-9ea1-0c13921269bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.181:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:15:16.560638 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:15:16.560601 4363 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.766s" Nov 01 16:15:16.560931 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:16.560911 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 16:15:16.561380 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:16.561359 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" Nov 01 16:15:16.561602 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:16.561587 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:15:16.567471 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:16.567450 4363 scope.go:122] "RemoveContainer" containerID="886b9459b11f0e6190e8d0002993fe3ed8fa448fad41a256ea6bc7f3f2d880c8" Nov 01 16:15:16.569950 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:15:16.569713 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a4240a-29892495-z49zr_odh-ai-gateway-infra(2a839fcd-b668-4ae2-912c-fc2c812c4a55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4240a-29892495-z49zr" podUID="2a839fcd-b668-4ae2-912c-fc2c812c4a55" Nov 01 16:15:16.639105 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:16.626808 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" Nov 01 16:15:16.661174 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:16.661131 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:15:16.695128 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:16.694754 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:15:16.744774 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:16.744693 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" podUID="d92f7196-153d-46b0-9e5b-51667329cd02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.169:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:15:16.745425 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:16.745391 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" podUID="d92f7196-153d-46b0-9e5b-51667329cd02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.169:8000/health\": context deadline exceeded" Nov 01 16:15:16.750680 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:16.750613 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:15:16.759004 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:16.758980 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 16:15:16.771354 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:16.771327 4363 scope.go:122] "RemoveContainer" containerID="e198ab11a3398898ec2ccb46ee7e9388622146ee2f8bc47245514796f818cc34" Nov 01 16:15:16.935462 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:16.822356 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-04f248/test-model-04f248-kserve-7cf9d4bf7f-kd4rz" podUID="3860672d-226a-47fd-a812-cca575bea518" containerName="main" probeResult="failure" output="Get \"https://10.134.0.186:8000/ready\": context deadline exceeded" Nov 01 16:15:16.944204 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:15:16.943577 4363 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff730377_456e_4937_9f9e_f3102f6c3f6a.slice/crio-928ce677b6baed29fac8c8449b46b4c30fd71b4c636ef316cfa13034fbc896c8.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4377fa0_6280_4da7_b589_2afd643d3bce.slice/crio-conmon-aaf9d05e7460bfef8653934af24b0726c7f0a96ba7c518d3f8a0b989c98c36b2.scope\": RecentStats: unable to find data in memory cache]" Nov 01 16:15:16.948717 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:16.948687 4363 generic.go:356] "Generic (PLEG): container finished" podID="ff730377-456e-4937-9f9e-f3102f6c3f6a" containerID="928ce677b6baed29fac8c8449b46b4c30fd71b4c636ef316cfa13034fbc896c8" exitCode=60 Nov 01 16:15:16.956731 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:16.956341 4363 generic.go:356] "Generic (PLEG): container finished" podID="1191cd6c-176c-4f44-97c3-06942c21b3d9" containerID="9fef4b9c73cecb711d5b26591c6433c78f30344096871596ea5ed4c43a89af82" exitCode=7 Nov 01 16:15:16.975547 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:16.975506 4363 patch_prober.go:29] interesting pod/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Nov 01 16:15:16.975807 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:16.975764 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" podUID="22eae6cc-4752-4174-a0f0-573adc82e32b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Nov 01 16:15:17.135618 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:17.115059 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" podUID="37773d39-9d52-4b1d-94c7-68e02714a476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.171:8000/ready\": context deadline exceeded" Nov 01 16:15:17.174534 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:17.174453 4363 generic.go:356] "Generic (PLEG): container finished" podID="896ad84c-84b6-4f21-b2b8-9689578e43c9" containerID="938f223c05db37d35963947683c72196aa9cef190863d34e8e39fc7cb8600447" exitCode=60 Nov 01 16:15:17.178664 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:17.178636 4363 generic.go:356] "Generic (PLEG): container finished" podID="a4377fa0-6280-4da7-b589-2afd643d3bce" containerID="aaf9d05e7460bfef8653934af24b0726c7f0a96ba7c518d3f8a0b989c98c36b2" exitCode=60 Nov 01 16:15:17.228324 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:17.225748 4363 scope.go:122] "RemoveContainer" containerID="9fef4b9c73cecb711d5b26591c6433c78f30344096871596ea5ed4c43a89af82" Nov 01 16:15:17.228324 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:15:17.226239 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-9e044c-29892495-pvbf4_odh-ai-gateway-infra(1191cd6c-176c-4f44-97c3-06942c21b3d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e044c-29892495-pvbf4" podUID="1191cd6c-176c-4f44-97c3-06942c21b3d9" Nov 01 16:15:17.228324 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:17.226559 4363 scope.go:122] "RemoveContainer" containerID="67d5e51f38ba092ec8f375e0fa71946572ed5eb7327e79010825f407b4b16f6d" Nov 01 16:15:17.228324 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:15:17.226924 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892495-v75rj_odh-ai-gateway-infra(cecab247-7b67-4a90-a3fd-99dfde6c8994)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892495-v75rj" podUID="cecab247-7b67-4a90-a3fd-99dfde6c8994" Nov 01 16:15:17.235290 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:17.232408 4363 scope.go:122] "RemoveContainer" containerID="aaf9d05e7460bfef8653934af24b0726c7f0a96ba7c518d3f8a0b989c98c36b2" Nov 01 16:15:17.235290 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:17.232407 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:15:17.235290 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:15:17.235237 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-99e837-29892495-lnsd2_odh-ai-gateway-infra(a4377fa0-6280-4da7-b589-2afd643d3bce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99e837-29892495-lnsd2" podUID="a4377fa0-6280-4da7-b589-2afd643d3bce" Nov 01 16:15:17.257769 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:17.237346 4363 scope.go:122] "RemoveContainer" containerID="928ce677b6baed29fac8c8449b46b4c30fd71b4c636ef316cfa13034fbc896c8" Nov 01 16:15:17.257769 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:15:17.237637 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-2ce08a-29892495-znkts_odh-ai-gateway-infra(ff730377-456e-4937-9f9e-f3102f6c3f6a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2ce08a-29892495-znkts" podUID="ff730377-456e-4937-9f9e-f3102f6c3f6a" Nov 01 16:15:17.257769 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:17.248949 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2ce08a-29892495-znkts" event={"ID":"ff730377-456e-4937-9f9e-f3102f6c3f6a","Type":"ContainerDied","Data":"928ce677b6baed29fac8c8449b46b4c30fd71b4c636ef316cfa13034fbc896c8"} Nov 01 16:15:17.257769 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:17.249002 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e044c-29892495-pvbf4" event={"ID":"1191cd6c-176c-4f44-97c3-06942c21b3d9","Type":"ContainerDied","Data":"9fef4b9c73cecb711d5b26591c6433c78f30344096871596ea5ed4c43a89af82"} Nov 01 16:15:17.257769 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:17.249048 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892495-2nh9k" event={"ID":"896ad84c-84b6-4f21-b2b8-9689578e43c9","Type":"ContainerDied","Data":"938f223c05db37d35963947683c72196aa9cef190863d34e8e39fc7cb8600447"} Nov 01 16:15:17.257769 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:17.249129 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" Nov 01 16:15:17.257769 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:17.249192 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" Nov 01 16:15:17.257769 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:17.249248 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" Nov 01 16:15:17.257769 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:17.249283 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:15:17.257769 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:17.249299 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99e837-29892495-lnsd2" event={"ID":"a4377fa0-6280-4da7-b589-2afd643d3bce","Type":"ContainerDied","Data":"aaf9d05e7460bfef8653934af24b0726c7f0a96ba7c518d3f8a0b989c98c36b2"} Nov 01 16:15:17.257769 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:17.249346 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" Nov 01 16:15:17.257769 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:17.249361 4363 scope.go:122] "RemoveContainer" containerID="01a7a44b32ed46964a61730b2c9f6e08e56a49cf5c950659796ec7725dd84aa7" Nov 01 16:15:17.257769 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:17.249381 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:15:17.257769 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:17.250502 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" Nov 01 16:15:17.261526 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:17.259949 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" Nov 01 16:15:17.261526 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:17.260938 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 16:15:17.261821 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:17.261734 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" Nov 01 16:15:17.262540 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:17.262518 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" Nov 01 16:15:17.267504 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:17.264790 4363 generic.go:356] "Generic (PLEG): container finished" podID="689843e8-64df-4f9c-9718-d121ba50a8cb" containerID="1701f18e4dd329c42f1818a138c56981685203602aec2c61fcb635ed1b27a8f1" exitCode=60 Nov 01 16:15:17.267504 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:17.264874 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892495-rwvhk" event={"ID":"689843e8-64df-4f9c-9718-d121ba50a8cb","Type":"ContainerDied","Data":"1701f18e4dd329c42f1818a138c56981685203602aec2c61fcb635ed1b27a8f1"} Nov 01 16:15:17.267504 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:17.265610 4363 scope.go:122] "RemoveContainer" containerID="1701f18e4dd329c42f1818a138c56981685203602aec2c61fcb635ed1b27a8f1" Nov 01 16:15:17.269835 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:17.269772 4363 generic.go:356] "Generic (PLEG): container finished" podID="876c5b6a-7cab-4f19-8300-0fb7624d3e8f" containerID="972d69bdc1778973c86691f0379d0e14e924479c70a2e3fa8295b31f8830278d" exitCode=60 Nov 01 16:15:17.270131 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:17.270083 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892495-wfnfr" event={"ID":"876c5b6a-7cab-4f19-8300-0fb7624d3e8f","Type":"ContainerDied","Data":"972d69bdc1778973c86691f0379d0e14e924479c70a2e3fa8295b31f8830278d"} Nov 01 16:15:17.270702 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:17.270669 4363 scope.go:122] "RemoveContainer" containerID="972d69bdc1778973c86691f0379d0e14e924479c70a2e3fa8295b31f8830278d" Nov 01 16:15:17.345296 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:17.345266 4363 scope.go:122] "RemoveContainer" containerID="6dd1acfdf17d4d378fd49e5391172dcd76f879400d6f198f6b75e1be3b78dd73" Nov 01 16:15:17.345621 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:17.345589 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892495-wgnf7" event={"ID":"381f38b2-e936-4b33-8cf1-121a2607045a","Type":"ContainerStarted","Data":"74e868ad10b281af60a888c0345e3ddd12d7ee4938f160b958c3190da0cce1ce"} Nov 01 16:15:17.348137 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:17.348114 4363 scope.go:122] "RemoveContainer" containerID="74e868ad10b281af60a888c0345e3ddd12d7ee4938f160b958c3190da0cce1ce" Nov 01 16:15:17.348622 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:15:17.348588 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-43fac8-29892495-wgnf7_odh-ai-gateway-infra(381f38b2-e936-4b33-8cf1-121a2607045a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892495-wgnf7" podUID="381f38b2-e936-4b33-8cf1-121a2607045a" Nov 01 16:15:17.472490 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:17.472393 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892495-2nh9k" podStartSLOduration=17.472377708 podStartE2EDuration="17.472377708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:15:17.471620572 +0000 UTC m=+1650.803358222" watchObservedRunningTime="2026-11-01 16:15:17.472377708 +0000 UTC m=+1650.804115358" Nov 01 16:15:17.535710 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:17.535681 4363 scope.go:122] "RemoveContainer" containerID="a5cbbca8a335687db39b7a8c61462f619b9c327d981f91905bcec6f57e7058ba" Nov 01 16:15:17.871091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:17.871061 4363 patch_prober.go:29] interesting pod/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Nov 01 16:15:17.871360 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:17.871328 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" podUID="22eae6cc-4752-4174-a0f0-573adc82e32b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Nov 01 16:15:17.954935 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:17.954911 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" Nov 01 16:15:18.345077 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:15:18.345009 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:15:18.356109 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:18.356078 4363 generic.go:356] "Generic (PLEG): container finished" podID="cecab247-7b67-4a90-a3fd-99dfde6c8994" containerID="67d5e51f38ba092ec8f375e0fa71946572ed5eb7327e79010825f407b4b16f6d" exitCode=60 Nov 01 16:15:18.356282 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:18.356159 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892495-v75rj" event={"ID":"cecab247-7b67-4a90-a3fd-99dfde6c8994","Type":"ContainerDied","Data":"67d5e51f38ba092ec8f375e0fa71946572ed5eb7327e79010825f407b4b16f6d"} Nov 01 16:15:18.356282 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:18.356230 4363 scope.go:122] "RemoveContainer" containerID="a50da91d9eae6bd9d78be9b6a4243ecb856f89e4892b708a3b1149e2c42cb5b6" Nov 01 16:15:18.356741 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:18.356722 4363 scope.go:122] "RemoveContainer" containerID="67d5e51f38ba092ec8f375e0fa71946572ed5eb7327e79010825f407b4b16f6d" Nov 01 16:15:18.357268 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:15:18.357245 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892495-v75rj_odh-ai-gateway-infra(cecab247-7b67-4a90-a3fd-99dfde6c8994)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892495-v75rj" podUID="cecab247-7b67-4a90-a3fd-99dfde6c8994" Nov 01 16:15:18.358813 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:18.358797 4363 scope.go:122] "RemoveContainer" containerID="928ce677b6baed29fac8c8449b46b4c30fd71b4c636ef316cfa13034fbc896c8" Nov 01 16:15:18.359309 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:15:18.359281 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-2ce08a-29892495-znkts_odh-ai-gateway-infra(ff730377-456e-4937-9f9e-f3102f6c3f6a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2ce08a-29892495-znkts" podUID="ff730377-456e-4937-9f9e-f3102f6c3f6a" Nov 01 16:15:18.360627 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:18.360608 4363 scope.go:122] "RemoveContainer" containerID="9fef4b9c73cecb711d5b26591c6433c78f30344096871596ea5ed4c43a89af82" Nov 01 16:15:18.361064 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:15:18.360998 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-9e044c-29892495-pvbf4_odh-ai-gateway-infra(1191cd6c-176c-4f44-97c3-06942c21b3d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e044c-29892495-pvbf4" podUID="1191cd6c-176c-4f44-97c3-06942c21b3d9" Nov 01 16:15:18.362281 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:18.362258 4363 generic.go:356] "Generic (PLEG): container finished" podID="a0918fbb-97c7-4653-ae3d-3de28dc050bf" containerID="26ced9fa18ee902a571a14168f18d847c5f9942d447fb4cc51eadfcd4997ba9b" exitCode=60 Nov 01 16:15:18.362385 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:18.362332 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892495-xxk9n" event={"ID":"a0918fbb-97c7-4653-ae3d-3de28dc050bf","Type":"ContainerDied","Data":"26ced9fa18ee902a571a14168f18d847c5f9942d447fb4cc51eadfcd4997ba9b"} Nov 01 16:15:18.362637 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:18.362620 4363 scope.go:122] "RemoveContainer" containerID="26ced9fa18ee902a571a14168f18d847c5f9942d447fb4cc51eadfcd4997ba9b" Nov 01 16:15:18.363058 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:15:18.363002 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bd831a-29892495-xxk9n_odh-ai-gateway-infra(a0918fbb-97c7-4653-ae3d-3de28dc050bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892495-xxk9n" podUID="a0918fbb-97c7-4653-ae3d-3de28dc050bf" Nov 01 16:15:18.364456 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:18.364434 4363 generic.go:356] "Generic (PLEG): container finished" podID="896ad84c-84b6-4f21-b2b8-9689578e43c9" containerID="3ef300a8a633d9d008321ad9303cb6c70b36a74021fd61671d11593c60a7139a" exitCode=60 Nov 01 16:15:18.364598 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:18.364574 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892495-2nh9k" event={"ID":"896ad84c-84b6-4f21-b2b8-9689578e43c9","Type":"ContainerDied","Data":"3ef300a8a633d9d008321ad9303cb6c70b36a74021fd61671d11593c60a7139a"} Nov 01 16:15:18.364850 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:18.364835 4363 scope.go:122] "RemoveContainer" containerID="3ef300a8a633d9d008321ad9303cb6c70b36a74021fd61671d11593c60a7139a" Nov 01 16:15:18.365758 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:15:18.365357 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892495-2nh9k_odh-ai-gateway-infra(896ad84c-84b6-4f21-b2b8-9689578e43c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892495-2nh9k" podUID="896ad84c-84b6-4f21-b2b8-9689578e43c9" Nov 01 16:15:18.366708 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:18.366692 4363 scope.go:122] "RemoveContainer" containerID="aaf9d05e7460bfef8653934af24b0726c7f0a96ba7c518d3f8a0b989c98c36b2" Nov 01 16:15:18.367075 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:15:18.367046 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-99e837-29892495-lnsd2_odh-ai-gateway-infra(a4377fa0-6280-4da7-b589-2afd643d3bce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99e837-29892495-lnsd2" podUID="a4377fa0-6280-4da7-b589-2afd643d3bce" Nov 01 16:15:18.368354 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:18.368335 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892495-rwvhk" event={"ID":"689843e8-64df-4f9c-9718-d121ba50a8cb","Type":"ContainerStarted","Data":"9a1bc30ac7bb65bb133c8fe489cec9660378384f807f77df0fe91f6725baac98"} Nov 01 16:15:18.370304 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:18.370283 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892495-wfnfr" event={"ID":"876c5b6a-7cab-4f19-8300-0fb7624d3e8f","Type":"ContainerStarted","Data":"efd48a99afbe8e80a8ed2cab492fe4e51cb80cdfabde7e3cc651d19abb67651b"} Nov 01 16:15:18.372155 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:18.372130 4363 generic.go:356] "Generic (PLEG): container finished" podID="381f38b2-e936-4b33-8cf1-121a2607045a" containerID="74e868ad10b281af60a888c0345e3ddd12d7ee4938f160b958c3190da0cce1ce" exitCode=60 Nov 01 16:15:18.373145 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:18.373098 4363 scope.go:122] "RemoveContainer" containerID="74e868ad10b281af60a888c0345e3ddd12d7ee4938f160b958c3190da0cce1ce" Nov 01 16:15:18.373446 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:15:18.373403 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-43fac8-29892495-wgnf7_odh-ai-gateway-infra(381f38b2-e936-4b33-8cf1-121a2607045a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892495-wgnf7" podUID="381f38b2-e936-4b33-8cf1-121a2607045a" Nov 01 16:15:18.373446 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:18.373444 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892495-wgnf7" event={"ID":"381f38b2-e936-4b33-8cf1-121a2607045a","Type":"ContainerDied","Data":"74e868ad10b281af60a888c0345e3ddd12d7ee4938f160b958c3190da0cce1ce"} Nov 01 16:15:18.373684 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:18.373668 4363 scope.go:122] "RemoveContainer" containerID="f305f7d4caec8fa8c9dc3504dc2b51cdb079d3a8a5a3c83d6928d89987863e1d" Nov 01 16:15:18.376987 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:18.376966 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892495-btfk6" event={"ID":"0bf581a3-128c-4b8a-a004-e61af24326c2","Type":"ContainerStarted","Data":"ab6ceafe8b3b65ae62866316e62cb6dd97c983395cf7dbd00e8fdd5f6ae9c784"} Nov 01 16:15:18.425292 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:18.425244 4363 generic.go:356] "Generic (PLEG): container finished" podID="2a839fcd-b668-4ae2-912c-fc2c812c4a55" containerID="886b9459b11f0e6190e8d0002993fe3ed8fa448fad41a256ea6bc7f3f2d880c8" exitCode=60 Nov 01 16:15:18.426491 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:18.426459 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4240a-29892495-z49zr" event={"ID":"2a839fcd-b668-4ae2-912c-fc2c812c4a55","Type":"ContainerDied","Data":"886b9459b11f0e6190e8d0002993fe3ed8fa448fad41a256ea6bc7f3f2d880c8"} Nov 01 16:15:18.426960 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:18.426940 4363 scope.go:122] "RemoveContainer" containerID="886b9459b11f0e6190e8d0002993fe3ed8fa448fad41a256ea6bc7f3f2d880c8" Nov 01 16:15:18.427376 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:15:18.427350 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a4240a-29892495-z49zr_odh-ai-gateway-infra(2a839fcd-b668-4ae2-912c-fc2c812c4a55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4240a-29892495-z49zr" podUID="2a839fcd-b668-4ae2-912c-fc2c812c4a55" Nov 01 16:15:18.429779 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:18.429761 4363 scope.go:122] "RemoveContainer" containerID="938f223c05db37d35963947683c72196aa9cef190863d34e8e39fc7cb8600447" Nov 01 16:15:18.446956 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:18.446921 4363 scope.go:122] "RemoveContainer" containerID="e8e5cc6140c89df9f578f45977a7d8d0fadeebf5e0584b5daa81035b4a697e08" Nov 01 16:15:18.463444 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:18.463419 4363 scope.go:122] "RemoveContainer" containerID="4210cb5d9ff1cdf6c4763debde79b570d914b7db4ec6d82c730976157648bd82" Nov 01 16:15:18.866448 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:18.866411 4363 patch_prober.go:29] interesting pod/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Nov 01 16:15:18.866625 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:18.866469 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" podUID="22eae6cc-4752-4174-a0f0-573adc82e32b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Nov 01 16:15:19.476944 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:19.476668 4363 generic.go:356] "Generic (PLEG): container finished" podID="689843e8-64df-4f9c-9718-d121ba50a8cb" containerID="9a1bc30ac7bb65bb133c8fe489cec9660378384f807f77df0fe91f6725baac98" exitCode=60 Nov 01 16:15:19.476944 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:19.476775 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892495-rwvhk" event={"ID":"689843e8-64df-4f9c-9718-d121ba50a8cb","Type":"ContainerDied","Data":"9a1bc30ac7bb65bb133c8fe489cec9660378384f807f77df0fe91f6725baac98"} Nov 01 16:15:19.476944 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:19.476833 4363 scope.go:122] "RemoveContainer" containerID="1701f18e4dd329c42f1818a138c56981685203602aec2c61fcb635ed1b27a8f1" Nov 01 16:15:19.477692 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:19.477432 4363 scope.go:122] "RemoveContainer" containerID="9a1bc30ac7bb65bb133c8fe489cec9660378384f807f77df0fe91f6725baac98" Nov 01 16:15:19.477983 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:15:19.477952 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892495-rwvhk_odh-ai-gateway-infra(689843e8-64df-4f9c-9718-d121ba50a8cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892495-rwvhk" podUID="689843e8-64df-4f9c-9718-d121ba50a8cb" Nov 01 16:15:19.498882 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:19.498265 4363 generic.go:356] "Generic (PLEG): container finished" podID="876c5b6a-7cab-4f19-8300-0fb7624d3e8f" containerID="efd48a99afbe8e80a8ed2cab492fe4e51cb80cdfabde7e3cc651d19abb67651b" exitCode=60 Nov 01 16:15:19.498882 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:19.498439 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892495-wfnfr" event={"ID":"876c5b6a-7cab-4f19-8300-0fb7624d3e8f","Type":"ContainerDied","Data":"efd48a99afbe8e80a8ed2cab492fe4e51cb80cdfabde7e3cc651d19abb67651b"} Nov 01 16:15:19.499232 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:19.499199 4363 scope.go:122] "RemoveContainer" containerID="efd48a99afbe8e80a8ed2cab492fe4e51cb80cdfabde7e3cc651d19abb67651b" Nov 01 16:15:19.499906 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:15:19.499861 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-dfff59-29892495-wfnfr_odh-ai-gateway-infra(876c5b6a-7cab-4f19-8300-0fb7624d3e8f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892495-wfnfr" podUID="876c5b6a-7cab-4f19-8300-0fb7624d3e8f" Nov 01 16:15:19.527906 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:19.526236 4363 scope.go:122] "RemoveContainer" containerID="74e868ad10b281af60a888c0345e3ddd12d7ee4938f160b958c3190da0cce1ce" Nov 01 16:15:19.527906 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:15:19.526732 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-43fac8-29892495-wgnf7_odh-ai-gateway-infra(381f38b2-e936-4b33-8cf1-121a2607045a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892495-wgnf7" podUID="381f38b2-e936-4b33-8cf1-121a2607045a" Nov 01 16:15:19.553043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:19.542001 4363 generic.go:356] "Generic (PLEG): container finished" podID="0bf581a3-128c-4b8a-a004-e61af24326c2" containerID="ab6ceafe8b3b65ae62866316e62cb6dd97c983395cf7dbd00e8fdd5f6ae9c784" exitCode=60 Nov 01 16:15:19.553043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:19.542115 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892495-btfk6" event={"ID":"0bf581a3-128c-4b8a-a004-e61af24326c2","Type":"ContainerDied","Data":"ab6ceafe8b3b65ae62866316e62cb6dd97c983395cf7dbd00e8fdd5f6ae9c784"} Nov 01 16:15:19.553043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:19.542922 4363 scope.go:122] "RemoveContainer" containerID="ab6ceafe8b3b65ae62866316e62cb6dd97c983395cf7dbd00e8fdd5f6ae9c784" Nov 01 16:15:19.553043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:19.547427 4363 scope.go:122] "RemoveContainer" containerID="972d69bdc1778973c86691f0379d0e14e924479c70a2e3fa8295b31f8830278d" Nov 01 16:15:19.553043 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:15:19.548855 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892495-btfk6_odh-ai-gateway-infra(0bf581a3-128c-4b8a-a004-e61af24326c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892495-btfk6" podUID="0bf581a3-128c-4b8a-a004-e61af24326c2" Nov 01 16:15:19.584989 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:19.584951 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892495-btfk6"] Nov 01 16:15:19.619527 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:19.619505 4363 scope.go:122] "RemoveContainer" containerID="e198ab11a3398898ec2ccb46ee7e9388622146ee2f8bc47245514796f818cc34" Nov 01 16:15:19.621121 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:19.621094 4363 scope.go:122] "RemoveContainer" containerID="26ced9fa18ee902a571a14168f18d847c5f9942d447fb4cc51eadfcd4997ba9b" Nov 01 16:15:19.621567 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:15:19.621538 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bd831a-29892495-xxk9n_odh-ai-gateway-infra(a0918fbb-97c7-4653-ae3d-3de28dc050bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892495-xxk9n" podUID="a0918fbb-97c7-4653-ae3d-3de28dc050bf" Nov 01 16:15:19.625526 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:19.625508 4363 scope.go:122] "RemoveContainer" containerID="3ef300a8a633d9d008321ad9303cb6c70b36a74021fd61671d11593c60a7139a" Nov 01 16:15:19.625855 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:15:19.625833 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892495-2nh9k_odh-ai-gateway-infra(896ad84c-84b6-4f21-b2b8-9689578e43c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892495-2nh9k" podUID="896ad84c-84b6-4f21-b2b8-9689578e43c9" Nov 01 16:15:19.871990 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:19.871963 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" Nov 01 16:15:19.873216 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:19.873192 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" Nov 01 16:15:20.856984 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:20.856947 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892495-btfk6" Nov 01 16:15:20.962219 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:20.962136 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bf581a3-128c-4b8a-a004-e61af24326c2-kube-api-access-pf6nr\" (UniqueName: \"kubernetes.io/projected/0bf581a3-128c-4b8a-a004-e61af24326c2-kube-api-access-pf6nr\") pod \"0bf581a3-128c-4b8a-a004-e61af24326c2\" (UID: \"0bf581a3-128c-4b8a-a004-e61af24326c2\") " Nov 01 16:15:20.962378 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:20.962302 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bf581a3-128c-4b8a-a004-e61af24326c2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bf581a3-128c-4b8a-a004-e61af24326c2-maas-api-service-ca\") pod \"0bf581a3-128c-4b8a-a004-e61af24326c2\" (UID: \"0bf581a3-128c-4b8a-a004-e61af24326c2\") " Nov 01 16:15:20.962652 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:20.962627 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bf581a3-128c-4b8a-a004-e61af24326c2-maas-api-service-ca" pod "0bf581a3-128c-4b8a-a004-e61af24326c2" (UID: "0bf581a3-128c-4b8a-a004-e61af24326c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:15:20.967061 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:20.964682 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bf581a3-128c-4b8a-a004-e61af24326c2-kube-api-access-pf6nr" pod "0bf581a3-128c-4b8a-a004-e61af24326c2" (UID: "0bf581a3-128c-4b8a-a004-e61af24326c2"). InnerVolumeSpecName "kube-api-access-pf6nr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:15:20.967061 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:20.965188 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf6nr\" (UniqueName: \"kubernetes.io/projected/0bf581a3-128c-4b8a-a004-e61af24326c2-kube-api-access-pf6nr\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:20.967061 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:20.965215 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bf581a3-128c-4b8a-a004-e61af24326c2-maas-api-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:21.652234 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:21.652196 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892495-btfk6" event={"ID":"0bf581a3-128c-4b8a-a004-e61af24326c2","Type":"ContainerDied","Data":"f5020bb807b588ad57b66067a3c905254dd4d7cece50bf946593acff5a6db0e0"} Nov 01 16:15:21.652444 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:21.652209 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892495-btfk6" Nov 01 16:15:21.652444 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:21.652293 4363 scope.go:122] "RemoveContainer" containerID="ab6ceafe8b3b65ae62866316e62cb6dd97c983395cf7dbd00e8fdd5f6ae9c784" Nov 01 16:15:21.681707 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:21.681679 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892495-btfk6"] Nov 01 16:15:21.687559 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:21.687534 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6b0e0f-29892495-btfk6"] Nov 01 16:15:22.803041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:22.796593 4363 scope.go:122] "RemoveContainer" containerID="9fef4b9c73cecb711d5b26591c6433c78f30344096871596ea5ed4c43a89af82" Nov 01 16:15:22.951050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:22.935372 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bf581a3-128c-4b8a-a004-e61af24326c2" path="/var/lib/kubelet/pods/0bf581a3-128c-4b8a-a004-e61af24326c2/volumes" Nov 01 16:15:23.779989 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:23.779951 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e044c-29892495-pvbf4" event={"ID":"1191cd6c-176c-4f44-97c3-06942c21b3d9","Type":"ContainerStarted","Data":"7537746560d592ec7700cc95b54cdcb0d13b13aa1759510b0a8a86d12c957fc8"} Nov 01 16:15:23.802156 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:23.802097 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e044c-29892495-pvbf4" podStartSLOduration=23.802078707 podStartE2EDuration="23.802078707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:15:23.798377183 +0000 UTC m=+1657.130114835" watchObservedRunningTime="2026-11-01 16:15:23.802078707 +0000 UTC m=+1657.133816359" Nov 01 16:15:24.790111 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:24.790064 4363 scope.go:122] "RemoveContainer" containerID="928ce677b6baed29fac8c8449b46b4c30fd71b4c636ef316cfa13034fbc896c8" Nov 01 16:15:24.797349 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:24.797313 4363 generic.go:356] "Generic (PLEG): container finished" podID="1191cd6c-176c-4f44-97c3-06942c21b3d9" containerID="7537746560d592ec7700cc95b54cdcb0d13b13aa1759510b0a8a86d12c957fc8" exitCode=60 Nov 01 16:15:24.807216 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:24.807184 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e044c-29892495-pvbf4" event={"ID":"1191cd6c-176c-4f44-97c3-06942c21b3d9","Type":"ContainerDied","Data":"7537746560d592ec7700cc95b54cdcb0d13b13aa1759510b0a8a86d12c957fc8"} Nov 01 16:15:24.807341 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:24.807256 4363 scope.go:122] "RemoveContainer" containerID="9fef4b9c73cecb711d5b26591c6433c78f30344096871596ea5ed4c43a89af82" Nov 01 16:15:24.807936 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:24.807795 4363 scope.go:122] "RemoveContainer" containerID="7537746560d592ec7700cc95b54cdcb0d13b13aa1759510b0a8a86d12c957fc8" Nov 01 16:15:24.808861 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:15:24.808523 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-9e044c-29892495-pvbf4_odh-ai-gateway-infra(1191cd6c-176c-4f44-97c3-06942c21b3d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e044c-29892495-pvbf4" podUID="1191cd6c-176c-4f44-97c3-06942c21b3d9" Nov 01 16:15:24.818193 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:24.818149 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e044c-29892495-pvbf4"] Nov 01 16:15:25.795038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:25.793819 4363 scope.go:122] "RemoveContainer" containerID="74e868ad10b281af60a888c0345e3ddd12d7ee4938f160b958c3190da0cce1ce" Nov 01 16:15:25.796318 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:25.796279 4363 scope.go:122] "RemoveContainer" containerID="aaf9d05e7460bfef8653934af24b0726c7f0a96ba7c518d3f8a0b989c98c36b2" Nov 01 16:15:25.845985 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:25.845941 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2ce08a-29892495-znkts" event={"ID":"ff730377-456e-4937-9f9e-f3102f6c3f6a","Type":"ContainerStarted","Data":"f2399c6d787a9e802afcb60764a9ee3662e8ad37ce8f3cc58522bc5b5d6c474f"} Nov 01 16:15:26.140173 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:26.140145 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e044c-29892495-pvbf4" Nov 01 16:15:26.212800 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:26.190892 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1191cd6c-176c-4f44-97c3-06942c21b3d9-kube-api-access-qj2xc\" (UniqueName: \"kubernetes.io/projected/1191cd6c-176c-4f44-97c3-06942c21b3d9-kube-api-access-qj2xc\") pod \"1191cd6c-176c-4f44-97c3-06942c21b3d9\" (UID: \"1191cd6c-176c-4f44-97c3-06942c21b3d9\") " Nov 01 16:15:26.212800 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:26.191279 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1191cd6c-176c-4f44-97c3-06942c21b3d9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1191cd6c-176c-4f44-97c3-06942c21b3d9-maas-api-service-ca\") pod \"1191cd6c-176c-4f44-97c3-06942c21b3d9\" (UID: \"1191cd6c-176c-4f44-97c3-06942c21b3d9\") " Nov 01 16:15:26.212800 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:26.211447 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1191cd6c-176c-4f44-97c3-06942c21b3d9-maas-api-service-ca" pod "1191cd6c-176c-4f44-97c3-06942c21b3d9" (UID: "1191cd6c-176c-4f44-97c3-06942c21b3d9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:15:26.215030 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:26.214986 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1191cd6c-176c-4f44-97c3-06942c21b3d9-kube-api-access-qj2xc" pod "1191cd6c-176c-4f44-97c3-06942c21b3d9" (UID: "1191cd6c-176c-4f44-97c3-06942c21b3d9"). InnerVolumeSpecName "kube-api-access-qj2xc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:15:26.329666 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:26.329609 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1191cd6c-176c-4f44-97c3-06942c21b3d9-maas-api-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:26.329928 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:26.329899 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj2xc\" (UniqueName: \"kubernetes.io/projected/1191cd6c-176c-4f44-97c3-06942c21b3d9-kube-api-access-qj2xc\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:26.858843 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:26.858806 4363 generic.go:356] "Generic (PLEG): container finished" podID="ff730377-456e-4937-9f9e-f3102f6c3f6a" containerID="f2399c6d787a9e802afcb60764a9ee3662e8ad37ce8f3cc58522bc5b5d6c474f" exitCode=60 Nov 01 16:15:26.859272 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:26.858899 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2ce08a-29892495-znkts" event={"ID":"ff730377-456e-4937-9f9e-f3102f6c3f6a","Type":"ContainerDied","Data":"f2399c6d787a9e802afcb60764a9ee3662e8ad37ce8f3cc58522bc5b5d6c474f"} Nov 01 16:15:26.859272 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:26.858962 4363 scope.go:122] "RemoveContainer" containerID="928ce677b6baed29fac8c8449b46b4c30fd71b4c636ef316cfa13034fbc896c8" Nov 01 16:15:26.859458 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:26.859433 4363 scope.go:122] "RemoveContainer" containerID="f2399c6d787a9e802afcb60764a9ee3662e8ad37ce8f3cc58522bc5b5d6c474f" Nov 01 16:15:26.859828 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:15:26.859791 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-2ce08a-29892495-znkts_odh-ai-gateway-infra(ff730377-456e-4937-9f9e-f3102f6c3f6a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2ce08a-29892495-znkts" podUID="ff730377-456e-4937-9f9e-f3102f6c3f6a" Nov 01 16:15:26.860580 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:26.860559 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e044c-29892495-pvbf4" Nov 01 16:15:26.860664 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:26.860577 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e044c-29892495-pvbf4" event={"ID":"1191cd6c-176c-4f44-97c3-06942c21b3d9","Type":"ContainerDied","Data":"71ffedb77443120cc778e5822ddf13378e72525b9d00defc1b8bbd06a835c05f"} Nov 01 16:15:26.862736 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:26.862716 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99e837-29892495-lnsd2" event={"ID":"a4377fa0-6280-4da7-b589-2afd643d3bce","Type":"ContainerStarted","Data":"8c2f85e052a17bb74d0ca8202505240d9ae4c3cd70b54a29c84f8363b5a7275a"} Nov 01 16:15:26.864939 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:26.864919 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892495-wgnf7" event={"ID":"381f38b2-e936-4b33-8cf1-121a2607045a","Type":"ContainerStarted","Data":"ecefb45590155dc4010cedbc87eb094a7614d97c0fb7de09e1989754d53dec3d"} Nov 01 16:15:26.881379 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:26.881354 4363 scope.go:122] "RemoveContainer" containerID="7537746560d592ec7700cc95b54cdcb0d13b13aa1759510b0a8a86d12c957fc8" Nov 01 16:15:26.893797 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:26.893769 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2ce08a-29892495-znkts"] Nov 01 16:15:27.027591 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:27.027535 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892495-wgnf7" podStartSLOduration=27.027516801 podStartE2EDuration="27.027516801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:15:27.020640079 +0000 UTC m=+1660.352377729" watchObservedRunningTime="2026-11-01 16:15:27.027516801 +0000 UTC m=+1660.359254452" Nov 01 16:15:27.059077 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:27.059009 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99e837-29892495-lnsd2" podStartSLOduration=27.058994233 podStartE2EDuration="27.058994233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:15:27.057190153 +0000 UTC m=+1660.388927802" watchObservedRunningTime="2026-11-01 16:15:27.058994233 +0000 UTC m=+1660.390731906" Nov 01 16:15:27.091361 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:27.091272 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e044c-29892495-pvbf4"] Nov 01 16:15:27.100204 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:27.100179 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e044c-29892495-pvbf4"] Nov 01 16:15:27.804875 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:27.804847 4363 scope.go:122] "RemoveContainer" containerID="886b9459b11f0e6190e8d0002993fe3ed8fa448fad41a256ea6bc7f3f2d880c8" Nov 01 16:15:27.877593 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:27.877556 4363 generic.go:356] "Generic (PLEG): container finished" podID="a4377fa0-6280-4da7-b589-2afd643d3bce" containerID="8c2f85e052a17bb74d0ca8202505240d9ae4c3cd70b54a29c84f8363b5a7275a" exitCode=60 Nov 01 16:15:27.878059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:27.877800 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99e837-29892495-lnsd2" event={"ID":"a4377fa0-6280-4da7-b589-2afd643d3bce","Type":"ContainerDied","Data":"8c2f85e052a17bb74d0ca8202505240d9ae4c3cd70b54a29c84f8363b5a7275a"} Nov 01 16:15:27.878059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:27.877876 4363 scope.go:122] "RemoveContainer" containerID="aaf9d05e7460bfef8653934af24b0726c7f0a96ba7c518d3f8a0b989c98c36b2" Nov 01 16:15:27.878575 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:27.878555 4363 scope.go:122] "RemoveContainer" containerID="8c2f85e052a17bb74d0ca8202505240d9ae4c3cd70b54a29c84f8363b5a7275a" Nov 01 16:15:27.879311 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:15:27.879208 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-99e837-29892495-lnsd2_odh-ai-gateway-infra(a4377fa0-6280-4da7-b589-2afd643d3bce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99e837-29892495-lnsd2" podUID="a4377fa0-6280-4da7-b589-2afd643d3bce" Nov 01 16:15:27.886069 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:27.882217 4363 generic.go:356] "Generic (PLEG): container finished" podID="381f38b2-e936-4b33-8cf1-121a2607045a" containerID="ecefb45590155dc4010cedbc87eb094a7614d97c0fb7de09e1989754d53dec3d" exitCode=60 Nov 01 16:15:27.886069 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:27.882396 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892495-wgnf7" event={"ID":"381f38b2-e936-4b33-8cf1-121a2607045a","Type":"ContainerDied","Data":"ecefb45590155dc4010cedbc87eb094a7614d97c0fb7de09e1989754d53dec3d"} Nov 01 16:15:27.886069 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:27.882844 4363 scope.go:122] "RemoveContainer" containerID="ecefb45590155dc4010cedbc87eb094a7614d97c0fb7de09e1989754d53dec3d" Nov 01 16:15:27.886069 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:15:27.883298 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-43fac8-29892495-wgnf7_odh-ai-gateway-infra(381f38b2-e936-4b33-8cf1-121a2607045a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892495-wgnf7" podUID="381f38b2-e936-4b33-8cf1-121a2607045a" Nov 01 16:15:28.001042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:28.000939 4363 scope.go:122] "RemoveContainer" containerID="74e868ad10b281af60a888c0345e3ddd12d7ee4938f160b958c3190da0cce1ce" Nov 01 16:15:28.059806 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:28.059741 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892495-wgnf7"] Nov 01 16:15:28.129007 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:28.128965 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99e837-29892495-lnsd2"] Nov 01 16:15:28.221882 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:28.221853 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2ce08a-29892495-znkts" Nov 01 16:15:28.246325 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:28.246262 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff730377-456e-4937-9f9e-f3102f6c3f6a-kube-api-access-fn97q\" (UniqueName: \"kubernetes.io/projected/ff730377-456e-4937-9f9e-f3102f6c3f6a-kube-api-access-fn97q\") pod \"ff730377-456e-4937-9f9e-f3102f6c3f6a\" (UID: \"ff730377-456e-4937-9f9e-f3102f6c3f6a\") " Nov 01 16:15:28.246705 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:28.246681 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff730377-456e-4937-9f9e-f3102f6c3f6a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff730377-456e-4937-9f9e-f3102f6c3f6a-maas-api-service-ca\") pod \"ff730377-456e-4937-9f9e-f3102f6c3f6a\" (UID: \"ff730377-456e-4937-9f9e-f3102f6c3f6a\") " Nov 01 16:15:28.247217 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:28.247192 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff730377-456e-4937-9f9e-f3102f6c3f6a-maas-api-service-ca" pod "ff730377-456e-4937-9f9e-f3102f6c3f6a" (UID: "ff730377-456e-4937-9f9e-f3102f6c3f6a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:15:28.248268 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:28.248243 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff730377-456e-4937-9f9e-f3102f6c3f6a-maas-api-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:28.249245 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:28.249219 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff730377-456e-4937-9f9e-f3102f6c3f6a-kube-api-access-fn97q" pod "ff730377-456e-4937-9f9e-f3102f6c3f6a" (UID: "ff730377-456e-4937-9f9e-f3102f6c3f6a"). InnerVolumeSpecName "kube-api-access-fn97q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:15:28.368041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:28.360311 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn97q\" (UniqueName: \"kubernetes.io/projected/ff730377-456e-4937-9f9e-f3102f6c3f6a-kube-api-access-fn97q\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:28.790950 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:28.790532 4363 scope.go:122] "RemoveContainer" containerID="67d5e51f38ba092ec8f375e0fa71946572ed5eb7327e79010825f407b4b16f6d" Nov 01 16:15:28.790950 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:28.790892 4363 scope.go:122] "RemoveContainer" containerID="26ced9fa18ee902a571a14168f18d847c5f9942d447fb4cc51eadfcd4997ba9b" Nov 01 16:15:28.793088 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:28.793069 4363 scope.go:122] "RemoveContainer" containerID="3ef300a8a633d9d008321ad9303cb6c70b36a74021fd61671d11593c60a7139a" Nov 01 16:15:28.820633 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:28.820598 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1191cd6c-176c-4f44-97c3-06942c21b3d9" path="/var/lib/kubelet/pods/1191cd6c-176c-4f44-97c3-06942c21b3d9/volumes" Nov 01 16:15:28.914412 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:28.914384 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4240a-29892495-z49zr" event={"ID":"2a839fcd-b668-4ae2-912c-fc2c812c4a55","Type":"ContainerStarted","Data":"89d62c4d1b1171660a89e96279e1a2c22adea8ed7b2accb0729ff7e6466c641c"} Nov 01 16:15:28.916342 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:28.916319 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2ce08a-29892495-znkts" Nov 01 16:15:28.916504 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:28.916344 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2ce08a-29892495-znkts" event={"ID":"ff730377-456e-4937-9f9e-f3102f6c3f6a","Type":"ContainerDied","Data":"38c3cab7d6f29ba2c55cf130b206d6e78f67e5449893eae74665859d25db9871"} Nov 01 16:15:28.916504 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:28.916396 4363 scope.go:122] "RemoveContainer" containerID="f2399c6d787a9e802afcb60764a9ee3662e8ad37ce8f3cc58522bc5b5d6c474f" Nov 01 16:15:29.107338 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:29.107302 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2ce08a-29892495-znkts"] Nov 01 16:15:29.116243 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:29.116218 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2ce08a-29892495-znkts"] Nov 01 16:15:29.287598 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:29.287571 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892495-wgnf7" Nov 01 16:15:29.409982 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:29.409960 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99e837-29892495-lnsd2" Nov 01 16:15:29.415369 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:29.415344 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/381f38b2-e936-4b33-8cf1-121a2607045a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/381f38b2-e936-4b33-8cf1-121a2607045a-maas-api-service-ca\") pod \"381f38b2-e936-4b33-8cf1-121a2607045a\" (UID: \"381f38b2-e936-4b33-8cf1-121a2607045a\") " Nov 01 16:15:29.415708 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:29.415683 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/381f38b2-e936-4b33-8cf1-121a2607045a-maas-api-service-ca" pod "381f38b2-e936-4b33-8cf1-121a2607045a" (UID: "381f38b2-e936-4b33-8cf1-121a2607045a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:15:29.415816 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:29.415693 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/381f38b2-e936-4b33-8cf1-121a2607045a-kube-api-access-b5pf2\" (UniqueName: \"kubernetes.io/projected/381f38b2-e936-4b33-8cf1-121a2607045a-kube-api-access-b5pf2\") pod \"381f38b2-e936-4b33-8cf1-121a2607045a\" (UID: \"381f38b2-e936-4b33-8cf1-121a2607045a\") " Nov 01 16:15:29.416574 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:29.416533 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/381f38b2-e936-4b33-8cf1-121a2607045a-maas-api-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:29.418745 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:29.418722 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/381f38b2-e936-4b33-8cf1-121a2607045a-kube-api-access-b5pf2" pod "381f38b2-e936-4b33-8cf1-121a2607045a" (UID: "381f38b2-e936-4b33-8cf1-121a2607045a"). InnerVolumeSpecName "kube-api-access-b5pf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:15:29.518053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:29.518025 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4377fa0-6280-4da7-b589-2afd643d3bce-kube-api-access-vhr9v\" (UniqueName: \"kubernetes.io/projected/a4377fa0-6280-4da7-b589-2afd643d3bce-kube-api-access-vhr9v\") pod \"a4377fa0-6280-4da7-b589-2afd643d3bce\" (UID: \"a4377fa0-6280-4da7-b589-2afd643d3bce\") " Nov 01 16:15:29.518303 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:29.518289 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4377fa0-6280-4da7-b589-2afd643d3bce-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4377fa0-6280-4da7-b589-2afd643d3bce-maas-api-service-ca\") pod \"a4377fa0-6280-4da7-b589-2afd643d3bce\" (UID: \"a4377fa0-6280-4da7-b589-2afd643d3bce\") " Nov 01 16:15:29.518624 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:29.518596 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4377fa0-6280-4da7-b589-2afd643d3bce-maas-api-service-ca" pod "a4377fa0-6280-4da7-b589-2afd643d3bce" (UID: "a4377fa0-6280-4da7-b589-2afd643d3bce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:15:29.519436 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:29.519411 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5pf2\" (UniqueName: \"kubernetes.io/projected/381f38b2-e936-4b33-8cf1-121a2607045a-kube-api-access-b5pf2\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:29.519547 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:29.519440 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4377fa0-6280-4da7-b589-2afd643d3bce-maas-api-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:29.520427 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:29.520402 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4377fa0-6280-4da7-b589-2afd643d3bce-kube-api-access-vhr9v" pod "a4377fa0-6280-4da7-b589-2afd643d3bce" (UID: "a4377fa0-6280-4da7-b589-2afd643d3bce"). InnerVolumeSpecName "kube-api-access-vhr9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:15:29.621705 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:29.621670 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhr9v\" (UniqueName: \"kubernetes.io/projected/a4377fa0-6280-4da7-b589-2afd643d3bce-kube-api-access-vhr9v\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:29.979363 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:29.979275 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892495-v75rj" event={"ID":"cecab247-7b67-4a90-a3fd-99dfde6c8994","Type":"ContainerStarted","Data":"1d2728cd1aff2bab59a1c7e84aa0347a5683141294314d0642116e0713ba7dd2"} Nov 01 16:15:29.987792 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:29.985419 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4240a-29892495-z49zr"] Nov 01 16:15:29.993724 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:29.993686 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892495-xxk9n" event={"ID":"a0918fbb-97c7-4653-ae3d-3de28dc050bf","Type":"ContainerStarted","Data":"f200ee222587b04e948025cc5899d8c844574610915c8c8212162fc51d61adfc"} Nov 01 16:15:29.997664 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:29.997629 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892495-2nh9k" event={"ID":"896ad84c-84b6-4f21-b2b8-9689578e43c9","Type":"ContainerStarted","Data":"5d5e96f53ebe3f0fc8c9cf150853d6697a09772efcbb6d22ac878003a99ef3f5"} Nov 01 16:15:30.000049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:29.999988 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99e837-29892495-lnsd2" event={"ID":"a4377fa0-6280-4da7-b589-2afd643d3bce","Type":"ContainerDied","Data":"70de9a563cca52ff157fdcf8da2e820b45f5c9e63afa1166666962b5ebd711a9"} Nov 01 16:15:30.000246 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:30.000229 4363 scope.go:122] "RemoveContainer" containerID="8c2f85e052a17bb74d0ca8202505240d9ae4c3cd70b54a29c84f8363b5a7275a" Nov 01 16:15:30.000475 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:30.000434 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99e837-29892495-lnsd2" Nov 01 16:15:30.003917 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:30.003806 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892495-wgnf7" event={"ID":"381f38b2-e936-4b33-8cf1-121a2607045a","Type":"ContainerDied","Data":"7892cff2a4a5d7d823033914cb2607b616ce52cfb68447c2cc0827611fafd7c1"} Nov 01 16:15:30.004056 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:30.003968 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892495-wgnf7" Nov 01 16:15:30.006883 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:30.006855 4363 generic.go:356] "Generic (PLEG): container finished" podID="2a839fcd-b668-4ae2-912c-fc2c812c4a55" containerID="89d62c4d1b1171660a89e96279e1a2c22adea8ed7b2accb0729ff7e6466c641c" exitCode=60 Nov 01 16:15:30.006989 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:30.006916 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4240a-29892495-z49zr" event={"ID":"2a839fcd-b668-4ae2-912c-fc2c812c4a55","Type":"ContainerDied","Data":"89d62c4d1b1171660a89e96279e1a2c22adea8ed7b2accb0729ff7e6466c641c"} Nov 01 16:15:30.024136 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:30.024112 4363 scope.go:122] "RemoveContainer" containerID="ecefb45590155dc4010cedbc87eb094a7614d97c0fb7de09e1989754d53dec3d" Nov 01 16:15:30.036775 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:30.036747 4363 scope.go:122] "RemoveContainer" containerID="886b9459b11f0e6190e8d0002993fe3ed8fa448fad41a256ea6bc7f3f2d880c8" Nov 01 16:15:30.076820 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:30.076780 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99e837-29892495-lnsd2"] Nov 01 16:15:30.085068 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:30.085039 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99e837-29892495-lnsd2"] Nov 01 16:15:30.097986 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:30.097946 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892495-wgnf7"] Nov 01 16:15:30.108561 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:30.108527 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892495-wgnf7"] Nov 01 16:15:30.372985 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:30.372944 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4240a-29892495-z49zr" Nov 01 16:15:30.470131 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:30.470047 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a839fcd-b668-4ae2-912c-fc2c812c4a55-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a839fcd-b668-4ae2-912c-fc2c812c4a55-maas-api-service-ca\") pod \"2a839fcd-b668-4ae2-912c-fc2c812c4a55\" (UID: \"2a839fcd-b668-4ae2-912c-fc2c812c4a55\") " Nov 01 16:15:30.470402 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:30.470378 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a839fcd-b668-4ae2-912c-fc2c812c4a55-kube-api-access-tsrtr\" (UniqueName: \"kubernetes.io/projected/2a839fcd-b668-4ae2-912c-fc2c812c4a55-kube-api-access-tsrtr\") pod \"2a839fcd-b668-4ae2-912c-fc2c812c4a55\" (UID: \"2a839fcd-b668-4ae2-912c-fc2c812c4a55\") " Nov 01 16:15:30.470492 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:30.470449 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a839fcd-b668-4ae2-912c-fc2c812c4a55-maas-api-service-ca" pod "2a839fcd-b668-4ae2-912c-fc2c812c4a55" (UID: "2a839fcd-b668-4ae2-912c-fc2c812c4a55"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:15:30.471333 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:30.471309 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a839fcd-b668-4ae2-912c-fc2c812c4a55-maas-api-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:30.472665 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:30.472627 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a839fcd-b668-4ae2-912c-fc2c812c4a55-kube-api-access-tsrtr" pod "2a839fcd-b668-4ae2-912c-fc2c812c4a55" (UID: "2a839fcd-b668-4ae2-912c-fc2c812c4a55"). InnerVolumeSpecName "kube-api-access-tsrtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:15:30.573420 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:30.573391 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsrtr\" (UniqueName: \"kubernetes.io/projected/2a839fcd-b668-4ae2-912c-fc2c812c4a55-kube-api-access-tsrtr\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:30.802729 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:30.802604 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="381f38b2-e936-4b33-8cf1-121a2607045a" path="/var/lib/kubelet/pods/381f38b2-e936-4b33-8cf1-121a2607045a/volumes" Nov 01 16:15:30.803287 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:30.803268 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4377fa0-6280-4da7-b589-2afd643d3bce" path="/var/lib/kubelet/pods/a4377fa0-6280-4da7-b589-2afd643d3bce/volumes" Nov 01 16:15:30.803711 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:30.803696 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff730377-456e-4937-9f9e-f3102f6c3f6a" path="/var/lib/kubelet/pods/ff730377-456e-4937-9f9e-f3102f6c3f6a/volumes" Nov 01 16:15:30.805263 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:30.805240 4363 scope.go:122] "RemoveContainer" containerID="9a1bc30ac7bb65bb133c8fe489cec9660378384f807f77df0fe91f6725baac98" Nov 01 16:15:31.010070 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:31.010039 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892495-v75rj"] Nov 01 16:15:31.022509 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:31.022472 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4240a-29892495-z49zr" event={"ID":"2a839fcd-b668-4ae2-912c-fc2c812c4a55","Type":"ContainerDied","Data":"15d6ad71ac6165bcc9af44e0eb0a96101281fdab808c6717767fefa3299b16f5"} Nov 01 16:15:31.022679 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:31.022541 4363 scope.go:122] "RemoveContainer" containerID="89d62c4d1b1171660a89e96279e1a2c22adea8ed7b2accb0729ff7e6466c641c" Nov 01 16:15:31.022679 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:31.022671 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4240a-29892495-z49zr" Nov 01 16:15:31.025915 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:31.025892 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892495-xxk9n"] Nov 01 16:15:31.026844 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:31.026813 4363 generic.go:356] "Generic (PLEG): container finished" podID="cecab247-7b67-4a90-a3fd-99dfde6c8994" containerID="1d2728cd1aff2bab59a1c7e84aa0347a5683141294314d0642116e0713ba7dd2" exitCode=60 Nov 01 16:15:31.026969 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:31.026944 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892495-v75rj" event={"ID":"cecab247-7b67-4a90-a3fd-99dfde6c8994","Type":"ContainerDied","Data":"1d2728cd1aff2bab59a1c7e84aa0347a5683141294314d0642116e0713ba7dd2"} Nov 01 16:15:31.030954 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:31.030929 4363 generic.go:356] "Generic (PLEG): container finished" podID="a0918fbb-97c7-4653-ae3d-3de28dc050bf" containerID="f200ee222587b04e948025cc5899d8c844574610915c8c8212162fc51d61adfc" exitCode=60 Nov 01 16:15:31.031099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:31.031045 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892495-xxk9n" event={"ID":"a0918fbb-97c7-4653-ae3d-3de28dc050bf","Type":"ContainerDied","Data":"f200ee222587b04e948025cc5899d8c844574610915c8c8212162fc51d61adfc"} Nov 01 16:15:31.033499 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:31.033478 4363 generic.go:356] "Generic (PLEG): container finished" podID="896ad84c-84b6-4f21-b2b8-9689578e43c9" containerID="5d5e96f53ebe3f0fc8c9cf150853d6697a09772efcbb6d22ac878003a99ef3f5" exitCode=60 Nov 01 16:15:31.033584 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:31.033537 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892495-2nh9k" event={"ID":"896ad84c-84b6-4f21-b2b8-9689578e43c9","Type":"ContainerDied","Data":"5d5e96f53ebe3f0fc8c9cf150853d6697a09772efcbb6d22ac878003a99ef3f5"} Nov 01 16:15:31.033963 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:31.033949 4363 scope.go:122] "RemoveContainer" containerID="5d5e96f53ebe3f0fc8c9cf150853d6697a09772efcbb6d22ac878003a99ef3f5" Nov 01 16:15:31.034368 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:15:31.034349 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892495-2nh9k_odh-ai-gateway-infra(896ad84c-84b6-4f21-b2b8-9689578e43c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892495-2nh9k" podUID="896ad84c-84b6-4f21-b2b8-9689578e43c9" Nov 01 16:15:31.039398 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:31.039382 4363 scope.go:122] "RemoveContainer" containerID="67d5e51f38ba092ec8f375e0fa71946572ed5eb7327e79010825f407b4b16f6d" Nov 01 16:15:31.053058 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:31.052969 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4240a-29892495-z49zr"] Nov 01 16:15:31.058661 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:31.058629 4363 scope.go:122] "RemoveContainer" containerID="26ced9fa18ee902a571a14168f18d847c5f9942d447fb4cc51eadfcd4997ba9b" Nov 01 16:15:31.060823 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:31.060798 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4240a-29892495-z49zr"] Nov 01 16:15:31.071680 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:31.071587 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892495-2nh9k"] Nov 01 16:15:31.084177 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:31.084151 4363 scope.go:122] "RemoveContainer" containerID="3ef300a8a633d9d008321ad9303cb6c70b36a74021fd61671d11593c60a7139a" Nov 01 16:15:31.320888 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:31.320805 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892495-v75rj" Nov 01 16:15:31.323741 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:31.323719 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892495-xxk9n" Nov 01 16:15:31.510429 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:31.510381 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0918fbb-97c7-4653-ae3d-3de28dc050bf-kube-api-access-kdl46\" (UniqueName: \"kubernetes.io/projected/a0918fbb-97c7-4653-ae3d-3de28dc050bf-kube-api-access-kdl46\") pod \"a0918fbb-97c7-4653-ae3d-3de28dc050bf\" (UID: \"a0918fbb-97c7-4653-ae3d-3de28dc050bf\") " Nov 01 16:15:31.511067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:31.511040 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cecab247-7b67-4a90-a3fd-99dfde6c8994-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cecab247-7b67-4a90-a3fd-99dfde6c8994-maas-api-service-ca\") pod \"cecab247-7b67-4a90-a3fd-99dfde6c8994\" (UID: \"cecab247-7b67-4a90-a3fd-99dfde6c8994\") " Nov 01 16:15:31.515034 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:31.511448 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cecab247-7b67-4a90-a3fd-99dfde6c8994-kube-api-access-6w4cw\" (UniqueName: \"kubernetes.io/projected/cecab247-7b67-4a90-a3fd-99dfde6c8994-kube-api-access-6w4cw\") pod \"cecab247-7b67-4a90-a3fd-99dfde6c8994\" (UID: \"cecab247-7b67-4a90-a3fd-99dfde6c8994\") " Nov 01 16:15:31.520942 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:31.520911 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0918fbb-97c7-4653-ae3d-3de28dc050bf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0918fbb-97c7-4653-ae3d-3de28dc050bf-maas-api-service-ca\") pod \"a0918fbb-97c7-4653-ae3d-3de28dc050bf\" (UID: \"a0918fbb-97c7-4653-ae3d-3de28dc050bf\") " Nov 01 16:15:31.522854 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:31.518665 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cecab247-7b67-4a90-a3fd-99dfde6c8994-maas-api-service-ca" pod "cecab247-7b67-4a90-a3fd-99dfde6c8994" (UID: "cecab247-7b67-4a90-a3fd-99dfde6c8994"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:15:31.522854 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:31.522805 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0918fbb-97c7-4653-ae3d-3de28dc050bf-maas-api-service-ca" pod "a0918fbb-97c7-4653-ae3d-3de28dc050bf" (UID: "a0918fbb-97c7-4653-ae3d-3de28dc050bf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:15:31.523872 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:31.523846 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cecab247-7b67-4a90-a3fd-99dfde6c8994-kube-api-access-6w4cw" pod "cecab247-7b67-4a90-a3fd-99dfde6c8994" (UID: "cecab247-7b67-4a90-a3fd-99dfde6c8994"). InnerVolumeSpecName "kube-api-access-6w4cw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:15:31.526462 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:31.526441 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0918fbb-97c7-4653-ae3d-3de28dc050bf-kube-api-access-kdl46" pod "a0918fbb-97c7-4653-ae3d-3de28dc050bf" (UID: "a0918fbb-97c7-4653-ae3d-3de28dc050bf"). InnerVolumeSpecName "kube-api-access-kdl46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:15:31.624921 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:31.624884 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdl46\" (UniqueName: \"kubernetes.io/projected/a0918fbb-97c7-4653-ae3d-3de28dc050bf-kube-api-access-kdl46\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:31.624921 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:31.624919 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cecab247-7b67-4a90-a3fd-99dfde6c8994-maas-api-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:31.624921 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:31.624929 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w4cw\" (UniqueName: \"kubernetes.io/projected/cecab247-7b67-4a90-a3fd-99dfde6c8994-kube-api-access-6w4cw\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:31.625241 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:31.624939 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0918fbb-97c7-4653-ae3d-3de28dc050bf-maas-api-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:32.065441 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:32.065407 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892495-v75rj" Nov 01 16:15:32.073166 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:32.072165 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892495-v75rj" event={"ID":"cecab247-7b67-4a90-a3fd-99dfde6c8994","Type":"ContainerDied","Data":"8f3be7c9b2c7a2a2539cb9d0c9ba7a8dd216d965216f6bf969224b8afb5c54dd"} Nov 01 16:15:32.073166 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:32.072259 4363 scope.go:122] "RemoveContainer" containerID="1d2728cd1aff2bab59a1c7e84aa0347a5683141294314d0642116e0713ba7dd2" Nov 01 16:15:32.115040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:32.092695 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892495-xxk9n" event={"ID":"a0918fbb-97c7-4653-ae3d-3de28dc050bf","Type":"ContainerDied","Data":"dcd9c42c934528824b00698cd420d4afd85676dcd985af2990980df9540c1015"} Nov 01 16:15:32.115040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:32.092830 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892495-xxk9n" Nov 01 16:15:32.187625 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:32.187600 4363 scope.go:122] "RemoveContainer" containerID="f200ee222587b04e948025cc5899d8c844574610915c8c8212162fc51d61adfc" Nov 01 16:15:32.238025 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:32.237979 4363 generic.go:356] "Generic (PLEG): container finished" podID="689843e8-64df-4f9c-9718-d121ba50a8cb" containerID="3e94f60ae60478ad03fd2dac474e2675fc897ca76a1d1b6f0bcbb66baf283149" exitCode=60 Nov 01 16:15:32.238200 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:32.238169 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892495-rwvhk" event={"ID":"689843e8-64df-4f9c-9718-d121ba50a8cb","Type":"ContainerDied","Data":"3e94f60ae60478ad03fd2dac474e2675fc897ca76a1d1b6f0bcbb66baf283149"} Nov 01 16:15:32.238683 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:32.238665 4363 scope.go:122] "RemoveContainer" containerID="3e94f60ae60478ad03fd2dac474e2675fc897ca76a1d1b6f0bcbb66baf283149" Nov 01 16:15:32.239633 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:15:32.239229 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892495-rwvhk_odh-ai-gateway-infra(689843e8-64df-4f9c-9718-d121ba50a8cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892495-rwvhk" podUID="689843e8-64df-4f9c-9718-d121ba50a8cb" Nov 01 16:15:32.245592 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:32.245443 4363 scope.go:122] "RemoveContainer" containerID="9a1bc30ac7bb65bb133c8fe489cec9660378384f807f77df0fe91f6725baac98" Nov 01 16:15:32.245972 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:32.245951 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892495-xxk9n"] Nov 01 16:15:32.253135 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:32.252341 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892495-xxk9n"] Nov 01 16:15:32.271433 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:32.271403 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892495-v75rj"] Nov 01 16:15:32.278215 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:32.278187 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892495-v75rj"] Nov 01 16:15:32.436354 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:32.436330 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892495-2nh9k" Nov 01 16:15:32.578829 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:32.578746 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/896ad84c-84b6-4f21-b2b8-9689578e43c9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/896ad84c-84b6-4f21-b2b8-9689578e43c9-maas-api-service-ca\") pod \"896ad84c-84b6-4f21-b2b8-9689578e43c9\" (UID: \"896ad84c-84b6-4f21-b2b8-9689578e43c9\") " Nov 01 16:15:32.579012 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:32.578834 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/896ad84c-84b6-4f21-b2b8-9689578e43c9-kube-api-access-vs6zn\" (UniqueName: \"kubernetes.io/projected/896ad84c-84b6-4f21-b2b8-9689578e43c9-kube-api-access-vs6zn\") pod \"896ad84c-84b6-4f21-b2b8-9689578e43c9\" (UID: \"896ad84c-84b6-4f21-b2b8-9689578e43c9\") " Nov 01 16:15:32.579225 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:32.579197 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/896ad84c-84b6-4f21-b2b8-9689578e43c9-maas-api-service-ca" pod "896ad84c-84b6-4f21-b2b8-9689578e43c9" (UID: "896ad84c-84b6-4f21-b2b8-9689578e43c9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:15:32.579734 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:32.579715 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/896ad84c-84b6-4f21-b2b8-9689578e43c9-maas-api-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:32.581030 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:32.580992 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/896ad84c-84b6-4f21-b2b8-9689578e43c9-kube-api-access-vs6zn" pod "896ad84c-84b6-4f21-b2b8-9689578e43c9" (UID: "896ad84c-84b6-4f21-b2b8-9689578e43c9"). InnerVolumeSpecName "kube-api-access-vs6zn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:15:32.682379 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:32.682350 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs6zn\" (UniqueName: \"kubernetes.io/projected/896ad84c-84b6-4f21-b2b8-9689578e43c9-kube-api-access-vs6zn\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:32.805921 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:32.805882 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a839fcd-b668-4ae2-912c-fc2c812c4a55" path="/var/lib/kubelet/pods/2a839fcd-b668-4ae2-912c-fc2c812c4a55/volumes" Nov 01 16:15:32.806643 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:32.806623 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0918fbb-97c7-4653-ae3d-3de28dc050bf" path="/var/lib/kubelet/pods/a0918fbb-97c7-4653-ae3d-3de28dc050bf/volumes" Nov 01 16:15:32.807255 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:32.807236 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cecab247-7b67-4a90-a3fd-99dfde6c8994" path="/var/lib/kubelet/pods/cecab247-7b67-4a90-a3fd-99dfde6c8994/volumes" Nov 01 16:15:32.809342 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:32.809301 4363 scope.go:122] "RemoveContainer" containerID="efd48a99afbe8e80a8ed2cab492fe4e51cb80cdfabde7e3cc651d19abb67651b" Nov 01 16:15:33.255307 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.255216 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892495-2nh9k" Nov 01 16:15:33.255307 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.255245 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892495-2nh9k" event={"ID":"896ad84c-84b6-4f21-b2b8-9689578e43c9","Type":"ContainerDied","Data":"47fe2cc5043e0c866887fb95b3bd471e692dbac64680b32d9391f0b7df4340eb"} Nov 01 16:15:33.255813 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.255341 4363 scope.go:122] "RemoveContainer" containerID="5d5e96f53ebe3f0fc8c9cf150853d6697a09772efcbb6d22ac878003a99ef3f5" Nov 01 16:15:33.279505 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.279469 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892495-2nh9k"] Nov 01 16:15:33.285145 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.285122 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892495-2nh9k"] Nov 01 16:15:33.301140 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.301047 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892495-rwvhk"] Nov 01 16:15:33.514211 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.514135 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892495-rwvhk" Nov 01 16:15:33.559708 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.559675 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf"] Nov 01 16:15:33.560677 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.560653 4363 cpu_manager.go:414] "containerMap: removing container" podUID="689843e8-64df-4f9c-9718-d121ba50a8cb" containerName="cleanup" Nov 01 16:15:33.560677 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.560672 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="689843e8-64df-4f9c-9718-d121ba50a8cb" containerName="cleanup" Nov 01 16:15:33.560865 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.560686 4363 cpu_manager.go:414] "containerMap: removing container" podUID="381f38b2-e936-4b33-8cf1-121a2607045a" containerName="cleanup" Nov 01 16:15:33.560865 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.560691 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="381f38b2-e936-4b33-8cf1-121a2607045a" containerName="cleanup" Nov 01 16:15:33.560865 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.560700 4363 cpu_manager.go:414] "containerMap: removing container" podUID="ff730377-456e-4937-9f9e-f3102f6c3f6a" containerName="cleanup" Nov 01 16:15:33.560865 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.560706 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff730377-456e-4937-9f9e-f3102f6c3f6a" containerName="cleanup" Nov 01 16:15:33.560865 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.560714 4363 cpu_manager.go:414] "containerMap: removing container" podUID="a0918fbb-97c7-4653-ae3d-3de28dc050bf" containerName="cleanup" Nov 01 16:15:33.560865 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.560719 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0918fbb-97c7-4653-ae3d-3de28dc050bf" containerName="cleanup" Nov 01 16:15:33.560865 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.560726 4363 cpu_manager.go:414] "containerMap: removing container" podUID="a4377fa0-6280-4da7-b589-2afd643d3bce" containerName="cleanup" Nov 01 16:15:33.560865 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.560730 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4377fa0-6280-4da7-b589-2afd643d3bce" containerName="cleanup" Nov 01 16:15:33.560865 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.560741 4363 cpu_manager.go:414] "containerMap: removing container" podUID="689843e8-64df-4f9c-9718-d121ba50a8cb" containerName="cleanup" Nov 01 16:15:33.560865 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.560745 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="689843e8-64df-4f9c-9718-d121ba50a8cb" containerName="cleanup" Nov 01 16:15:33.560865 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.560753 4363 cpu_manager.go:414] "containerMap: removing container" podUID="cecab247-7b67-4a90-a3fd-99dfde6c8994" containerName="cleanup" Nov 01 16:15:33.560865 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.560759 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cecab247-7b67-4a90-a3fd-99dfde6c8994" containerName="cleanup" Nov 01 16:15:33.560865 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.560770 4363 cpu_manager.go:414] "containerMap: removing container" podUID="0bf581a3-128c-4b8a-a004-e61af24326c2" containerName="cleanup" Nov 01 16:15:33.560865 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.560776 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bf581a3-128c-4b8a-a004-e61af24326c2" containerName="cleanup" Nov 01 16:15:33.560865 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.560789 4363 cpu_manager.go:414] "containerMap: removing container" podUID="896ad84c-84b6-4f21-b2b8-9689578e43c9" containerName="cleanup" Nov 01 16:15:33.560865 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.560796 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="896ad84c-84b6-4f21-b2b8-9689578e43c9" containerName="cleanup" Nov 01 16:15:33.560865 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.560807 4363 cpu_manager.go:414] "containerMap: removing container" podUID="1191cd6c-176c-4f44-97c3-06942c21b3d9" containerName="cleanup" Nov 01 16:15:33.560865 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.560815 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1191cd6c-176c-4f44-97c3-06942c21b3d9" containerName="cleanup" Nov 01 16:15:33.560865 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.560827 4363 cpu_manager.go:414] "containerMap: removing container" podUID="a4377fa0-6280-4da7-b589-2afd643d3bce" containerName="cleanup" Nov 01 16:15:33.560865 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.560832 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4377fa0-6280-4da7-b589-2afd643d3bce" containerName="cleanup" Nov 01 16:15:33.560865 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.560837 4363 cpu_manager.go:414] "containerMap: removing container" podUID="1191cd6c-176c-4f44-97c3-06942c21b3d9" containerName="cleanup" Nov 01 16:15:33.560865 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.560842 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1191cd6c-176c-4f44-97c3-06942c21b3d9" containerName="cleanup" Nov 01 16:15:33.560865 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.560853 4363 cpu_manager.go:414] "containerMap: removing container" podUID="381f38b2-e936-4b33-8cf1-121a2607045a" containerName="cleanup" Nov 01 16:15:33.560865 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.560861 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="381f38b2-e936-4b33-8cf1-121a2607045a" containerName="cleanup" Nov 01 16:15:33.560865 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.560872 4363 cpu_manager.go:414] "containerMap: removing container" podUID="2a839fcd-b668-4ae2-912c-fc2c812c4a55" containerName="cleanup" Nov 01 16:15:33.560865 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.560877 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a839fcd-b668-4ae2-912c-fc2c812c4a55" containerName="cleanup" Nov 01 16:15:33.562305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.560915 4363 cpu_manager.go:414] "containerMap: removing container" podUID="1191cd6c-176c-4f44-97c3-06942c21b3d9" containerName="cleanup" Nov 01 16:15:33.562305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.560920 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1191cd6c-176c-4f44-97c3-06942c21b3d9" containerName="cleanup" Nov 01 16:15:33.562305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.560934 4363 cpu_manager.go:414] "containerMap: removing container" podUID="896ad84c-84b6-4f21-b2b8-9689578e43c9" containerName="cleanup" Nov 01 16:15:33.562305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.560939 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="896ad84c-84b6-4f21-b2b8-9689578e43c9" containerName="cleanup" Nov 01 16:15:33.562305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.560947 4363 cpu_manager.go:414] "containerMap: removing container" podUID="2a839fcd-b668-4ae2-912c-fc2c812c4a55" containerName="cleanup" Nov 01 16:15:33.562305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.560952 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a839fcd-b668-4ae2-912c-fc2c812c4a55" containerName="cleanup" Nov 01 16:15:33.562305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.560959 4363 cpu_manager.go:414] "containerMap: removing container" podUID="896ad84c-84b6-4f21-b2b8-9689578e43c9" containerName="cleanup" Nov 01 16:15:33.562305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.560963 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="896ad84c-84b6-4f21-b2b8-9689578e43c9" containerName="cleanup" Nov 01 16:15:33.562305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.560971 4363 cpu_manager.go:414] "containerMap: removing container" podUID="ff730377-456e-4937-9f9e-f3102f6c3f6a" containerName="cleanup" Nov 01 16:15:33.562305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.560975 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff730377-456e-4937-9f9e-f3102f6c3f6a" containerName="cleanup" Nov 01 16:15:33.562305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.560985 4363 cpu_manager.go:414] "containerMap: removing container" podUID="cecab247-7b67-4a90-a3fd-99dfde6c8994" containerName="cleanup" Nov 01 16:15:33.562305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.560989 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cecab247-7b67-4a90-a3fd-99dfde6c8994" containerName="cleanup" Nov 01 16:15:33.562305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.560995 4363 cpu_manager.go:414] "containerMap: removing container" podUID="0bf581a3-128c-4b8a-a004-e61af24326c2" containerName="cleanup" Nov 01 16:15:33.562305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.560999 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bf581a3-128c-4b8a-a004-e61af24326c2" containerName="cleanup" Nov 01 16:15:33.562305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.561006 4363 cpu_manager.go:414] "containerMap: removing container" podUID="0bf581a3-128c-4b8a-a004-e61af24326c2" containerName="cleanup" Nov 01 16:15:33.562305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.561010 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bf581a3-128c-4b8a-a004-e61af24326c2" containerName="cleanup" Nov 01 16:15:33.562305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.561036 4363 cpu_manager.go:414] "containerMap: removing container" podUID="a0918fbb-97c7-4653-ae3d-3de28dc050bf" containerName="cleanup" Nov 01 16:15:33.562305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.561042 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0918fbb-97c7-4653-ae3d-3de28dc050bf" containerName="cleanup" Nov 01 16:15:33.562305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.561049 4363 cpu_manager.go:414] "containerMap: removing container" podUID="2a839fcd-b668-4ae2-912c-fc2c812c4a55" containerName="cleanup" Nov 01 16:15:33.562305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.561054 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a839fcd-b668-4ae2-912c-fc2c812c4a55" containerName="cleanup" Nov 01 16:15:33.562305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.561193 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff730377-456e-4937-9f9e-f3102f6c3f6a" containerName="cleanup" Nov 01 16:15:33.562305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.561200 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="896ad84c-84b6-4f21-b2b8-9689578e43c9" containerName="cleanup" Nov 01 16:15:33.562305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.561207 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a839fcd-b668-4ae2-912c-fc2c812c4a55" containerName="cleanup" Nov 01 16:15:33.562305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.561213 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="896ad84c-84b6-4f21-b2b8-9689578e43c9" containerName="cleanup" Nov 01 16:15:33.562305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.561220 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff730377-456e-4937-9f9e-f3102f6c3f6a" containerName="cleanup" Nov 01 16:15:33.562305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.561227 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="cecab247-7b67-4a90-a3fd-99dfde6c8994" containerName="cleanup" Nov 01 16:15:33.562305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.561231 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bf581a3-128c-4b8a-a004-e61af24326c2" containerName="cleanup" Nov 01 16:15:33.562305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.561236 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bf581a3-128c-4b8a-a004-e61af24326c2" containerName="cleanup" Nov 01 16:15:33.562305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.561242 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0918fbb-97c7-4653-ae3d-3de28dc050bf" containerName="cleanup" Nov 01 16:15:33.562305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.561248 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4377fa0-6280-4da7-b589-2afd643d3bce" containerName="cleanup" Nov 01 16:15:33.562305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.561253 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a839fcd-b668-4ae2-912c-fc2c812c4a55" containerName="cleanup" Nov 01 16:15:33.562305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.561257 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="689843e8-64df-4f9c-9718-d121ba50a8cb" containerName="cleanup" Nov 01 16:15:33.562305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.561264 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="381f38b2-e936-4b33-8cf1-121a2607045a" containerName="cleanup" Nov 01 16:15:33.562305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.561270 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0918fbb-97c7-4653-ae3d-3de28dc050bf" containerName="cleanup" Nov 01 16:15:33.562305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.561274 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4377fa0-6280-4da7-b589-2afd643d3bce" containerName="cleanup" Nov 01 16:15:33.562305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.561280 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="689843e8-64df-4f9c-9718-d121ba50a8cb" containerName="cleanup" Nov 01 16:15:33.562305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.561284 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="cecab247-7b67-4a90-a3fd-99dfde6c8994" containerName="cleanup" Nov 01 16:15:33.562305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.561291 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="896ad84c-84b6-4f21-b2b8-9689578e43c9" containerName="cleanup" Nov 01 16:15:33.562305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.561298 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="1191cd6c-176c-4f44-97c3-06942c21b3d9" containerName="cleanup" Nov 01 16:15:33.562305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.561303 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="1191cd6c-176c-4f44-97c3-06942c21b3d9" containerName="cleanup" Nov 01 16:15:33.562305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.561310 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="381f38b2-e936-4b33-8cf1-121a2607045a" containerName="cleanup" Nov 01 16:15:33.562305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.561316 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="381f38b2-e936-4b33-8cf1-121a2607045a" containerName="cleanup" Nov 01 16:15:33.562305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.561325 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="cecab247-7b67-4a90-a3fd-99dfde6c8994" containerName="cleanup" Nov 01 16:15:33.562305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.561331 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="1191cd6c-176c-4f44-97c3-06942c21b3d9" containerName="cleanup" Nov 01 16:15:33.568385 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.568361 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf" Nov 01 16:15:33.571817 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.570805 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf"] Nov 01 16:15:33.575308 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.575275 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-0c091d-serving-cert\"" Nov 01 16:15:33.620985 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.619961 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/689843e8-64df-4f9c-9718-d121ba50a8cb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/689843e8-64df-4f9c-9718-d121ba50a8cb-maas-api-service-ca\") pod \"689843e8-64df-4f9c-9718-d121ba50a8cb\" (UID: \"689843e8-64df-4f9c-9718-d121ba50a8cb\") " Nov 01 16:15:33.620985 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.620067 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/689843e8-64df-4f9c-9718-d121ba50a8cb-kube-api-access-tblg5\" (UniqueName: \"kubernetes.io/projected/689843e8-64df-4f9c-9718-d121ba50a8cb-kube-api-access-tblg5\") pod \"689843e8-64df-4f9c-9718-d121ba50a8cb\" (UID: \"689843e8-64df-4f9c-9718-d121ba50a8cb\") " Nov 01 16:15:33.620985 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.620386 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/689843e8-64df-4f9c-9718-d121ba50a8cb-maas-api-service-ca" pod "689843e8-64df-4f9c-9718-d121ba50a8cb" (UID: "689843e8-64df-4f9c-9718-d121ba50a8cb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:15:33.620985 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.620938 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/689843e8-64df-4f9c-9718-d121ba50a8cb-maas-api-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:33.626643 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.626591 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/689843e8-64df-4f9c-9718-d121ba50a8cb-kube-api-access-tblg5" pod "689843e8-64df-4f9c-9718-d121ba50a8cb" (UID: "689843e8-64df-4f9c-9718-d121ba50a8cb"). InnerVolumeSpecName "kube-api-access-tblg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:15:33.726443 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.724887 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/958066a5-3a94-4ed1-9801-9c011b5cf61c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf\" (UID: \"958066a5-3a94-4ed1-9801-9c011b5cf61c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf" Nov 01 16:15:33.726443 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.724940 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/958066a5-3a94-4ed1-9801-9c011b5cf61c-maas-api-tls\") pod \"maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf\" (UID: \"958066a5-3a94-4ed1-9801-9c011b5cf61c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf" Nov 01 16:15:33.726443 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.725050 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/958066a5-3a94-4ed1-9801-9c011b5cf61c-metrics-certs\") pod \"maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf\" (UID: \"958066a5-3a94-4ed1-9801-9c011b5cf61c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf" Nov 01 16:15:33.726443 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.725236 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shjdk\" (UniqueName: \"kubernetes.io/projected/958066a5-3a94-4ed1-9801-9c011b5cf61c-kube-api-access-shjdk\") pod \"maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf\" (UID: \"958066a5-3a94-4ed1-9801-9c011b5cf61c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf" Nov 01 16:15:33.726443 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.725427 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tblg5\" (UniqueName: \"kubernetes.io/projected/689843e8-64df-4f9c-9718-d121ba50a8cb-kube-api-access-tblg5\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:33.847223 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.847187 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/958066a5-3a94-4ed1-9801-9c011b5cf61c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf\" (UID: \"958066a5-3a94-4ed1-9801-9c011b5cf61c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf" Nov 01 16:15:33.847412 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.847256 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/958066a5-3a94-4ed1-9801-9c011b5cf61c-maas-api-tls\") pod \"maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf\" (UID: \"958066a5-3a94-4ed1-9801-9c011b5cf61c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf" Nov 01 16:15:33.847412 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.847377 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/958066a5-3a94-4ed1-9801-9c011b5cf61c-metrics-certs\") pod \"maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf\" (UID: \"958066a5-3a94-4ed1-9801-9c011b5cf61c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf" Nov 01 16:15:33.847568 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.847546 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shjdk\" (UniqueName: \"kubernetes.io/projected/958066a5-3a94-4ed1-9801-9c011b5cf61c-kube-api-access-shjdk\") pod \"maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf\" (UID: \"958066a5-3a94-4ed1-9801-9c011b5cf61c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf" Nov 01 16:15:33.848648 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.848620 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/958066a5-3a94-4ed1-9801-9c011b5cf61c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf\" (UID: \"958066a5-3a94-4ed1-9801-9c011b5cf61c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf" Nov 01 16:15:33.848756 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:15:33.848740 4363 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-0c091d-serving-cert: secret "maas-api-e2e-inf-b-0c091d-serving-cert" not found Nov 01 16:15:33.848814 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:15:33.848799 4363 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/958066a5-3a94-4ed1-9801-9c011b5cf61c-maas-api-tls podName:958066a5-3a94-4ed1-9801-9c011b5cf61c nodeName:}" failed. No retries permitted until 2026-11-01 16:15:34.348779475 +0000 UTC m=+1667.680517118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/958066a5-3a94-4ed1-9801-9c011b5cf61c-maas-api-tls") pod "maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf" (UID: "958066a5-3a94-4ed1-9801-9c011b5cf61c") : secret "maas-api-e2e-inf-b-0c091d-serving-cert" not found Nov 01 16:15:33.856766 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.856735 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/958066a5-3a94-4ed1-9801-9c011b5cf61c-metrics-certs\") pod \"maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf\" (UID: \"958066a5-3a94-4ed1-9801-9c011b5cf61c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf" Nov 01 16:15:33.876617 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:33.876589 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shjdk\" (UniqueName: \"kubernetes.io/projected/958066a5-3a94-4ed1-9801-9c011b5cf61c-kube-api-access-shjdk\") pod \"maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf\" (UID: \"958066a5-3a94-4ed1-9801-9c011b5cf61c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf" Nov 01 16:15:34.323144 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:34.319510 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892495-rwvhk" Nov 01 16:15:34.323144 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:34.319509 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892495-rwvhk" event={"ID":"689843e8-64df-4f9c-9718-d121ba50a8cb","Type":"ContainerDied","Data":"4f4580b112ba3659e0f6a1c03e79da755924d0d2a48d326d2893fb615ae6df73"} Nov 01 16:15:34.323144 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:34.319683 4363 scope.go:122] "RemoveContainer" containerID="3e94f60ae60478ad03fd2dac474e2675fc897ca76a1d1b6f0bcbb66baf283149" Nov 01 16:15:34.326082 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:34.324539 4363 generic.go:356] "Generic (PLEG): container finished" podID="876c5b6a-7cab-4f19-8300-0fb7624d3e8f" containerID="500d0a8bdf7d0e6e6de4f83b2d48d059ece1cf310f8e89af79a02965e1456fe7" exitCode=60 Nov 01 16:15:34.326082 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:34.324835 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892495-wfnfr" event={"ID":"876c5b6a-7cab-4f19-8300-0fb7624d3e8f","Type":"ContainerDied","Data":"500d0a8bdf7d0e6e6de4f83b2d48d059ece1cf310f8e89af79a02965e1456fe7"} Nov 01 16:15:34.326082 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:34.325412 4363 scope.go:122] "RemoveContainer" containerID="500d0a8bdf7d0e6e6de4f83b2d48d059ece1cf310f8e89af79a02965e1456fe7" Nov 01 16:15:34.326082 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:15:34.325835 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-dfff59-29892495-wfnfr_odh-ai-gateway-infra(876c5b6a-7cab-4f19-8300-0fb7624d3e8f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892495-wfnfr" podUID="876c5b6a-7cab-4f19-8300-0fb7624d3e8f" Nov 01 16:15:34.343468 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:34.343398 4363 scope.go:122] "RemoveContainer" containerID="efd48a99afbe8e80a8ed2cab492fe4e51cb80cdfabde7e3cc651d19abb67651b" Nov 01 16:15:34.362349 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:34.362058 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892495-rwvhk"] Nov 01 16:15:34.366962 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:34.366939 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892495-rwvhk"] Nov 01 16:15:34.375235 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:34.375210 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/958066a5-3a94-4ed1-9801-9c011b5cf61c-maas-api-tls\") pod \"maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf\" (UID: \"958066a5-3a94-4ed1-9801-9c011b5cf61c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf" Nov 01 16:15:34.377679 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:34.377651 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/958066a5-3a94-4ed1-9801-9c011b5cf61c-maas-api-tls\") pod \"maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf\" (UID: \"958066a5-3a94-4ed1-9801-9c011b5cf61c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf" Nov 01 16:15:34.501810 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:34.501774 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf" Nov 01 16:15:34.805105 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:34.805054 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="689843e8-64df-4f9c-9718-d121ba50a8cb" path="/var/lib/kubelet/pods/689843e8-64df-4f9c-9718-d121ba50a8cb/volumes" Nov 01 16:15:34.805859 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:34.805840 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="896ad84c-84b6-4f21-b2b8-9689578e43c9" path="/var/lib/kubelet/pods/896ad84c-84b6-4f21-b2b8-9689578e43c9/volumes" Nov 01 16:15:34.969522 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:15:34.969470 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod958066a5_3a94_4ed1_9801_9c011b5cf61c.slice/crio-756b4ca1bd25ea5669247988401f2428bda4f25a28108452348159a7409e2016 WatchSource:0}: Error finding container 756b4ca1bd25ea5669247988401f2428bda4f25a28108452348159a7409e2016: Status 404 returned error can't find the container with id 756b4ca1bd25ea5669247988401f2428bda4f25a28108452348159a7409e2016 Nov 01 16:15:34.969910 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:34.969888 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf"] Nov 01 16:15:34.971186 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:34.971166 4363 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 16:15:35.389600 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:35.389432 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf" event={"ID":"958066a5-3a94-4ed1-9801-9c011b5cf61c","Type":"ContainerStarted","Data":"756b4ca1bd25ea5669247988401f2428bda4f25a28108452348159a7409e2016"} Nov 01 16:15:35.403653 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:35.403612 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892495-wfnfr"] Nov 01 16:15:35.664446 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:35.664419 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892495-wfnfr" Nov 01 16:15:35.759811 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:35.759783 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/876c5b6a-7cab-4f19-8300-0fb7624d3e8f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/876c5b6a-7cab-4f19-8300-0fb7624d3e8f-maas-api-service-ca\") pod \"876c5b6a-7cab-4f19-8300-0fb7624d3e8f\" (UID: \"876c5b6a-7cab-4f19-8300-0fb7624d3e8f\") " Nov 01 16:15:35.760105 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:35.760089 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/876c5b6a-7cab-4f19-8300-0fb7624d3e8f-kube-api-access-6hgtp\" (UniqueName: \"kubernetes.io/projected/876c5b6a-7cab-4f19-8300-0fb7624d3e8f-kube-api-access-6hgtp\") pod \"876c5b6a-7cab-4f19-8300-0fb7624d3e8f\" (UID: \"876c5b6a-7cab-4f19-8300-0fb7624d3e8f\") " Nov 01 16:15:35.760172 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:35.760136 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/876c5b6a-7cab-4f19-8300-0fb7624d3e8f-maas-api-service-ca" pod "876c5b6a-7cab-4f19-8300-0fb7624d3e8f" (UID: "876c5b6a-7cab-4f19-8300-0fb7624d3e8f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:15:35.760733 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:35.760718 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/876c5b6a-7cab-4f19-8300-0fb7624d3e8f-maas-api-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:35.762203 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:35.762182 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/876c5b6a-7cab-4f19-8300-0fb7624d3e8f-kube-api-access-6hgtp" pod "876c5b6a-7cab-4f19-8300-0fb7624d3e8f" (UID: "876c5b6a-7cab-4f19-8300-0fb7624d3e8f"). InnerVolumeSpecName "kube-api-access-6hgtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:15:35.862534 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:35.862504 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hgtp\" (UniqueName: \"kubernetes.io/projected/876c5b6a-7cab-4f19-8300-0fb7624d3e8f-kube-api-access-6hgtp\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:36.420033 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:36.419822 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf" event={"ID":"958066a5-3a94-4ed1-9801-9c011b5cf61c","Type":"ContainerStarted","Data":"a7de2c32a8369e1508612cfa0e4921e4b431f884999333812ae36514b277f9ac"} Nov 01 16:15:36.420581 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:36.420555 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf" Nov 01 16:15:36.421670 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:36.421645 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892495-wfnfr" event={"ID":"876c5b6a-7cab-4f19-8300-0fb7624d3e8f","Type":"ContainerDied","Data":"d0e9f18d7080262ecd64e1c649b7b49c255e17f09e8b6a5de4960de91da92ac7"} Nov 01 16:15:36.421788 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:36.421676 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892495-wfnfr" Nov 01 16:15:36.421788 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:36.421697 4363 scope.go:122] "RemoveContainer" containerID="500d0a8bdf7d0e6e6de4f83b2d48d059ece1cf310f8e89af79a02965e1456fe7" Nov 01 16:15:36.440985 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:36.440943 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf" podStartSLOduration=2.897902461 podStartE2EDuration="3.44092941s" totalImagesPullingTime="543.026949ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:15:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:15:36.437365812 +0000 UTC m=+1669.769103461" watchObservedRunningTime="2026-11-01 16:15:36.44092941 +0000 UTC m=+1669.772667060" Nov 01 16:15:36.455619 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:36.455595 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892495-wfnfr"] Nov 01 16:15:36.464221 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:36.464198 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892495-wfnfr"] Nov 01 16:15:36.805373 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:36.805290 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="876c5b6a-7cab-4f19-8300-0fb7624d3e8f" path="/var/lib/kubelet/pods/876c5b6a-7cab-4f19-8300-0fb7624d3e8f/volumes" Nov 01 16:15:42.437702 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:15:42.437664 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf" Nov 01 16:15:48.345682 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:15:48.345642 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:16:18.317823 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:16:18.317780 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:16:33.569693 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:33.569657 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84"] Nov 01 16:16:33.570210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:33.569934 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" podUID="afd20bf6-a51f-4006-85a9-1db75176c696" containerName="istio-proxy" containerID="cri-o://b8dc44b4544b118f285de5dbcc15bf3cefde3bed81a0bf0e20d02d96b2c5ece0" gracePeriod=30 Nov 01 16:16:33.577270 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:33.577238 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x"] Nov 01 16:16:33.577507 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:33.577477 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" podUID="4387988a-aa6e-4596-b718-09b8b6de4ac9" containerName="istio-proxy" containerID="cri-o://ab87e1f3aca756b5f373a56bd47328eed33ade55968ec662b6b2dce42ca51600" gracePeriod=30 Nov 01 16:16:39.031522 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.031485 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" Nov 01 16:16:39.110546 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.103353 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl"] Nov 01 16:16:39.110546 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.104428 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" Nov 01 16:16:39.110546 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.105182 4363 cpu_manager.go:414] "containerMap: removing container" podUID="876c5b6a-7cab-4f19-8300-0fb7624d3e8f" containerName="cleanup" Nov 01 16:16:39.110546 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.105201 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="876c5b6a-7cab-4f19-8300-0fb7624d3e8f" containerName="cleanup" Nov 01 16:16:39.110546 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.105217 4363 cpu_manager.go:414] "containerMap: removing container" podUID="a0918fbb-97c7-4653-ae3d-3de28dc050bf" containerName="cleanup" Nov 01 16:16:39.110546 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.105225 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0918fbb-97c7-4653-ae3d-3de28dc050bf" containerName="cleanup" Nov 01 16:16:39.110546 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.105239 4363 cpu_manager.go:414] "containerMap: removing container" podUID="876c5b6a-7cab-4f19-8300-0fb7624d3e8f" containerName="cleanup" Nov 01 16:16:39.110546 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.105246 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="876c5b6a-7cab-4f19-8300-0fb7624d3e8f" containerName="cleanup" Nov 01 16:16:39.110546 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.105257 4363 cpu_manager.go:414] "containerMap: removing container" podUID="a4377fa0-6280-4da7-b589-2afd643d3bce" containerName="cleanup" Nov 01 16:16:39.110546 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.105264 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4377fa0-6280-4da7-b589-2afd643d3bce" containerName="cleanup" Nov 01 16:16:39.110546 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.105278 4363 cpu_manager.go:414] "containerMap: removing container" podUID="689843e8-64df-4f9c-9718-d121ba50a8cb" containerName="cleanup" Nov 01 16:16:39.110546 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.105286 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="689843e8-64df-4f9c-9718-d121ba50a8cb" containerName="cleanup" Nov 01 16:16:39.110546 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.105336 4363 cpu_manager.go:414] "containerMap: removing container" podUID="381f38b2-e936-4b33-8cf1-121a2607045a" containerName="cleanup" Nov 01 16:16:39.110546 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.105344 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="381f38b2-e936-4b33-8cf1-121a2607045a" containerName="cleanup" Nov 01 16:16:39.110546 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.105386 4363 cpu_manager.go:414] "containerMap: removing container" podUID="876c5b6a-7cab-4f19-8300-0fb7624d3e8f" containerName="cleanup" Nov 01 16:16:39.110546 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.105393 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="876c5b6a-7cab-4f19-8300-0fb7624d3e8f" containerName="cleanup" Nov 01 16:16:39.110546 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.105409 4363 cpu_manager.go:414] "containerMap: removing container" podUID="afd20bf6-a51f-4006-85a9-1db75176c696" containerName="istio-proxy" Nov 01 16:16:39.110546 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.105417 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afd20bf6-a51f-4006-85a9-1db75176c696" containerName="istio-proxy" Nov 01 16:16:39.110546 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.105431 4363 cpu_manager.go:414] "containerMap: removing container" podUID="4387988a-aa6e-4596-b718-09b8b6de4ac9" containerName="istio-proxy" Nov 01 16:16:39.110546 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.105437 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4387988a-aa6e-4596-b718-09b8b6de4ac9" containerName="istio-proxy" Nov 01 16:16:39.110546 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.105467 4363 cpu_manager.go:414] "containerMap: removing container" podUID="ff730377-456e-4937-9f9e-f3102f6c3f6a" containerName="cleanup" Nov 01 16:16:39.110546 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.105473 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff730377-456e-4937-9f9e-f3102f6c3f6a" containerName="cleanup" Nov 01 16:16:39.110546 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.105493 4363 cpu_manager.go:414] "containerMap: removing container" podUID="cecab247-7b67-4a90-a3fd-99dfde6c8994" containerName="cleanup" Nov 01 16:16:39.110546 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.105501 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cecab247-7b67-4a90-a3fd-99dfde6c8994" containerName="cleanup" Nov 01 16:16:39.110546 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.105731 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="876c5b6a-7cab-4f19-8300-0fb7624d3e8f" containerName="cleanup" Nov 01 16:16:39.110546 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.105749 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="afd20bf6-a51f-4006-85a9-1db75176c696" containerName="istio-proxy" Nov 01 16:16:39.110546 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.105762 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="4387988a-aa6e-4596-b718-09b8b6de4ac9" containerName="istio-proxy" Nov 01 16:16:39.110546 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.105775 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff730377-456e-4937-9f9e-f3102f6c3f6a" containerName="cleanup" Nov 01 16:16:39.110546 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.105788 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a839fcd-b668-4ae2-912c-fc2c812c4a55" containerName="cleanup" Nov 01 16:16:39.110546 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.105803 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bf581a3-128c-4b8a-a004-e61af24326c2" containerName="cleanup" Nov 01 16:16:39.110546 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.105813 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="689843e8-64df-4f9c-9718-d121ba50a8cb" containerName="cleanup" Nov 01 16:16:39.110546 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.105826 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="876c5b6a-7cab-4f19-8300-0fb7624d3e8f" containerName="cleanup" Nov 01 16:16:39.110546 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.105837 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0918fbb-97c7-4653-ae3d-3de28dc050bf" containerName="cleanup" Nov 01 16:16:39.110546 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.105848 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="876c5b6a-7cab-4f19-8300-0fb7624d3e8f" containerName="cleanup" Nov 01 16:16:39.110546 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.105859 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4377fa0-6280-4da7-b589-2afd643d3bce" containerName="cleanup" Nov 01 16:16:39.114082 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.112681 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" Nov 01 16:16:39.114943 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.114924 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl"] Nov 01 16:16:39.115629 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.115605 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ae79df-openshift-default-dockercfg-mwxlm\"" Nov 01 16:16:39.138777 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.138748 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246"] Nov 01 16:16:39.144781 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.144759 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" Nov 01 16:16:39.158547 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.158517 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246"] Nov 01 16:16:39.203167 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.203132 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4387988a-aa6e-4596-b718-09b8b6de4ac9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4387988a-aa6e-4596-b718-09b8b6de4ac9-workload-certs\") pod \"4387988a-aa6e-4596-b718-09b8b6de4ac9\" (UID: \"4387988a-aa6e-4596-b718-09b8b6de4ac9\") " Nov 01 16:16:39.203358 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.203245 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4387988a-aa6e-4596-b718-09b8b6de4ac9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4387988a-aa6e-4596-b718-09b8b6de4ac9-istio-podinfo\") pod \"4387988a-aa6e-4596-b718-09b8b6de4ac9\" (UID: \"4387988a-aa6e-4596-b718-09b8b6de4ac9\") " Nov 01 16:16:39.203358 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.203305 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afd20bf6-a51f-4006-85a9-1db75176c696-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd20bf6-a51f-4006-85a9-1db75176c696-workload-socket\") pod \"afd20bf6-a51f-4006-85a9-1db75176c696\" (UID: \"afd20bf6-a51f-4006-85a9-1db75176c696\") " Nov 01 16:16:39.203475 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.203346 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afd20bf6-a51f-4006-85a9-1db75176c696-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd20bf6-a51f-4006-85a9-1db75176c696-workload-certs\") pod \"afd20bf6-a51f-4006-85a9-1db75176c696\" (UID: \"afd20bf6-a51f-4006-85a9-1db75176c696\") " Nov 01 16:16:39.203475 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.203415 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afd20bf6-a51f-4006-85a9-1db75176c696-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd20bf6-a51f-4006-85a9-1db75176c696-istio-data\") pod \"afd20bf6-a51f-4006-85a9-1db75176c696\" (UID: \"afd20bf6-a51f-4006-85a9-1db75176c696\") " Nov 01 16:16:39.203475 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.203440 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4387988a-aa6e-4596-b718-09b8b6de4ac9-workload-certs" pod "4387988a-aa6e-4596-b718-09b8b6de4ac9" (UID: "4387988a-aa6e-4596-b718-09b8b6de4ac9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:16:39.203626 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.203494 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afd20bf6-a51f-4006-85a9-1db75176c696-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd20bf6-a51f-4006-85a9-1db75176c696-istio-envoy\") pod \"afd20bf6-a51f-4006-85a9-1db75176c696\" (UID: \"afd20bf6-a51f-4006-85a9-1db75176c696\") " Nov 01 16:16:39.203626 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.203597 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4387988a-aa6e-4596-b718-09b8b6de4ac9-kube-api-access-hcjwp\" (UniqueName: \"kubernetes.io/projected/4387988a-aa6e-4596-b718-09b8b6de4ac9-kube-api-access-hcjwp\") pod \"4387988a-aa6e-4596-b718-09b8b6de4ac9\" (UID: \"4387988a-aa6e-4596-b718-09b8b6de4ac9\") " Nov 01 16:16:39.203724 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.203640 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4387988a-aa6e-4596-b718-09b8b6de4ac9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4387988a-aa6e-4596-b718-09b8b6de4ac9-credential-socket\") pod \"4387988a-aa6e-4596-b718-09b8b6de4ac9\" (UID: \"4387988a-aa6e-4596-b718-09b8b6de4ac9\") " Nov 01 16:16:39.203724 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.203660 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd20bf6-a51f-4006-85a9-1db75176c696-workload-socket" pod "afd20bf6-a51f-4006-85a9-1db75176c696" (UID: "afd20bf6-a51f-4006-85a9-1db75176c696"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:16:39.203724 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.203695 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4387988a-aa6e-4596-b718-09b8b6de4ac9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4387988a-aa6e-4596-b718-09b8b6de4ac9-istio-envoy\") pod \"4387988a-aa6e-4596-b718-09b8b6de4ac9\" (UID: \"4387988a-aa6e-4596-b718-09b8b6de4ac9\") " Nov 01 16:16:39.203861 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.203787 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afd20bf6-a51f-4006-85a9-1db75176c696-kube-api-access-ngncl\" (UniqueName: \"kubernetes.io/projected/afd20bf6-a51f-4006-85a9-1db75176c696-kube-api-access-ngncl\") pod \"afd20bf6-a51f-4006-85a9-1db75176c696\" (UID: \"afd20bf6-a51f-4006-85a9-1db75176c696\") " Nov 01 16:16:39.203861 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.203844 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4387988a-aa6e-4596-b718-09b8b6de4ac9-istio-token\" (UniqueName: \"kubernetes.io/projected/4387988a-aa6e-4596-b718-09b8b6de4ac9-istio-token\") pod \"4387988a-aa6e-4596-b718-09b8b6de4ac9\" (UID: \"4387988a-aa6e-4596-b718-09b8b6de4ac9\") " Nov 01 16:16:39.203957 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.203917 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/afd20bf6-a51f-4006-85a9-1db75176c696-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd20bf6-a51f-4006-85a9-1db75176c696-istio-podinfo\") pod \"afd20bf6-a51f-4006-85a9-1db75176c696\" (UID: \"afd20bf6-a51f-4006-85a9-1db75176c696\") " Nov 01 16:16:39.204008 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.203967 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4387988a-aa6e-4596-b718-09b8b6de4ac9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4387988a-aa6e-4596-b718-09b8b6de4ac9-istiod-ca-cert\") pod \"4387988a-aa6e-4596-b718-09b8b6de4ac9\" (UID: \"4387988a-aa6e-4596-b718-09b8b6de4ac9\") " Nov 01 16:16:39.204106 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.204049 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afd20bf6-a51f-4006-85a9-1db75176c696-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd20bf6-a51f-4006-85a9-1db75176c696-istiod-ca-cert\") pod \"afd20bf6-a51f-4006-85a9-1db75176c696\" (UID: \"afd20bf6-a51f-4006-85a9-1db75176c696\") " Nov 01 16:16:39.204173 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.204157 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afd20bf6-a51f-4006-85a9-1db75176c696-istio-token\" (UniqueName: \"kubernetes.io/projected/afd20bf6-a51f-4006-85a9-1db75176c696-istio-token\") pod \"afd20bf6-a51f-4006-85a9-1db75176c696\" (UID: \"afd20bf6-a51f-4006-85a9-1db75176c696\") " Nov 01 16:16:39.204458 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.204365 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4387988a-aa6e-4596-b718-09b8b6de4ac9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4387988a-aa6e-4596-b718-09b8b6de4ac9-istio-data\") pod \"4387988a-aa6e-4596-b718-09b8b6de4ac9\" (UID: \"4387988a-aa6e-4596-b718-09b8b6de4ac9\") " Nov 01 16:16:39.204458 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.204406 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4387988a-aa6e-4596-b718-09b8b6de4ac9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4387988a-aa6e-4596-b718-09b8b6de4ac9-workload-socket\") pod \"4387988a-aa6e-4596-b718-09b8b6de4ac9\" (UID: \"4387988a-aa6e-4596-b718-09b8b6de4ac9\") " Nov 01 16:16:39.204595 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.204572 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afd20bf6-a51f-4006-85a9-1db75176c696-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd20bf6-a51f-4006-85a9-1db75176c696-credential-socket\") pod \"afd20bf6-a51f-4006-85a9-1db75176c696\" (UID: \"afd20bf6-a51f-4006-85a9-1db75176c696\") " Nov 01 16:16:39.205311 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.205039 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-istiod-ca-cert\") pod \"e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl\" (UID: \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\") " pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" Nov 01 16:16:39.205311 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.205084 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-workload-socket\") pod \"e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl\" (UID: \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\") " pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" Nov 01 16:16:39.205311 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.205149 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-istio-token\") pod \"e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl\" (UID: \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\") " pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" Nov 01 16:16:39.205519 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.205432 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-istio-envoy\") pod \"e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl\" (UID: \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\") " pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" Nov 01 16:16:39.205519 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.205495 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-workload-certs\") pod \"e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl\" (UID: \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\") " pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" Nov 01 16:16:39.205734 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.205715 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-credential-socket\") pod \"e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl\" (UID: \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\") " pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" Nov 01 16:16:39.205799 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.205750 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-istio-data\") pod \"e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl\" (UID: \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\") " pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" Nov 01 16:16:39.205799 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.205784 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-istio-podinfo\") pod \"e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl\" (UID: \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\") " pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" Nov 01 16:16:39.205894 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.205880 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8xph\" (UniqueName: \"kubernetes.io/projected/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-kube-api-access-b8xph\") pod \"e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl\" (UID: \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\") " pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" Nov 01 16:16:39.205972 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.205958 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4387988a-aa6e-4596-b718-09b8b6de4ac9-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:16:39.206044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.205980 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd20bf6-a51f-4006-85a9-1db75176c696-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:16:39.206044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.205997 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afd20bf6-a51f-4006-85a9-1db75176c696-istiod-ca-cert" pod "afd20bf6-a51f-4006-85a9-1db75176c696" (UID: "afd20bf6-a51f-4006-85a9-1db75176c696"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:16:39.206311 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.206231 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4387988a-aa6e-4596-b718-09b8b6de4ac9-istio-data" pod "4387988a-aa6e-4596-b718-09b8b6de4ac9" (UID: "4387988a-aa6e-4596-b718-09b8b6de4ac9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:16:39.206364 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.206305 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd20bf6-a51f-4006-85a9-1db75176c696-workload-certs" pod "afd20bf6-a51f-4006-85a9-1db75176c696" (UID: "afd20bf6-a51f-4006-85a9-1db75176c696"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:16:39.206424 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.206406 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4387988a-aa6e-4596-b718-09b8b6de4ac9-workload-socket" pod "4387988a-aa6e-4596-b718-09b8b6de4ac9" (UID: "4387988a-aa6e-4596-b718-09b8b6de4ac9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:16:39.206477 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.206420 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd20bf6-a51f-4006-85a9-1db75176c696-istio-data" pod "afd20bf6-a51f-4006-85a9-1db75176c696" (UID: "afd20bf6-a51f-4006-85a9-1db75176c696"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:16:39.206593 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.206572 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4387988a-aa6e-4596-b718-09b8b6de4ac9-istiod-ca-cert" pod "4387988a-aa6e-4596-b718-09b8b6de4ac9" (UID: "4387988a-aa6e-4596-b718-09b8b6de4ac9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:16:39.206665 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.206584 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd20bf6-a51f-4006-85a9-1db75176c696-credential-socket" pod "afd20bf6-a51f-4006-85a9-1db75176c696" (UID: "afd20bf6-a51f-4006-85a9-1db75176c696"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:16:39.206899 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.206878 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4387988a-aa6e-4596-b718-09b8b6de4ac9-credential-socket" pod "4387988a-aa6e-4596-b718-09b8b6de4ac9" (UID: "4387988a-aa6e-4596-b718-09b8b6de4ac9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:16:39.209202 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.209158 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/afd20bf6-a51f-4006-85a9-1db75176c696-istio-podinfo" pod "afd20bf6-a51f-4006-85a9-1db75176c696" (UID: "afd20bf6-a51f-4006-85a9-1db75176c696"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:16:39.210082 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.209993 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4387988a-aa6e-4596-b718-09b8b6de4ac9-kube-api-access-hcjwp" pod "4387988a-aa6e-4596-b718-09b8b6de4ac9" (UID: "4387988a-aa6e-4596-b718-09b8b6de4ac9"). InnerVolumeSpecName "kube-api-access-hcjwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:16:39.212418 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.212382 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4387988a-aa6e-4596-b718-09b8b6de4ac9-istio-token" pod "4387988a-aa6e-4596-b718-09b8b6de4ac9" (UID: "4387988a-aa6e-4596-b718-09b8b6de4ac9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:16:39.212633 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.212591 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4387988a-aa6e-4596-b718-09b8b6de4ac9-istio-podinfo" pod "4387988a-aa6e-4596-b718-09b8b6de4ac9" (UID: "4387988a-aa6e-4596-b718-09b8b6de4ac9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:16:39.212780 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.212744 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd20bf6-a51f-4006-85a9-1db75176c696-istio-token" pod "afd20bf6-a51f-4006-85a9-1db75176c696" (UID: "afd20bf6-a51f-4006-85a9-1db75176c696"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:16:39.213085 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.213055 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4387988a-aa6e-4596-b718-09b8b6de4ac9-istio-envoy" pod "4387988a-aa6e-4596-b718-09b8b6de4ac9" (UID: "4387988a-aa6e-4596-b718-09b8b6de4ac9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:16:39.213403 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.213384 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd20bf6-a51f-4006-85a9-1db75176c696-kube-api-access-ngncl" pod "afd20bf6-a51f-4006-85a9-1db75176c696" (UID: "afd20bf6-a51f-4006-85a9-1db75176c696"). InnerVolumeSpecName "kube-api-access-ngncl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:16:39.215147 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.215126 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd20bf6-a51f-4006-85a9-1db75176c696-istio-envoy" pod "afd20bf6-a51f-4006-85a9-1db75176c696" (UID: "afd20bf6-a51f-4006-85a9-1db75176c696"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:16:39.215866 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.215851 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2001b0-openshift-default-dockercfg-dwtwn\"" Nov 01 16:16:39.307483 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.307448 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8xph\" (UniqueName: \"kubernetes.io/projected/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-kube-api-access-b8xph\") pod \"e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl\" (UID: \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\") " pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" Nov 01 16:16:39.307676 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.307516 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/387bf77d-04f3-4915-926c-96bdb07374e6-workload-certs\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246\" (UID: \"387bf77d-04f3-4915-926c-96bdb07374e6\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" Nov 01 16:16:39.307721 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.307694 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/387bf77d-04f3-4915-926c-96bdb07374e6-istio-envoy\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246\" (UID: \"387bf77d-04f3-4915-926c-96bdb07374e6\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" Nov 01 16:16:39.307721 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.307712 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-istiod-ca-cert\") pod \"e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl\" (UID: \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\") " pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" Nov 01 16:16:39.307788 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.307732 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-workload-socket\") pod \"e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl\" (UID: \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\") " pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" Nov 01 16:16:39.307788 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.307767 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/387bf77d-04f3-4915-926c-96bdb07374e6-credential-socket\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246\" (UID: \"387bf77d-04f3-4915-926c-96bdb07374e6\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" Nov 01 16:16:39.307883 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.307869 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-istio-token\") pod \"e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl\" (UID: \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\") " pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" Nov 01 16:16:39.307926 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.307911 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/387bf77d-04f3-4915-926c-96bdb07374e6-istio-podinfo\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246\" (UID: \"387bf77d-04f3-4915-926c-96bdb07374e6\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" Nov 01 16:16:39.307999 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.307981 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/387bf77d-04f3-4915-926c-96bdb07374e6-istio-token\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246\" (UID: \"387bf77d-04f3-4915-926c-96bdb07374e6\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" Nov 01 16:16:39.308124 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.308107 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/387bf77d-04f3-4915-926c-96bdb07374e6-workload-socket\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246\" (UID: \"387bf77d-04f3-4915-926c-96bdb07374e6\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" Nov 01 16:16:39.308228 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.308203 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/387bf77d-04f3-4915-926c-96bdb07374e6-istiod-ca-cert\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246\" (UID: \"387bf77d-04f3-4915-926c-96bdb07374e6\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" Nov 01 16:16:39.308293 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.308254 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-workload-socket\") pod \"e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl\" (UID: \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\") " pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" Nov 01 16:16:39.308395 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.308369 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-istio-envoy\") pod \"e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl\" (UID: \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\") " pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" Nov 01 16:16:39.308463 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.308446 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-workload-certs\") pod \"e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl\" (UID: \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\") " pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" Nov 01 16:16:39.308564 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.308542 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-istiod-ca-cert\") pod \"e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl\" (UID: \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\") " pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" Nov 01 16:16:39.308612 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.308546 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/387bf77d-04f3-4915-926c-96bdb07374e6-istio-data\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246\" (UID: \"387bf77d-04f3-4915-926c-96bdb07374e6\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" Nov 01 16:16:39.308671 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.308649 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb6mp\" (UniqueName: \"kubernetes.io/projected/387bf77d-04f3-4915-926c-96bdb07374e6-kube-api-access-rb6mp\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246\" (UID: \"387bf77d-04f3-4915-926c-96bdb07374e6\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" Nov 01 16:16:39.308789 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.308767 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-credential-socket\") pod \"e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl\" (UID: \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\") " pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" Nov 01 16:16:39.308789 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.308793 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-istio-data\") pod \"e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl\" (UID: \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\") " pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" Nov 01 16:16:39.309118 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.308774 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-workload-certs\") pod \"e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl\" (UID: \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\") " pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" Nov 01 16:16:39.309118 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.308815 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-istio-podinfo\") pod \"e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl\" (UID: \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\") " pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" Nov 01 16:16:39.309118 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.308954 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4387988a-aa6e-4596-b718-09b8b6de4ac9-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:16:39.309118 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.308972 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd20bf6-a51f-4006-85a9-1db75176c696-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:16:39.309118 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.308987 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4387988a-aa6e-4596-b718-09b8b6de4ac9-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:16:39.309118 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.308993 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-credential-socket\") pod \"e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl\" (UID: \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\") " pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" Nov 01 16:16:39.309118 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.309000 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd20bf6-a51f-4006-85a9-1db75176c696-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:16:39.309118 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.309066 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd20bf6-a51f-4006-85a9-1db75176c696-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:16:39.309118 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.309083 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd20bf6-a51f-4006-85a9-1db75176c696-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:16:39.309118 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.309097 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcjwp\" (UniqueName: \"kubernetes.io/projected/4387988a-aa6e-4596-b718-09b8b6de4ac9-kube-api-access-hcjwp\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:16:39.309118 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.309112 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4387988a-aa6e-4596-b718-09b8b6de4ac9-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:16:39.309572 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.309127 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4387988a-aa6e-4596-b718-09b8b6de4ac9-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:16:39.309572 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.309142 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngncl\" (UniqueName: \"kubernetes.io/projected/afd20bf6-a51f-4006-85a9-1db75176c696-kube-api-access-ngncl\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:16:39.309572 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.309156 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4387988a-aa6e-4596-b718-09b8b6de4ac9-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:16:39.309572 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.309170 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd20bf6-a51f-4006-85a9-1db75176c696-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:16:39.309572 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.309185 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4387988a-aa6e-4596-b718-09b8b6de4ac9-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:16:39.309572 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.309201 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd20bf6-a51f-4006-85a9-1db75176c696-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:16:39.309572 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.309215 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd20bf6-a51f-4006-85a9-1db75176c696-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:16:39.309572 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.309232 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4387988a-aa6e-4596-b718-09b8b6de4ac9-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:16:39.309572 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.309182 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-istio-data\") pod \"e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl\" (UID: \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\") " pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" Nov 01 16:16:39.310669 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.310650 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-istio-envoy\") pod \"e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl\" (UID: \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\") " pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" Nov 01 16:16:39.311147 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.311129 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-istio-podinfo\") pod \"e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl\" (UID: \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\") " pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" Nov 01 16:16:39.315216 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.315196 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-istio-token\") pod \"e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl\" (UID: \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\") " pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" Nov 01 16:16:39.315587 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.315568 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8xph\" (UniqueName: \"kubernetes.io/projected/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-kube-api-access-b8xph\") pod \"e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl\" (UID: \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\") " pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" Nov 01 16:16:39.410672 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.410595 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/387bf77d-04f3-4915-926c-96bdb07374e6-workload-certs\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246\" (UID: \"387bf77d-04f3-4915-926c-96bdb07374e6\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" Nov 01 16:16:39.410672 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.410650 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/387bf77d-04f3-4915-926c-96bdb07374e6-istio-envoy\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246\" (UID: \"387bf77d-04f3-4915-926c-96bdb07374e6\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" Nov 01 16:16:39.410887 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.410706 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/387bf77d-04f3-4915-926c-96bdb07374e6-credential-socket\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246\" (UID: \"387bf77d-04f3-4915-926c-96bdb07374e6\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" Nov 01 16:16:39.410887 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.410755 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/387bf77d-04f3-4915-926c-96bdb07374e6-istio-podinfo\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246\" (UID: \"387bf77d-04f3-4915-926c-96bdb07374e6\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" Nov 01 16:16:39.410887 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.410812 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/387bf77d-04f3-4915-926c-96bdb07374e6-istio-token\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246\" (UID: \"387bf77d-04f3-4915-926c-96bdb07374e6\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" Nov 01 16:16:39.410887 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.410878 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/387bf77d-04f3-4915-926c-96bdb07374e6-workload-socket\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246\" (UID: \"387bf77d-04f3-4915-926c-96bdb07374e6\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" Nov 01 16:16:39.411122 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.410927 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/387bf77d-04f3-4915-926c-96bdb07374e6-istiod-ca-cert\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246\" (UID: \"387bf77d-04f3-4915-926c-96bdb07374e6\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" Nov 01 16:16:39.411122 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.411109 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/387bf77d-04f3-4915-926c-96bdb07374e6-workload-certs\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246\" (UID: \"387bf77d-04f3-4915-926c-96bdb07374e6\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" Nov 01 16:16:39.411230 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.411145 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/387bf77d-04f3-4915-926c-96bdb07374e6-istio-data\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246\" (UID: \"387bf77d-04f3-4915-926c-96bdb07374e6\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" Nov 01 16:16:39.411230 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.411204 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb6mp\" (UniqueName: \"kubernetes.io/projected/387bf77d-04f3-4915-926c-96bdb07374e6-kube-api-access-rb6mp\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246\" (UID: \"387bf77d-04f3-4915-926c-96bdb07374e6\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" Nov 01 16:16:39.411336 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.411233 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/387bf77d-04f3-4915-926c-96bdb07374e6-workload-socket\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246\" (UID: \"387bf77d-04f3-4915-926c-96bdb07374e6\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" Nov 01 16:16:39.411534 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.411503 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/387bf77d-04f3-4915-926c-96bdb07374e6-credential-socket\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246\" (UID: \"387bf77d-04f3-4915-926c-96bdb07374e6\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" Nov 01 16:16:39.411625 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.411541 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/387bf77d-04f3-4915-926c-96bdb07374e6-istio-data\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246\" (UID: \"387bf77d-04f3-4915-926c-96bdb07374e6\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" Nov 01 16:16:39.411892 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.411870 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/387bf77d-04f3-4915-926c-96bdb07374e6-istiod-ca-cert\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246\" (UID: \"387bf77d-04f3-4915-926c-96bdb07374e6\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" Nov 01 16:16:39.413223 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.413206 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/387bf77d-04f3-4915-926c-96bdb07374e6-istio-envoy\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246\" (UID: \"387bf77d-04f3-4915-926c-96bdb07374e6\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" Nov 01 16:16:39.413456 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.413430 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/387bf77d-04f3-4915-926c-96bdb07374e6-istio-podinfo\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246\" (UID: \"387bf77d-04f3-4915-926c-96bdb07374e6\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" Nov 01 16:16:39.418828 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.418809 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/387bf77d-04f3-4915-926c-96bdb07374e6-istio-token\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246\" (UID: \"387bf77d-04f3-4915-926c-96bdb07374e6\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" Nov 01 16:16:39.419252 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.419227 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb6mp\" (UniqueName: \"kubernetes.io/projected/387bf77d-04f3-4915-926c-96bdb07374e6-kube-api-access-rb6mp\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246\" (UID: \"387bf77d-04f3-4915-926c-96bdb07374e6\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" Nov 01 16:16:39.433510 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.433486 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" Nov 01 16:16:39.466832 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.466751 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" Nov 01 16:16:39.508750 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.508645 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" event={"ID":"afd20bf6-a51f-4006-85a9-1db75176c696","Type":"ContainerDied","Data":"b8dc44b4544b118f285de5dbcc15bf3cefde3bed81a0bf0e20d02d96b2c5ece0"} Nov 01 16:16:39.508750 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.508655 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" Nov 01 16:16:39.508750 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.508716 4363 scope.go:122] "RemoveContainer" containerID="b8dc44b4544b118f285de5dbcc15bf3cefde3bed81a0bf0e20d02d96b2c5ece0" Nov 01 16:16:39.510834 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.508598 4363 generic.go:356] "Generic (PLEG): container finished" podID="afd20bf6-a51f-4006-85a9-1db75176c696" containerID="b8dc44b4544b118f285de5dbcc15bf3cefde3bed81a0bf0e20d02d96b2c5ece0" exitCode=0 Nov 01 16:16:39.510929 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.510909 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84" event={"ID":"afd20bf6-a51f-4006-85a9-1db75176c696","Type":"ContainerDied","Data":"21e884374ad19f6540e3973ca8ae4be5ba1e74637b9b583871c733c7cfbf3a17"} Nov 01 16:16:39.539187 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.526056 4363 generic.go:356] "Generic (PLEG): container finished" podID="4387988a-aa6e-4596-b718-09b8b6de4ac9" containerID="ab87e1f3aca756b5f373a56bd47328eed33ade55968ec662b6b2dce42ca51600" exitCode=0 Nov 01 16:16:39.539187 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.526102 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" event={"ID":"4387988a-aa6e-4596-b718-09b8b6de4ac9","Type":"ContainerDied","Data":"ab87e1f3aca756b5f373a56bd47328eed33ade55968ec662b6b2dce42ca51600"} Nov 01 16:16:39.539187 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.526154 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" event={"ID":"4387988a-aa6e-4596-b718-09b8b6de4ac9","Type":"ContainerDied","Data":"7f358b106c41ca958190eba96818488509d6f3fb3cab53668686835419e18f13"} Nov 01 16:16:39.539187 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.526233 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x" Nov 01 16:16:39.674010 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.654061 4363 scope.go:122] "RemoveContainer" containerID="b8dc44b4544b118f285de5dbcc15bf3cefde3bed81a0bf0e20d02d96b2c5ece0" Nov 01 16:16:39.674010 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:16:39.668678 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8dc44b4544b118f285de5dbcc15bf3cefde3bed81a0bf0e20d02d96b2c5ece0\": container with ID starting with b8dc44b4544b118f285de5dbcc15bf3cefde3bed81a0bf0e20d02d96b2c5ece0 not found: ID does not exist" containerID="b8dc44b4544b118f285de5dbcc15bf3cefde3bed81a0bf0e20d02d96b2c5ece0" Nov 01 16:16:39.674010 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.668721 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8dc44b4544b118f285de5dbcc15bf3cefde3bed81a0bf0e20d02d96b2c5ece0"} err="failed to get container status \"b8dc44b4544b118f285de5dbcc15bf3cefde3bed81a0bf0e20d02d96b2c5ece0\": rpc error: code = NotFound desc = could not find container \"b8dc44b4544b118f285de5dbcc15bf3cefde3bed81a0bf0e20d02d96b2c5ece0\": container with ID starting with b8dc44b4544b118f285de5dbcc15bf3cefde3bed81a0bf0e20d02d96b2c5ece0 not found: ID does not exist" Nov 01 16:16:39.674010 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.668747 4363 scope.go:122] "RemoveContainer" containerID="ab87e1f3aca756b5f373a56bd47328eed33ade55968ec662b6b2dce42ca51600" Nov 01 16:16:39.674010 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.668873 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84"] Nov 01 16:16:39.743455 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.737404 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r5m84"] Nov 01 16:16:39.745784 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.745464 4363 scope.go:122] "RemoveContainer" containerID="ab87e1f3aca756b5f373a56bd47328eed33ade55968ec662b6b2dce42ca51600" Nov 01 16:16:39.749440 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:16:39.746971 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab87e1f3aca756b5f373a56bd47328eed33ade55968ec662b6b2dce42ca51600\": container with ID starting with ab87e1f3aca756b5f373a56bd47328eed33ade55968ec662b6b2dce42ca51600 not found: ID does not exist" containerID="ab87e1f3aca756b5f373a56bd47328eed33ade55968ec662b6b2dce42ca51600" Nov 01 16:16:39.749440 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.747009 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab87e1f3aca756b5f373a56bd47328eed33ade55968ec662b6b2dce42ca51600"} err="failed to get container status \"ab87e1f3aca756b5f373a56bd47328eed33ade55968ec662b6b2dce42ca51600\": rpc error: code = NotFound desc = could not find container \"ab87e1f3aca756b5f373a56bd47328eed33ade55968ec662b6b2dce42ca51600\": container with ID starting with ab87e1f3aca756b5f373a56bd47328eed33ade55968ec662b6b2dce42ca51600 not found: ID does not exist" Nov 01 16:16:39.749440 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.747132 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x"] Nov 01 16:16:39.757190 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.757164 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q757x"] Nov 01 16:16:39.891078 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:16:39.891003 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod387bf77d_04f3_4915_926c_96bdb07374e6.slice/crio-62aba302bd119edec1826677b4a6e631e1992e91d8061d7ace3f7148ff9ed0a4 WatchSource:0}: Error finding container 62aba302bd119edec1826677b4a6e631e1992e91d8061d7ace3f7148ff9ed0a4: Status 404 returned error can't find the container with id 62aba302bd119edec1826677b4a6e631e1992e91d8061d7ace3f7148ff9ed0a4 Nov 01 16:16:39.895830 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.895799 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246"] Nov 01 16:16:39.896957 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.896918 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:16:39.897100 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.897080 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:16:39.897158 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.897134 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:16:39.958794 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.958768 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl"] Nov 01 16:16:39.959706 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:16:39.959672 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46b2f8eb_f7ee_4cbc_a7db_d910ba142c9e.slice/crio-6a2903101daf1c515b92be3331cb903d2a1eba896ba1f73b3961d892693e72ac WatchSource:0}: Error finding container 6a2903101daf1c515b92be3331cb903d2a1eba896ba1f73b3961d892693e72ac: Status 404 returned error can't find the container with id 6a2903101daf1c515b92be3331cb903d2a1eba896ba1f73b3961d892693e72ac Nov 01 16:16:39.963414 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.963381 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:16:39.963528 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.963444 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:16:39.963528 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:39.963472 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:16:40.539965 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:40.539930 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" event={"ID":"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e","Type":"ContainerStarted","Data":"ec7d89b3ee03862754bd3ca26d3f980319834332ecc5d21c73ca9f51fb7ebc85"} Nov 01 16:16:40.540581 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:40.540546 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" event={"ID":"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e","Type":"ContainerStarted","Data":"6a2903101daf1c515b92be3331cb903d2a1eba896ba1f73b3961d892693e72ac"} Nov 01 16:16:40.542351 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:40.542324 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" event={"ID":"387bf77d-04f3-4915-926c-96bdb07374e6","Type":"ContainerStarted","Data":"155a7788883e55c3998cd540714dfbe616961a4b54eed9ac14bf144d0a501bbf"} Nov 01 16:16:40.542452 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:40.542374 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" event={"ID":"387bf77d-04f3-4915-926c-96bdb07374e6","Type":"ContainerStarted","Data":"62aba302bd119edec1826677b4a6e631e1992e91d8061d7ace3f7148ff9ed0a4"} Nov 01 16:16:40.562085 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:40.562008 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" podStartSLOduration=73.561990028 podStartE2EDuration="1m13.561990028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:16:40.558535216 +0000 UTC m=+1733.890272868" watchObservedRunningTime="2026-11-01 16:16:40.561990028 +0000 UTC m=+1733.893727679" Nov 01 16:16:40.585685 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:40.585627 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" podStartSLOduration=78.585609441 podStartE2EDuration="1m18.585609441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:15:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:16:40.579328636 +0000 UTC m=+1733.911066288" watchObservedRunningTime="2026-11-01 16:16:40.585609441 +0000 UTC m=+1733.917347093" Nov 01 16:16:40.842148 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:40.842067 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4387988a-aa6e-4596-b718-09b8b6de4ac9" path="/var/lib/kubelet/pods/4387988a-aa6e-4596-b718-09b8b6de4ac9/volumes" Nov 01 16:16:40.843229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:40.843208 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afd20bf6-a51f-4006-85a9-1db75176c696" path="/var/lib/kubelet/pods/afd20bf6-a51f-4006-85a9-1db75176c696/volumes" Nov 01 16:16:41.434391 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:41.434354 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" Nov 01 16:16:41.441387 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:41.441362 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" Nov 01 16:16:41.467669 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:41.467640 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" Nov 01 16:16:41.469105 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:41.469080 4363 patch_prober.go:29] interesting pod/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 01 16:16:41.469229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:41.469124 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" podUID="387bf77d-04f3-4915-926c-96bdb07374e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 01 16:16:41.554449 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:41.554410 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" Nov 01 16:16:41.555441 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:41.555417 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" Nov 01 16:16:42.467700 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:42.467663 4363 patch_prober.go:29] interesting pod/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 01 16:16:42.467866 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:42.467745 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" podUID="387bf77d-04f3-4915-926c-96bdb07374e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 01 16:16:43.467257 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:43.467224 4363 patch_prober.go:29] interesting pod/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 01 16:16:43.467673 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:43.467281 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" podUID="387bf77d-04f3-4915-926c-96bdb07374e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 01 16:16:44.490530 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:44.490493 4363 patch_prober.go:29] interesting pod/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 01 16:16:44.507133 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:44.507087 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" podUID="387bf77d-04f3-4915-926c-96bdb07374e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 01 16:16:45.468077 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:45.468041 4363 patch_prober.go:29] interesting pod/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 01 16:16:45.468256 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:45.468105 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" podUID="387bf77d-04f3-4915-926c-96bdb07374e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 01 16:16:46.467570 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:46.467535 4363 patch_prober.go:29] interesting pod/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 01 16:16:46.467994 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:46.467601 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" podUID="387bf77d-04f3-4915-926c-96bdb07374e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 01 16:16:47.472212 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:47.472179 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" Nov 01 16:16:47.472650 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:47.472625 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" Nov 01 16:16:47.473277 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:16:47.473252 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" Nov 01 16:16:48.311556 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:16:48.311515 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:17:18.311980 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:17:18.311945 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:17:30.461074 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:30.460921 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-9e044c/test-model-9e044c-kserve-588c95d597-r2hz8"] Nov 01 16:17:30.468851 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:30.468824 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9e044c/test-model-9e044c-kserve-588c95d597-r2hz8" Nov 01 16:17:30.475376 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:30.473658 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9e044c\"/\"openshift-service-ca.crt\"" Nov 01 16:17:30.475376 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:30.473745 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9e044c\"/\"default-dockercfg-rg8sw\"" Nov 01 16:17:30.475376 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:30.473816 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9e044c\"/\"test-model-9e044c-kserve-self-signed-certs\"" Nov 01 16:17:30.475376 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:30.473864 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9e044c\"/\"kube-root-ca.crt\"" Nov 01 16:17:30.475376 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:30.474240 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9e044c/test-model-9e044c-kserve-588c95d597-r2hz8"] Nov 01 16:17:30.509538 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:30.509503 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a04c3a0-59d4-4214-81a4-5d840c85ce35-model-cache\") pod \"test-model-9e044c-kserve-588c95d597-r2hz8\" (UID: \"6a04c3a0-59d4-4214-81a4-5d840c85ce35\") " pod="ai-tenant-e2e-inf-a-9e044c/test-model-9e044c-kserve-588c95d597-r2hz8" Nov 01 16:17:30.509725 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:30.509557 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a04c3a0-59d4-4214-81a4-5d840c85ce35-home\") pod \"test-model-9e044c-kserve-588c95d597-r2hz8\" (UID: \"6a04c3a0-59d4-4214-81a4-5d840c85ce35\") " pod="ai-tenant-e2e-inf-a-9e044c/test-model-9e044c-kserve-588c95d597-r2hz8" Nov 01 16:17:30.509725 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:30.509651 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a04c3a0-59d4-4214-81a4-5d840c85ce35-dshm\") pod \"test-model-9e044c-kserve-588c95d597-r2hz8\" (UID: \"6a04c3a0-59d4-4214-81a4-5d840c85ce35\") " pod="ai-tenant-e2e-inf-a-9e044c/test-model-9e044c-kserve-588c95d597-r2hz8" Nov 01 16:17:30.509725 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:30.509695 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a04c3a0-59d4-4214-81a4-5d840c85ce35-tmp-dir\") pod \"test-model-9e044c-kserve-588c95d597-r2hz8\" (UID: \"6a04c3a0-59d4-4214-81a4-5d840c85ce35\") " pod="ai-tenant-e2e-inf-a-9e044c/test-model-9e044c-kserve-588c95d597-r2hz8" Nov 01 16:17:30.510055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:30.509855 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a04c3a0-59d4-4214-81a4-5d840c85ce35-tls-certs\") pod \"test-model-9e044c-kserve-588c95d597-r2hz8\" (UID: \"6a04c3a0-59d4-4214-81a4-5d840c85ce35\") " pod="ai-tenant-e2e-inf-a-9e044c/test-model-9e044c-kserve-588c95d597-r2hz8" Nov 01 16:17:30.510055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:30.509921 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szjzn\" (UniqueName: \"kubernetes.io/projected/6a04c3a0-59d4-4214-81a4-5d840c85ce35-kube-api-access-szjzn\") pod \"test-model-9e044c-kserve-588c95d597-r2hz8\" (UID: \"6a04c3a0-59d4-4214-81a4-5d840c85ce35\") " pod="ai-tenant-e2e-inf-a-9e044c/test-model-9e044c-kserve-588c95d597-r2hz8" Nov 01 16:17:30.638038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:30.625475 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a04c3a0-59d4-4214-81a4-5d840c85ce35-tmp-dir\") pod \"test-model-9e044c-kserve-588c95d597-r2hz8\" (UID: \"6a04c3a0-59d4-4214-81a4-5d840c85ce35\") " pod="ai-tenant-e2e-inf-a-9e044c/test-model-9e044c-kserve-588c95d597-r2hz8" Nov 01 16:17:30.664163 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:30.664124 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a04c3a0-59d4-4214-81a4-5d840c85ce35-tls-certs\") pod \"test-model-9e044c-kserve-588c95d597-r2hz8\" (UID: \"6a04c3a0-59d4-4214-81a4-5d840c85ce35\") " pod="ai-tenant-e2e-inf-a-9e044c/test-model-9e044c-kserve-588c95d597-r2hz8" Nov 01 16:17:30.664520 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:30.664496 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szjzn\" (UniqueName: \"kubernetes.io/projected/6a04c3a0-59d4-4214-81a4-5d840c85ce35-kube-api-access-szjzn\") pod \"test-model-9e044c-kserve-588c95d597-r2hz8\" (UID: \"6a04c3a0-59d4-4214-81a4-5d840c85ce35\") " pod="ai-tenant-e2e-inf-a-9e044c/test-model-9e044c-kserve-588c95d597-r2hz8" Nov 01 16:17:30.664842 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:30.664811 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a04c3a0-59d4-4214-81a4-5d840c85ce35-model-cache\") pod \"test-model-9e044c-kserve-588c95d597-r2hz8\" (UID: \"6a04c3a0-59d4-4214-81a4-5d840c85ce35\") " pod="ai-tenant-e2e-inf-a-9e044c/test-model-9e044c-kserve-588c95d597-r2hz8" Nov 01 16:17:30.689687 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:30.689652 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a04c3a0-59d4-4214-81a4-5d840c85ce35-home\") pod \"test-model-9e044c-kserve-588c95d597-r2hz8\" (UID: \"6a04c3a0-59d4-4214-81a4-5d840c85ce35\") " pod="ai-tenant-e2e-inf-a-9e044c/test-model-9e044c-kserve-588c95d597-r2hz8" Nov 01 16:17:30.689969 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:30.689949 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a04c3a0-59d4-4214-81a4-5d840c85ce35-dshm\") pod \"test-model-9e044c-kserve-588c95d597-r2hz8\" (UID: \"6a04c3a0-59d4-4214-81a4-5d840c85ce35\") " pod="ai-tenant-e2e-inf-a-9e044c/test-model-9e044c-kserve-588c95d597-r2hz8" Nov 01 16:17:30.699045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:30.691865 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a04c3a0-59d4-4214-81a4-5d840c85ce35-home\") pod \"test-model-9e044c-kserve-588c95d597-r2hz8\" (UID: \"6a04c3a0-59d4-4214-81a4-5d840c85ce35\") " pod="ai-tenant-e2e-inf-a-9e044c/test-model-9e044c-kserve-588c95d597-r2hz8" Nov 01 16:17:30.699295 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:30.669396 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a04c3a0-59d4-4214-81a4-5d840c85ce35-model-cache\") pod \"test-model-9e044c-kserve-588c95d597-r2hz8\" (UID: \"6a04c3a0-59d4-4214-81a4-5d840c85ce35\") " pod="ai-tenant-e2e-inf-a-9e044c/test-model-9e044c-kserve-588c95d597-r2hz8" Nov 01 16:17:30.699422 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:30.627475 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a04c3a0-59d4-4214-81a4-5d840c85ce35-tmp-dir\") pod \"test-model-9e044c-kserve-588c95d597-r2hz8\" (UID: \"6a04c3a0-59d4-4214-81a4-5d840c85ce35\") " pod="ai-tenant-e2e-inf-a-9e044c/test-model-9e044c-kserve-588c95d597-r2hz8" Nov 01 16:17:30.706191 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:30.706161 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a04c3a0-59d4-4214-81a4-5d840c85ce35-tls-certs\") pod \"test-model-9e044c-kserve-588c95d597-r2hz8\" (UID: \"6a04c3a0-59d4-4214-81a4-5d840c85ce35\") " pod="ai-tenant-e2e-inf-a-9e044c/test-model-9e044c-kserve-588c95d597-r2hz8" Nov 01 16:17:30.710711 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:30.710679 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a04c3a0-59d4-4214-81a4-5d840c85ce35-dshm\") pod \"test-model-9e044c-kserve-588c95d597-r2hz8\" (UID: \"6a04c3a0-59d4-4214-81a4-5d840c85ce35\") " pod="ai-tenant-e2e-inf-a-9e044c/test-model-9e044c-kserve-588c95d597-r2hz8" Nov 01 16:17:30.736083 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:30.728715 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szjzn\" (UniqueName: \"kubernetes.io/projected/6a04c3a0-59d4-4214-81a4-5d840c85ce35-kube-api-access-szjzn\") pod \"test-model-9e044c-kserve-588c95d597-r2hz8\" (UID: \"6a04c3a0-59d4-4214-81a4-5d840c85ce35\") " pod="ai-tenant-e2e-inf-a-9e044c/test-model-9e044c-kserve-588c95d597-r2hz8" Nov 01 16:17:30.792947 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:30.789802 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9e044c/test-model-9e044c-kserve-588c95d597-r2hz8" Nov 01 16:17:31.237641 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:17:31.237610 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a04c3a0_59d4_4214_81a4_5d840c85ce35.slice/crio-4a5e37ac717bfdb356c96e6b65e87df98860d94f40649b030e6c190ddc5c6391 WatchSource:0}: Error finding container 4a5e37ac717bfdb356c96e6b65e87df98860d94f40649b030e6c190ddc5c6391: Status 404 returned error can't find the container with id 4a5e37ac717bfdb356c96e6b65e87df98860d94f40649b030e6c190ddc5c6391 Nov 01 16:17:31.237889 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:31.237868 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9e044c/test-model-9e044c-kserve-588c95d597-r2hz8"] Nov 01 16:17:31.515094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:31.512198 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9e044c/test-model-9e044c-kserve-588c95d597-r2hz8" event={"ID":"6a04c3a0-59d4-4214-81a4-5d840c85ce35","Type":"ContainerStarted","Data":"8c024ef8e3bd9207a7bcc7b8561ab997e958ca3ffc7b0cd8e94fb00bfd3e383c"} Nov 01 16:17:31.515094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:31.512268 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9e044c/test-model-9e044c-kserve-588c95d597-r2hz8" event={"ID":"6a04c3a0-59d4-4214-81a4-5d840c85ce35","Type":"ContainerStarted","Data":"4a5e37ac717bfdb356c96e6b65e87df98860d94f40649b030e6c190ddc5c6391"} Nov 01 16:17:31.535406 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:31.535355 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-9e044c/test-model-9e044c-kserve-588c95d597-r2hz8" podStartSLOduration=1.535339217 podStartE2EDuration="1.535339217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:17:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:17:31.529459915 +0000 UTC m=+1784.861197567" watchObservedRunningTime="2026-11-01 16:17:31.535339217 +0000 UTC m=+1784.867076866" Nov 01 16:17:31.790258 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:31.790167 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-9e044c/test-model-9e044c-kserve-588c95d597-r2hz8" Nov 01 16:17:40.792039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:40.790588 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-9e044c/test-model-9e044c-kserve-588c95d597-r2hz8" Nov 01 16:17:40.860345 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:40.860308 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-9e044c/test-model-9e044c-kserve-588c95d597-r2hz8" Nov 01 16:17:41.717192 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:41.717160 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-9e044c/test-model-9e044c-kserve-588c95d597-r2hz8" Nov 01 16:17:45.449701 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:45.449669 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ae79df/test-model-ae79df-kserve-7c554b5c5-k248t"] Nov 01 16:17:45.469217 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:45.469179 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ae79df/test-model-ae79df-kserve-7c554b5c5-k248t"] Nov 01 16:17:45.469375 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:45.469316 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ae79df/test-model-ae79df-kserve-7c554b5c5-k248t" Nov 01 16:17:45.473501 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:45.473476 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ae79df\"/\"kube-root-ca.crt\"" Nov 01 16:17:45.473952 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:45.473931 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ae79df\"/\"openshift-service-ca.crt\"" Nov 01 16:17:45.478750 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:45.478730 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ae79df\"/\"default-dockercfg-fdk28\"" Nov 01 16:17:45.478889 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:45.478778 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ae79df\"/\"test-model-ae79df-kserve-self-signed-certs\"" Nov 01 16:17:45.494603 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:45.494554 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/251044fd-4f4d-4423-9f9d-685c35f259a9-tmp-dir\") pod \"test-model-ae79df-kserve-7c554b5c5-k248t\" (UID: \"251044fd-4f4d-4423-9f9d-685c35f259a9\") " pod="ai-tenant-e2e-inf-b-ae79df/test-model-ae79df-kserve-7c554b5c5-k248t" Nov 01 16:17:45.494887 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:45.494860 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smfgs\" (UniqueName: \"kubernetes.io/projected/251044fd-4f4d-4423-9f9d-685c35f259a9-kube-api-access-smfgs\") pod \"test-model-ae79df-kserve-7c554b5c5-k248t\" (UID: \"251044fd-4f4d-4423-9f9d-685c35f259a9\") " pod="ai-tenant-e2e-inf-b-ae79df/test-model-ae79df-kserve-7c554b5c5-k248t" Nov 01 16:17:45.494989 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:45.494909 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/251044fd-4f4d-4423-9f9d-685c35f259a9-home\") pod \"test-model-ae79df-kserve-7c554b5c5-k248t\" (UID: \"251044fd-4f4d-4423-9f9d-685c35f259a9\") " pod="ai-tenant-e2e-inf-b-ae79df/test-model-ae79df-kserve-7c554b5c5-k248t" Nov 01 16:17:45.495114 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:45.495096 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/251044fd-4f4d-4423-9f9d-685c35f259a9-dshm\") pod \"test-model-ae79df-kserve-7c554b5c5-k248t\" (UID: \"251044fd-4f4d-4423-9f9d-685c35f259a9\") " pod="ai-tenant-e2e-inf-b-ae79df/test-model-ae79df-kserve-7c554b5c5-k248t" Nov 01 16:17:45.495185 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:45.495143 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/251044fd-4f4d-4423-9f9d-685c35f259a9-model-cache\") pod \"test-model-ae79df-kserve-7c554b5c5-k248t\" (UID: \"251044fd-4f4d-4423-9f9d-685c35f259a9\") " pod="ai-tenant-e2e-inf-b-ae79df/test-model-ae79df-kserve-7c554b5c5-k248t" Nov 01 16:17:45.495318 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:45.495303 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/251044fd-4f4d-4423-9f9d-685c35f259a9-tls-certs\") pod \"test-model-ae79df-kserve-7c554b5c5-k248t\" (UID: \"251044fd-4f4d-4423-9f9d-685c35f259a9\") " pod="ai-tenant-e2e-inf-b-ae79df/test-model-ae79df-kserve-7c554b5c5-k248t" Nov 01 16:17:45.597605 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:45.597569 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/251044fd-4f4d-4423-9f9d-685c35f259a9-tls-certs\") pod \"test-model-ae79df-kserve-7c554b5c5-k248t\" (UID: \"251044fd-4f4d-4423-9f9d-685c35f259a9\") " pod="ai-tenant-e2e-inf-b-ae79df/test-model-ae79df-kserve-7c554b5c5-k248t" Nov 01 16:17:45.597813 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:45.597768 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/251044fd-4f4d-4423-9f9d-685c35f259a9-tmp-dir\") pod \"test-model-ae79df-kserve-7c554b5c5-k248t\" (UID: \"251044fd-4f4d-4423-9f9d-685c35f259a9\") " pod="ai-tenant-e2e-inf-b-ae79df/test-model-ae79df-kserve-7c554b5c5-k248t" Nov 01 16:17:45.597885 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:45.597862 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smfgs\" (UniqueName: \"kubernetes.io/projected/251044fd-4f4d-4423-9f9d-685c35f259a9-kube-api-access-smfgs\") pod \"test-model-ae79df-kserve-7c554b5c5-k248t\" (UID: \"251044fd-4f4d-4423-9f9d-685c35f259a9\") " pod="ai-tenant-e2e-inf-b-ae79df/test-model-ae79df-kserve-7c554b5c5-k248t" Nov 01 16:17:45.597981 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:45.597902 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/251044fd-4f4d-4423-9f9d-685c35f259a9-home\") pod \"test-model-ae79df-kserve-7c554b5c5-k248t\" (UID: \"251044fd-4f4d-4423-9f9d-685c35f259a9\") " pod="ai-tenant-e2e-inf-b-ae79df/test-model-ae79df-kserve-7c554b5c5-k248t" Nov 01 16:17:45.598124 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:45.598105 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/251044fd-4f4d-4423-9f9d-685c35f259a9-dshm\") pod \"test-model-ae79df-kserve-7c554b5c5-k248t\" (UID: \"251044fd-4f4d-4423-9f9d-685c35f259a9\") " pod="ai-tenant-e2e-inf-b-ae79df/test-model-ae79df-kserve-7c554b5c5-k248t" Nov 01 16:17:45.598299 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:45.598278 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/251044fd-4f4d-4423-9f9d-685c35f259a9-model-cache\") pod \"test-model-ae79df-kserve-7c554b5c5-k248t\" (UID: \"251044fd-4f4d-4423-9f9d-685c35f259a9\") " pod="ai-tenant-e2e-inf-b-ae79df/test-model-ae79df-kserve-7c554b5c5-k248t" Nov 01 16:17:45.598384 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:45.598235 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/251044fd-4f4d-4423-9f9d-685c35f259a9-tmp-dir\") pod \"test-model-ae79df-kserve-7c554b5c5-k248t\" (UID: \"251044fd-4f4d-4423-9f9d-685c35f259a9\") " pod="ai-tenant-e2e-inf-b-ae79df/test-model-ae79df-kserve-7c554b5c5-k248t" Nov 01 16:17:45.598697 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:45.598673 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/251044fd-4f4d-4423-9f9d-685c35f259a9-model-cache\") pod \"test-model-ae79df-kserve-7c554b5c5-k248t\" (UID: \"251044fd-4f4d-4423-9f9d-685c35f259a9\") " pod="ai-tenant-e2e-inf-b-ae79df/test-model-ae79df-kserve-7c554b5c5-k248t" Nov 01 16:17:45.598856 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:45.598761 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/251044fd-4f4d-4423-9f9d-685c35f259a9-home\") pod \"test-model-ae79df-kserve-7c554b5c5-k248t\" (UID: \"251044fd-4f4d-4423-9f9d-685c35f259a9\") " pod="ai-tenant-e2e-inf-b-ae79df/test-model-ae79df-kserve-7c554b5c5-k248t" Nov 01 16:17:45.600915 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:45.600887 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/251044fd-4f4d-4423-9f9d-685c35f259a9-dshm\") pod \"test-model-ae79df-kserve-7c554b5c5-k248t\" (UID: \"251044fd-4f4d-4423-9f9d-685c35f259a9\") " pod="ai-tenant-e2e-inf-b-ae79df/test-model-ae79df-kserve-7c554b5c5-k248t" Nov 01 16:17:45.601215 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:45.601195 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/251044fd-4f4d-4423-9f9d-685c35f259a9-tls-certs\") pod \"test-model-ae79df-kserve-7c554b5c5-k248t\" (UID: \"251044fd-4f4d-4423-9f9d-685c35f259a9\") " pod="ai-tenant-e2e-inf-b-ae79df/test-model-ae79df-kserve-7c554b5c5-k248t" Nov 01 16:17:45.606098 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:45.605996 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smfgs\" (UniqueName: \"kubernetes.io/projected/251044fd-4f4d-4423-9f9d-685c35f259a9-kube-api-access-smfgs\") pod \"test-model-ae79df-kserve-7c554b5c5-k248t\" (UID: \"251044fd-4f4d-4423-9f9d-685c35f259a9\") " pod="ai-tenant-e2e-inf-b-ae79df/test-model-ae79df-kserve-7c554b5c5-k248t" Nov 01 16:17:45.801738 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:45.801548 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ae79df/test-model-ae79df-kserve-7c554b5c5-k248t" Nov 01 16:17:46.265468 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:46.265394 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ae79df/test-model-ae79df-kserve-7c554b5c5-k248t"] Nov 01 16:17:47.007038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:47.000838 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ae79df/test-model-ae79df-kserve-7c554b5c5-k248t" event={"ID":"251044fd-4f4d-4423-9f9d-685c35f259a9","Type":"ContainerStarted","Data":"18fea58b64ec153d557805b6a0835c2e0f5c360a66e945f244666491cfb5c2fe"} Nov 01 16:17:47.007038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:47.000893 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ae79df/test-model-ae79df-kserve-7c554b5c5-k248t" event={"ID":"251044fd-4f4d-4423-9f9d-685c35f259a9","Type":"ContainerStarted","Data":"5f1d5d0ac3c02fa1d349117a3137ad505a0fb5fc5163ca61d1abed33d10cf913"} Nov 01 16:17:47.038039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:47.031822 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ae79df/test-model-ae79df-kserve-7c554b5c5-k248t" podStartSLOduration=2.031802508 podStartE2EDuration="2.031802508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:17:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:17:47.023569604 +0000 UTC m=+1800.355307255" watchObservedRunningTime="2026-11-01 16:17:47.031802508 +0000 UTC m=+1800.363540161" Nov 01 16:17:47.679921 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:47.679889 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtwhw_8254aafa-dd5b-4150-87e4-d165dab6bcb5/ovn-acl-logging/0.log" Nov 01 16:17:47.680480 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:47.680457 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtwhw_8254aafa-dd5b-4150-87e4-d165dab6bcb5/ovn-acl-logging/0.log" Nov 01 16:17:47.802638 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:47.802600 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ae79df/test-model-ae79df-kserve-7c554b5c5-k248t" Nov 01 16:17:48.313369 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:17:48.313330 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:17:55.824091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:55.807828 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ae79df/test-model-ae79df-kserve-7c554b5c5-k248t" Nov 01 16:17:55.868599 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:55.865164 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ae79df/test-model-ae79df-kserve-7c554b5c5-k248t" Nov 01 16:17:56.160580 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:56.160546 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ae79df/test-model-ae79df-kserve-7c554b5c5-k248t" Nov 01 16:17:57.599202 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:57.599148 4363 patch_prober.go:29] interesting pod/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:17:57.599726 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:17:57.599235 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" podUID="46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:18:01.199958 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:01.199926 4363 generic.go:356] "Generic (PLEG): container finished" podID="46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e" containerID="ec7d89b3ee03862754bd3ca26d3f980319834332ecc5d21c73ca9f51fb7ebc85" exitCode=0 Nov 01 16:18:01.199958 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:01.199962 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" event={"ID":"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e","Type":"ContainerDied","Data":"ec7d89b3ee03862754bd3ca26d3f980319834332ecc5d21c73ca9f51fb7ebc85"} Nov 01 16:18:01.200656 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:01.200393 4363 scope.go:122] "RemoveContainer" containerID="ec7d89b3ee03862754bd3ca26d3f980319834332ecc5d21c73ca9f51fb7ebc85" Nov 01 16:18:01.202449 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:01.202416 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:18:01.202558 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:01.202492 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:18:01.202558 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:01.202526 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:18:02.356272 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:02.356232 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" event={"ID":"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e","Type":"ContainerStarted","Data":"e4b4be64281d6c8c85fa6b77e79170b4da2abada82b76cf8e6ff679c6cfa106c"} Nov 01 16:18:02.433779 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:02.433740 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" Nov 01 16:18:02.433954 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:02.433927 4363 patch_prober.go:29] interesting pod/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Nov 01 16:18:02.434009 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:02.433986 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" podUID="46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Nov 01 16:18:03.434857 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:03.434819 4363 patch_prober.go:29] interesting pod/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Nov 01 16:18:03.435301 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:03.434886 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" podUID="46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Nov 01 16:18:04.449936 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:04.449901 4363 patch_prober.go:29] interesting pod/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Nov 01 16:18:04.456847 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:04.456809 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" podUID="46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Nov 01 16:18:05.434592 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:05.434559 4363 patch_prober.go:29] interesting pod/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Nov 01 16:18:05.434814 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:05.434612 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" podUID="46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Nov 01 16:18:05.669716 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:05.669682 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv"] Nov 01 16:18:06.435304 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:06.435262 4363 patch_prober.go:29] interesting pod/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Nov 01 16:18:06.435529 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:06.435327 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" podUID="46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Nov 01 16:18:07.434086 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:07.434047 4363 patch_prober.go:29] interesting pod/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Nov 01 16:18:07.434495 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:07.434102 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" podUID="46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Nov 01 16:18:08.439480 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:08.439451 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" Nov 01 16:18:08.439997 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:08.439972 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" Nov 01 16:18:08.440909 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:08.440889 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" Nov 01 16:18:18.312547 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:18:18.312507 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:18:21.340737 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:21.340686 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" podUID="a15aa191-e376-490a-8763-16cdd7685b5c" containerName="main" containerID="cri-o://ef485f1fb3ff984f14ae6058a97a1dafe21b4c7319c1dc2a89dfeda2ccf16116" gracePeriod=45 Nov 01 16:18:21.686621 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:21.686585 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" Nov 01 16:18:21.829832 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:21.829798 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a15aa191-e376-490a-8763-16cdd7685b5c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a15aa191-e376-490a-8763-16cdd7685b5c-tmp-dir\") pod \"a15aa191-e376-490a-8763-16cdd7685b5c\" (UID: \"a15aa191-e376-490a-8763-16cdd7685b5c\") " Nov 01 16:18:21.830132 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:21.830111 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a15aa191-e376-490a-8763-16cdd7685b5c-tls-certs\" (UniqueName: \"kubernetes.io/secret/a15aa191-e376-490a-8763-16cdd7685b5c-tls-certs\") pod \"a15aa191-e376-490a-8763-16cdd7685b5c\" (UID: \"a15aa191-e376-490a-8763-16cdd7685b5c\") " Nov 01 16:18:21.830301 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:21.830285 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a15aa191-e376-490a-8763-16cdd7685b5c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a15aa191-e376-490a-8763-16cdd7685b5c-model-cache\") pod \"a15aa191-e376-490a-8763-16cdd7685b5c\" (UID: \"a15aa191-e376-490a-8763-16cdd7685b5c\") " Nov 01 16:18:21.830559 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:21.830542 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a15aa191-e376-490a-8763-16cdd7685b5c-home\" (UniqueName: \"kubernetes.io/empty-dir/a15aa191-e376-490a-8763-16cdd7685b5c-home\") pod \"a15aa191-e376-490a-8763-16cdd7685b5c\" (UID: \"a15aa191-e376-490a-8763-16cdd7685b5c\") " Nov 01 16:18:21.830814 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:21.830798 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a15aa191-e376-490a-8763-16cdd7685b5c-kube-api-access-s4t5m\" (UniqueName: \"kubernetes.io/projected/a15aa191-e376-490a-8763-16cdd7685b5c-kube-api-access-s4t5m\") pod \"a15aa191-e376-490a-8763-16cdd7685b5c\" (UID: \"a15aa191-e376-490a-8763-16cdd7685b5c\") " Nov 01 16:18:21.831100 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:21.831072 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a15aa191-e376-490a-8763-16cdd7685b5c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a15aa191-e376-490a-8763-16cdd7685b5c-dshm\") pod \"a15aa191-e376-490a-8763-16cdd7685b5c\" (UID: \"a15aa191-e376-490a-8763-16cdd7685b5c\") " Nov 01 16:18:21.833861 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:21.833600 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a15aa191-e376-490a-8763-16cdd7685b5c-tmp-dir" pod "a15aa191-e376-490a-8763-16cdd7685b5c" (UID: "a15aa191-e376-490a-8763-16cdd7685b5c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:21.833861 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:21.833679 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a15aa191-e376-490a-8763-16cdd7685b5c-home" pod "a15aa191-e376-490a-8763-16cdd7685b5c" (UID: "a15aa191-e376-490a-8763-16cdd7685b5c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:21.834457 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:21.834433 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a15aa191-e376-490a-8763-16cdd7685b5c-model-cache" pod "a15aa191-e376-490a-8763-16cdd7685b5c" (UID: "a15aa191-e376-490a-8763-16cdd7685b5c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:21.874817 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:21.874267 4363 generic.go:356] "Generic (PLEG): container finished" podID="a15aa191-e376-490a-8763-16cdd7685b5c" containerID="ef485f1fb3ff984f14ae6058a97a1dafe21b4c7319c1dc2a89dfeda2ccf16116" exitCode=0 Nov 01 16:18:21.874817 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:21.874519 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" event={"ID":"a15aa191-e376-490a-8763-16cdd7685b5c","Type":"ContainerDied","Data":"ef485f1fb3ff984f14ae6058a97a1dafe21b4c7319c1dc2a89dfeda2ccf16116"} Nov 01 16:18:21.874817 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:21.874557 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" event={"ID":"a15aa191-e376-490a-8763-16cdd7685b5c","Type":"ContainerDied","Data":"99e1d2294aaa2b032bf227f67c32c73f305352abf7299e6d67ba6f278761cf1d"} Nov 01 16:18:21.874817 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:21.874580 4363 scope.go:122] "RemoveContainer" containerID="ef485f1fb3ff984f14ae6058a97a1dafe21b4c7319c1dc2a89dfeda2ccf16116" Nov 01 16:18:21.874817 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:21.874742 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" Nov 01 16:18:21.874817 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:21.874740 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a15aa191-e376-490a-8763-16cdd7685b5c-kube-api-access-s4t5m" pod "a15aa191-e376-490a-8763-16cdd7685b5c" (UID: "a15aa191-e376-490a-8763-16cdd7685b5c"). InnerVolumeSpecName "kube-api-access-s4t5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:18:21.875337 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:21.875310 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a15aa191-e376-490a-8763-16cdd7685b5c-tls-certs" pod "a15aa191-e376-490a-8763-16cdd7685b5c" (UID: "a15aa191-e376-490a-8763-16cdd7685b5c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:18:21.884051 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:21.881466 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a15aa191-e376-490a-8763-16cdd7685b5c-dshm" pod "a15aa191-e376-490a-8763-16cdd7685b5c" (UID: "a15aa191-e376-490a-8763-16cdd7685b5c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:21.925427 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:21.925398 4363 scope.go:122] "RemoveContainer" containerID="ef485f1fb3ff984f14ae6058a97a1dafe21b4c7319c1dc2a89dfeda2ccf16116" Nov 01 16:18:21.925873 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:18:21.925842 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef485f1fb3ff984f14ae6058a97a1dafe21b4c7319c1dc2a89dfeda2ccf16116\": container with ID starting with ef485f1fb3ff984f14ae6058a97a1dafe21b4c7319c1dc2a89dfeda2ccf16116 not found: ID does not exist" containerID="ef485f1fb3ff984f14ae6058a97a1dafe21b4c7319c1dc2a89dfeda2ccf16116" Nov 01 16:18:21.925982 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:21.925884 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef485f1fb3ff984f14ae6058a97a1dafe21b4c7319c1dc2a89dfeda2ccf16116"} err="failed to get container status \"ef485f1fb3ff984f14ae6058a97a1dafe21b4c7319c1dc2a89dfeda2ccf16116\": rpc error: code = NotFound desc = could not find container \"ef485f1fb3ff984f14ae6058a97a1dafe21b4c7319c1dc2a89dfeda2ccf16116\": container with ID starting with ef485f1fb3ff984f14ae6058a97a1dafe21b4c7319c1dc2a89dfeda2ccf16116 not found: ID does not exist" Nov 01 16:18:21.934576 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:21.934552 4363 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a15aa191-e376-490a-8763-16cdd7685b5c-home\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.934667 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:21.934580 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4t5m\" (UniqueName: \"kubernetes.io/projected/a15aa191-e376-490a-8763-16cdd7685b5c-kube-api-access-s4t5m\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.934667 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:21.934590 4363 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a15aa191-e376-490a-8763-16cdd7685b5c-dshm\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.934667 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:21.934601 4363 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a15aa191-e376-490a-8763-16cdd7685b5c-tmp-dir\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.934667 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:21.934610 4363 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a15aa191-e376-490a-8763-16cdd7685b5c-tls-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.934667 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:21.934619 4363 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a15aa191-e376-490a-8763-16cdd7685b5c-model-cache\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:22.201107 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:22.201075 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv"] Nov 01 16:18:22.205175 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:22.205149 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv"] Nov 01 16:18:22.625686 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:22.625646 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f5dd93/test-model-f5dd93-kserve-9b467c7d9-6tbdv" podUID="a15aa191-e376-490a-8763-16cdd7685b5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": context deadline exceeded" Nov 01 16:18:22.804668 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:22.804631 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a15aa191-e376-490a-8763-16cdd7685b5c" path="/var/lib/kubelet/pods/a15aa191-e376-490a-8763-16cdd7685b5c/volumes" Nov 01 16:18:28.460042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:28.459205 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" podUID="9cbd41ee-3134-4b54-9ea1-0c13921269bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.181:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:18:32.181084 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:32.180832 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s"] Nov 01 16:18:45.005514 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:45.005465 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6"] Nov 01 16:18:46.823735 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:46.823705 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88"] Nov 01 16:18:46.824512 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:46.824475 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" podUID="ef9cc82b-caf7-4108-b80b-d10135f512b1" containerName="istio-proxy" containerID="cri-o://c82a018a2f289f1106c4d3a0ef4e70cdd256a70d2b376e251eda93cb845ea2aa" gracePeriod=30 Nov 01 16:18:47.246089 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:47.242476 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" podUID="d92f7196-153d-46b0-9e5b-51667329cd02" containerName="main" containerID="cri-o://2b7bc55b89ac0c3303ba9b335155b86b7354d2adacb94b078a951619c8e9bf46" gracePeriod=45 Nov 01 16:18:47.388304 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:47.388268 4363 generic.go:356] "Generic (PLEG): container finished" podID="d92f7196-153d-46b0-9e5b-51667329cd02" containerID="2b7bc55b89ac0c3303ba9b335155b86b7354d2adacb94b078a951619c8e9bf46" exitCode=0 Nov 01 16:18:47.388488 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:47.388341 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" event={"ID":"d92f7196-153d-46b0-9e5b-51667329cd02","Type":"ContainerDied","Data":"2b7bc55b89ac0c3303ba9b335155b86b7354d2adacb94b078a951619c8e9bf46"} Nov 01 16:18:47.688927 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:47.688897 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" Nov 01 16:18:47.811222 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:47.811190 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d92f7196-153d-46b0-9e5b-51667329cd02-kube-api-access-h2jwx\" (UniqueName: \"kubernetes.io/projected/d92f7196-153d-46b0-9e5b-51667329cd02-kube-api-access-h2jwx\") pod \"d92f7196-153d-46b0-9e5b-51667329cd02\" (UID: \"d92f7196-153d-46b0-9e5b-51667329cd02\") " Nov 01 16:18:47.811462 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:47.811249 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d92f7196-153d-46b0-9e5b-51667329cd02-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d92f7196-153d-46b0-9e5b-51667329cd02-tmp-dir\") pod \"d92f7196-153d-46b0-9e5b-51667329cd02\" (UID: \"d92f7196-153d-46b0-9e5b-51667329cd02\") " Nov 01 16:18:47.811462 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:47.811279 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d92f7196-153d-46b0-9e5b-51667329cd02-home\" (UniqueName: \"kubernetes.io/empty-dir/d92f7196-153d-46b0-9e5b-51667329cd02-home\") pod \"d92f7196-153d-46b0-9e5b-51667329cd02\" (UID: \"d92f7196-153d-46b0-9e5b-51667329cd02\") " Nov 01 16:18:47.811462 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:47.811323 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d92f7196-153d-46b0-9e5b-51667329cd02-tls-certs\" (UniqueName: \"kubernetes.io/secret/d92f7196-153d-46b0-9e5b-51667329cd02-tls-certs\") pod \"d92f7196-153d-46b0-9e5b-51667329cd02\" (UID: \"d92f7196-153d-46b0-9e5b-51667329cd02\") " Nov 01 16:18:47.811462 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:47.811449 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d92f7196-153d-46b0-9e5b-51667329cd02-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d92f7196-153d-46b0-9e5b-51667329cd02-dshm\") pod \"d92f7196-153d-46b0-9e5b-51667329cd02\" (UID: \"d92f7196-153d-46b0-9e5b-51667329cd02\") " Nov 01 16:18:47.811718 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:47.811518 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d92f7196-153d-46b0-9e5b-51667329cd02-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d92f7196-153d-46b0-9e5b-51667329cd02-model-cache\") pod \"d92f7196-153d-46b0-9e5b-51667329cd02\" (UID: \"d92f7196-153d-46b0-9e5b-51667329cd02\") " Nov 01 16:18:47.811718 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:47.811597 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d92f7196-153d-46b0-9e5b-51667329cd02-home" pod "d92f7196-153d-46b0-9e5b-51667329cd02" (UID: "d92f7196-153d-46b0-9e5b-51667329cd02"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:47.811718 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:47.811642 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d92f7196-153d-46b0-9e5b-51667329cd02-tmp-dir" pod "d92f7196-153d-46b0-9e5b-51667329cd02" (UID: "d92f7196-153d-46b0-9e5b-51667329cd02"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:47.811999 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:47.811974 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d92f7196-153d-46b0-9e5b-51667329cd02-model-cache" pod "d92f7196-153d-46b0-9e5b-51667329cd02" (UID: "d92f7196-153d-46b0-9e5b-51667329cd02"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:47.812386 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:47.812364 4363 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d92f7196-153d-46b0-9e5b-51667329cd02-tmp-dir\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:47.812386 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:47.812386 4363 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d92f7196-153d-46b0-9e5b-51667329cd02-home\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:47.812537 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:47.812396 4363 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d92f7196-153d-46b0-9e5b-51667329cd02-model-cache\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:47.813921 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:47.813898 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d92f7196-153d-46b0-9e5b-51667329cd02-dshm" pod "d92f7196-153d-46b0-9e5b-51667329cd02" (UID: "d92f7196-153d-46b0-9e5b-51667329cd02"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:47.814135 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:47.814114 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d92f7196-153d-46b0-9e5b-51667329cd02-tls-certs" pod "d92f7196-153d-46b0-9e5b-51667329cd02" (UID: "d92f7196-153d-46b0-9e5b-51667329cd02"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:18:47.814218 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:47.814113 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d92f7196-153d-46b0-9e5b-51667329cd02-kube-api-access-h2jwx" pod "d92f7196-153d-46b0-9e5b-51667329cd02" (UID: "d92f7196-153d-46b0-9e5b-51667329cd02"). InnerVolumeSpecName "kube-api-access-h2jwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:18:47.915377 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:47.915294 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2jwx\" (UniqueName: \"kubernetes.io/projected/d92f7196-153d-46b0-9e5b-51667329cd02-kube-api-access-h2jwx\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:47.915377 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:47.915329 4363 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d92f7196-153d-46b0-9e5b-51667329cd02-tls-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:47.915377 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:47.915343 4363 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d92f7196-153d-46b0-9e5b-51667329cd02-dshm\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:48.312095 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:18:48.311976 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:18:48.400641 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:48.400608 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" Nov 01 16:18:48.400827 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:48.400646 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s" event={"ID":"d92f7196-153d-46b0-9e5b-51667329cd02","Type":"ContainerDied","Data":"39f38d451b096b879c5d4d8bbd6a6f376f342ca8ce3074ab359bedd958227ab2"} Nov 01 16:18:48.400827 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:48.400710 4363 scope.go:122] "RemoveContainer" containerID="2b7bc55b89ac0c3303ba9b335155b86b7354d2adacb94b078a951619c8e9bf46" Nov 01 16:18:48.511047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:48.510973 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s"] Nov 01 16:18:48.512127 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:48.512094 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-1c65e3/test-model-1c65e3-kserve-57bc967dfc-kjh5s"] Nov 01 16:18:48.806873 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:48.806834 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d92f7196-153d-46b0-9e5b-51667329cd02" path="/var/lib/kubelet/pods/d92f7196-153d-46b0-9e5b-51667329cd02/volumes" Nov 01 16:18:52.307306 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:52.307112 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" Nov 01 16:18:52.386004 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:52.385968 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef9cc82b-caf7-4108-b80b-d10135f512b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef9cc82b-caf7-4108-b80b-d10135f512b1-istio-data\") pod \"ef9cc82b-caf7-4108-b80b-d10135f512b1\" (UID: \"ef9cc82b-caf7-4108-b80b-d10135f512b1\") " Nov 01 16:18:52.386292 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:52.386273 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef9cc82b-caf7-4108-b80b-d10135f512b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef9cc82b-caf7-4108-b80b-d10135f512b1-workload-certs\") pod \"ef9cc82b-caf7-4108-b80b-d10135f512b1\" (UID: \"ef9cc82b-caf7-4108-b80b-d10135f512b1\") " Nov 01 16:18:52.386466 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:52.386451 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef9cc82b-caf7-4108-b80b-d10135f512b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9cc82b-caf7-4108-b80b-d10135f512b1-workload-socket\") pod \"ef9cc82b-caf7-4108-b80b-d10135f512b1\" (UID: \"ef9cc82b-caf7-4108-b80b-d10135f512b1\") " Nov 01 16:18:52.386850 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:52.386832 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef9cc82b-caf7-4108-b80b-d10135f512b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef9cc82b-caf7-4108-b80b-d10135f512b1-istio-podinfo\") pod \"ef9cc82b-caf7-4108-b80b-d10135f512b1\" (UID: \"ef9cc82b-caf7-4108-b80b-d10135f512b1\") " Nov 01 16:18:52.387033 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:52.387003 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef9cc82b-caf7-4108-b80b-d10135f512b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9cc82b-caf7-4108-b80b-d10135f512b1-credential-socket\") pod \"ef9cc82b-caf7-4108-b80b-d10135f512b1\" (UID: \"ef9cc82b-caf7-4108-b80b-d10135f512b1\") " Nov 01 16:18:52.387179 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:52.387162 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef9cc82b-caf7-4108-b80b-d10135f512b1-kube-api-access-rqtdq\" (UniqueName: \"kubernetes.io/projected/ef9cc82b-caf7-4108-b80b-d10135f512b1-kube-api-access-rqtdq\") pod \"ef9cc82b-caf7-4108-b80b-d10135f512b1\" (UID: \"ef9cc82b-caf7-4108-b80b-d10135f512b1\") " Nov 01 16:18:52.387347 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:52.387334 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef9cc82b-caf7-4108-b80b-d10135f512b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef9cc82b-caf7-4108-b80b-d10135f512b1-istiod-ca-cert\") pod \"ef9cc82b-caf7-4108-b80b-d10135f512b1\" (UID: \"ef9cc82b-caf7-4108-b80b-d10135f512b1\") " Nov 01 16:18:52.387469 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:52.387457 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef9cc82b-caf7-4108-b80b-d10135f512b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef9cc82b-caf7-4108-b80b-d10135f512b1-istio-envoy\") pod \"ef9cc82b-caf7-4108-b80b-d10135f512b1\" (UID: \"ef9cc82b-caf7-4108-b80b-d10135f512b1\") " Nov 01 16:18:52.387735 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:52.387720 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef9cc82b-caf7-4108-b80b-d10135f512b1-istio-token\" (UniqueName: \"kubernetes.io/projected/ef9cc82b-caf7-4108-b80b-d10135f512b1-istio-token\") pod \"ef9cc82b-caf7-4108-b80b-d10135f512b1\" (UID: \"ef9cc82b-caf7-4108-b80b-d10135f512b1\") " Nov 01 16:18:52.391109 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:52.390792 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef9cc82b-caf7-4108-b80b-d10135f512b1-istiod-ca-cert" pod "ef9cc82b-caf7-4108-b80b-d10135f512b1" (UID: "ef9cc82b-caf7-4108-b80b-d10135f512b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:18:52.391452 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:52.391414 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef9cc82b-caf7-4108-b80b-d10135f512b1-istio-data" pod "ef9cc82b-caf7-4108-b80b-d10135f512b1" (UID: "ef9cc82b-caf7-4108-b80b-d10135f512b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:52.391720 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:52.391689 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef9cc82b-caf7-4108-b80b-d10135f512b1-workload-socket" pod "ef9cc82b-caf7-4108-b80b-d10135f512b1" (UID: "ef9cc82b-caf7-4108-b80b-d10135f512b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:52.391964 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:52.391937 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef9cc82b-caf7-4108-b80b-d10135f512b1-credential-socket" pod "ef9cc82b-caf7-4108-b80b-d10135f512b1" (UID: "ef9cc82b-caf7-4108-b80b-d10135f512b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:52.392166 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:52.392138 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef9cc82b-caf7-4108-b80b-d10135f512b1-workload-certs" pod "ef9cc82b-caf7-4108-b80b-d10135f512b1" (UID: "ef9cc82b-caf7-4108-b80b-d10135f512b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:52.397435 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:52.397401 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef9cc82b-caf7-4108-b80b-d10135f512b1-kube-api-access-rqtdq" pod "ef9cc82b-caf7-4108-b80b-d10135f512b1" (UID: "ef9cc82b-caf7-4108-b80b-d10135f512b1"). InnerVolumeSpecName "kube-api-access-rqtdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:18:52.410120 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:52.409092 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef9cc82b-caf7-4108-b80b-d10135f512b1-istio-token" pod "ef9cc82b-caf7-4108-b80b-d10135f512b1" (UID: "ef9cc82b-caf7-4108-b80b-d10135f512b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:18:52.411135 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:52.410876 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef9cc82b-caf7-4108-b80b-d10135f512b1-istio-envoy" pod "ef9cc82b-caf7-4108-b80b-d10135f512b1" (UID: "ef9cc82b-caf7-4108-b80b-d10135f512b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:52.411651 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:52.411599 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef9cc82b-caf7-4108-b80b-d10135f512b1-istio-podinfo" pod "ef9cc82b-caf7-4108-b80b-d10135f512b1" (UID: "ef9cc82b-caf7-4108-b80b-d10135f512b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:18:52.490451 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:52.490421 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef9cc82b-caf7-4108-b80b-d10135f512b1-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:52.490451 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:52.490448 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9cc82b-caf7-4108-b80b-d10135f512b1-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:52.490638 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:52.490459 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqtdq\" (UniqueName: \"kubernetes.io/projected/ef9cc82b-caf7-4108-b80b-d10135f512b1-kube-api-access-rqtdq\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:52.490638 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:52.490469 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef9cc82b-caf7-4108-b80b-d10135f512b1-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:52.490638 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:52.490479 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef9cc82b-caf7-4108-b80b-d10135f512b1-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:52.490638 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:52.490487 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef9cc82b-caf7-4108-b80b-d10135f512b1-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:52.490638 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:52.490504 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef9cc82b-caf7-4108-b80b-d10135f512b1-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:52.490638 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:52.490513 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef9cc82b-caf7-4108-b80b-d10135f512b1-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:52.490638 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:52.490521 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9cc82b-caf7-4108-b80b-d10135f512b1-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:52.568953 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:52.568872 4363 generic.go:356] "Generic (PLEG): container finished" podID="ef9cc82b-caf7-4108-b80b-d10135f512b1" containerID="c82a018a2f289f1106c4d3a0ef4e70cdd256a70d2b376e251eda93cb845ea2aa" exitCode=0 Nov 01 16:18:52.568953 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:52.568918 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" Nov 01 16:18:52.569162 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:52.568956 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" event={"ID":"ef9cc82b-caf7-4108-b80b-d10135f512b1","Type":"ContainerDied","Data":"c82a018a2f289f1106c4d3a0ef4e70cdd256a70d2b376e251eda93cb845ea2aa"} Nov 01 16:18:52.569162 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:52.569026 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88" event={"ID":"ef9cc82b-caf7-4108-b80b-d10135f512b1","Type":"ContainerDied","Data":"5b0be85fbec09466d052d8b9174e5500c42e20ce1d9e88005970130848683131"} Nov 01 16:18:52.569162 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:52.569050 4363 scope.go:122] "RemoveContainer" containerID="c82a018a2f289f1106c4d3a0ef4e70cdd256a70d2b376e251eda93cb845ea2aa" Nov 01 16:18:52.588368 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:52.588347 4363 scope.go:122] "RemoveContainer" containerID="a2fdb4ea49a46cd17108656febd6573d9ebf7f3d7d40960311021f074ae2fca6" Nov 01 16:18:52.607534 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:52.607482 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88"] Nov 01 16:18:52.610260 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:52.610130 4363 scope.go:122] "RemoveContainer" containerID="c82a018a2f289f1106c4d3a0ef4e70cdd256a70d2b376e251eda93cb845ea2aa" Nov 01 16:18:52.612399 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:18:52.612362 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c82a018a2f289f1106c4d3a0ef4e70cdd256a70d2b376e251eda93cb845ea2aa\": container with ID starting with c82a018a2f289f1106c4d3a0ef4e70cdd256a70d2b376e251eda93cb845ea2aa not found: ID does not exist" containerID="c82a018a2f289f1106c4d3a0ef4e70cdd256a70d2b376e251eda93cb845ea2aa" Nov 01 16:18:52.612519 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:52.612398 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c82a018a2f289f1106c4d3a0ef4e70cdd256a70d2b376e251eda93cb845ea2aa"} err="failed to get container status \"c82a018a2f289f1106c4d3a0ef4e70cdd256a70d2b376e251eda93cb845ea2aa\": rpc error: code = NotFound desc = could not find container \"c82a018a2f289f1106c4d3a0ef4e70cdd256a70d2b376e251eda93cb845ea2aa\": container with ID starting with c82a018a2f289f1106c4d3a0ef4e70cdd256a70d2b376e251eda93cb845ea2aa not found: ID does not exist" Nov 01 16:18:52.612519 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:52.612423 4363 scope.go:122] "RemoveContainer" containerID="a2fdb4ea49a46cd17108656febd6573d9ebf7f3d7d40960311021f074ae2fca6" Nov 01 16:18:52.612798 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:18:52.612769 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2fdb4ea49a46cd17108656febd6573d9ebf7f3d7d40960311021f074ae2fca6\": container with ID starting with a2fdb4ea49a46cd17108656febd6573d9ebf7f3d7d40960311021f074ae2fca6 not found: ID does not exist" containerID="a2fdb4ea49a46cd17108656febd6573d9ebf7f3d7d40960311021f074ae2fca6" Nov 01 16:18:52.612883 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:52.612807 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2fdb4ea49a46cd17108656febd6573d9ebf7f3d7d40960311021f074ae2fca6"} err="failed to get container status \"a2fdb4ea49a46cd17108656febd6573d9ebf7f3d7d40960311021f074ae2fca6\": rpc error: code = NotFound desc = could not find container \"a2fdb4ea49a46cd17108656febd6573d9ebf7f3d7d40960311021f074ae2fca6\": container with ID starting with a2fdb4ea49a46cd17108656febd6573d9ebf7f3d7d40960311021f074ae2fca6 not found: ID does not exist" Nov 01 16:18:52.616284 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:52.616260 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-jrh88"] Nov 01 16:18:52.805572 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:52.805538 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef9cc82b-caf7-4108-b80b-d10135f512b1" path="/var/lib/kubelet/pods/ef9cc82b-caf7-4108-b80b-d10135f512b1/volumes" Nov 01 16:18:55.566221 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:55.566186 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1c65e3-54b4ddd7d7-lbrcc"] Nov 01 16:18:55.580719 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:55.580687 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1c65e3-77686b98cf-wzm9h"] Nov 01 16:18:57.479456 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:57.479411 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-375350-b98cbd657-76rtq"] Nov 01 16:18:57.492858 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:18:57.492828 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-375350-68b888b95-4mx6d"] Nov 01 16:19:00.038989 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:00.038954 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" podUID="37773d39-9d52-4b1d-94c7-68e02714a476" containerName="main" containerID="cri-o://b92daf01c2ff7518c17a6c102771978ffb6ef1ef7f54c62ac98853a7390fc354" gracePeriod=45 Nov 01 16:19:00.067847 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:00.067817 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" podUID="37773d39-9d52-4b1d-94c7-68e02714a476" containerName="main" probeResult="failure" output="Get \"https://10.134.0.171:8000/ready\": dial tcp 10.134.0.171:8000: connect: connection refused" Nov 01 16:19:00.452160 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:00.452128 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" Nov 01 16:19:00.534937 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:00.534897 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37773d39-9d52-4b1d-94c7-68e02714a476-dshm\" (UniqueName: \"kubernetes.io/empty-dir/37773d39-9d52-4b1d-94c7-68e02714a476-dshm\") pod \"37773d39-9d52-4b1d-94c7-68e02714a476\" (UID: \"37773d39-9d52-4b1d-94c7-68e02714a476\") " Nov 01 16:19:00.535144 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:00.535115 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37773d39-9d52-4b1d-94c7-68e02714a476-kube-api-access-2cwxh\" (UniqueName: \"kubernetes.io/projected/37773d39-9d52-4b1d-94c7-68e02714a476-kube-api-access-2cwxh\") pod \"37773d39-9d52-4b1d-94c7-68e02714a476\" (UID: \"37773d39-9d52-4b1d-94c7-68e02714a476\") " Nov 01 16:19:00.535216 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:00.535163 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37773d39-9d52-4b1d-94c7-68e02714a476-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37773d39-9d52-4b1d-94c7-68e02714a476-model-cache\") pod \"37773d39-9d52-4b1d-94c7-68e02714a476\" (UID: \"37773d39-9d52-4b1d-94c7-68e02714a476\") " Nov 01 16:19:00.535280 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:00.535249 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37773d39-9d52-4b1d-94c7-68e02714a476-tls-certs\" (UniqueName: \"kubernetes.io/secret/37773d39-9d52-4b1d-94c7-68e02714a476-tls-certs\") pod \"37773d39-9d52-4b1d-94c7-68e02714a476\" (UID: \"37773d39-9d52-4b1d-94c7-68e02714a476\") " Nov 01 16:19:00.535365 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:00.535347 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37773d39-9d52-4b1d-94c7-68e02714a476-home\" (UniqueName: \"kubernetes.io/empty-dir/37773d39-9d52-4b1d-94c7-68e02714a476-home\") pod \"37773d39-9d52-4b1d-94c7-68e02714a476\" (UID: \"37773d39-9d52-4b1d-94c7-68e02714a476\") " Nov 01 16:19:00.535575 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:00.535555 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37773d39-9d52-4b1d-94c7-68e02714a476-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37773d39-9d52-4b1d-94c7-68e02714a476-tmp-dir\") pod \"37773d39-9d52-4b1d-94c7-68e02714a476\" (UID: \"37773d39-9d52-4b1d-94c7-68e02714a476\") " Nov 01 16:19:00.535898 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:00.535566 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37773d39-9d52-4b1d-94c7-68e02714a476-model-cache" pod "37773d39-9d52-4b1d-94c7-68e02714a476" (UID: "37773d39-9d52-4b1d-94c7-68e02714a476"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:00.536711 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:00.536680 4363 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37773d39-9d52-4b1d-94c7-68e02714a476-model-cache\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:00.537511 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:00.537484 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37773d39-9d52-4b1d-94c7-68e02714a476-tmp-dir" pod "37773d39-9d52-4b1d-94c7-68e02714a476" (UID: "37773d39-9d52-4b1d-94c7-68e02714a476"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:00.537704 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:00.537683 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37773d39-9d52-4b1d-94c7-68e02714a476-home" pod "37773d39-9d52-4b1d-94c7-68e02714a476" (UID: "37773d39-9d52-4b1d-94c7-68e02714a476"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:00.537841 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:00.537816 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37773d39-9d52-4b1d-94c7-68e02714a476-kube-api-access-2cwxh" pod "37773d39-9d52-4b1d-94c7-68e02714a476" (UID: "37773d39-9d52-4b1d-94c7-68e02714a476"). InnerVolumeSpecName "kube-api-access-2cwxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:00.537918 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:00.537899 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37773d39-9d52-4b1d-94c7-68e02714a476-tls-certs" pod "37773d39-9d52-4b1d-94c7-68e02714a476" (UID: "37773d39-9d52-4b1d-94c7-68e02714a476"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:19:00.539181 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:00.539154 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37773d39-9d52-4b1d-94c7-68e02714a476-dshm" pod "37773d39-9d52-4b1d-94c7-68e02714a476" (UID: "37773d39-9d52-4b1d-94c7-68e02714a476"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:00.567351 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:00.567321 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-1c65e3-54b4ddd7d7-lbrcc" podUID="85d2d15c-3c9a-4b88-89cb-5f5c06979aa9" containerName="payload-processing" containerID="cri-o://ebca0458ff258072c27d4077a4ca73acb1ad6360cd26932e70e502456fd38405" gracePeriod=25 Nov 01 16:19:00.581847 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:00.581814 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c65e3-77686b98cf-wzm9h" podUID="06fcb8d1-bff1-4182-92d2-98a57314f6fd" containerName="payload-pre-processing" containerID="cri-o://e2f2704741032bd934dedbe00593467488a3e6e04e845624b1d3ee7d22b461f3" gracePeriod=25 Nov 01 16:19:00.638896 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:00.638863 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cwxh\" (UniqueName: \"kubernetes.io/projected/37773d39-9d52-4b1d-94c7-68e02714a476-kube-api-access-2cwxh\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:00.638896 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:00.638890 4363 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37773d39-9d52-4b1d-94c7-68e02714a476-tls-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:00.638896 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:00.638899 4363 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37773d39-9d52-4b1d-94c7-68e02714a476-home\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:00.639110 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:00.638908 4363 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37773d39-9d52-4b1d-94c7-68e02714a476-tmp-dir\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:00.639110 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:00.638916 4363 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37773d39-9d52-4b1d-94c7-68e02714a476-dshm\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:00.743233 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:00.743192 4363 generic.go:356] "Generic (PLEG): container finished" podID="37773d39-9d52-4b1d-94c7-68e02714a476" containerID="b92daf01c2ff7518c17a6c102771978ffb6ef1ef7f54c62ac98853a7390fc354" exitCode=0 Nov 01 16:19:00.743434 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:00.743300 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" event={"ID":"37773d39-9d52-4b1d-94c7-68e02714a476","Type":"ContainerDied","Data":"b92daf01c2ff7518c17a6c102771978ffb6ef1ef7f54c62ac98853a7390fc354"} Nov 01 16:19:00.743434 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:00.743338 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" event={"ID":"37773d39-9d52-4b1d-94c7-68e02714a476","Type":"ContainerDied","Data":"1e2eadc95d28c75dc5d3479e5b899551a3f4638d73cd21abed6f73edb0862892"} Nov 01 16:19:00.743434 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:00.743360 4363 scope.go:122] "RemoveContainer" containerID="b92daf01c2ff7518c17a6c102771978ffb6ef1ef7f54c62ac98853a7390fc354" Nov 01 16:19:00.743624 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:00.743524 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6" Nov 01 16:19:00.746384 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:00.746354 4363 generic.go:356] "Generic (PLEG): container finished" podID="85d2d15c-3c9a-4b88-89cb-5f5c06979aa9" containerID="ebca0458ff258072c27d4077a4ca73acb1ad6360cd26932e70e502456fd38405" exitCode=0 Nov 01 16:19:00.746593 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:00.746573 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1c65e3-54b4ddd7d7-lbrcc" event={"ID":"85d2d15c-3c9a-4b88-89cb-5f5c06979aa9","Type":"ContainerDied","Data":"ebca0458ff258072c27d4077a4ca73acb1ad6360cd26932e70e502456fd38405"} Nov 01 16:19:00.760544 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:00.760528 4363 scope.go:122] "RemoveContainer" containerID="b92daf01c2ff7518c17a6c102771978ffb6ef1ef7f54c62ac98853a7390fc354" Nov 01 16:19:00.761053 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:19:00.760950 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b92daf01c2ff7518c17a6c102771978ffb6ef1ef7f54c62ac98853a7390fc354\": container with ID starting with b92daf01c2ff7518c17a6c102771978ffb6ef1ef7f54c62ac98853a7390fc354 not found: ID does not exist" containerID="b92daf01c2ff7518c17a6c102771978ffb6ef1ef7f54c62ac98853a7390fc354" Nov 01 16:19:00.761053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:00.760987 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b92daf01c2ff7518c17a6c102771978ffb6ef1ef7f54c62ac98853a7390fc354"} err="failed to get container status \"b92daf01c2ff7518c17a6c102771978ffb6ef1ef7f54c62ac98853a7390fc354\": rpc error: code = NotFound desc = could not find container \"b92daf01c2ff7518c17a6c102771978ffb6ef1ef7f54c62ac98853a7390fc354\": container with ID starting with b92daf01c2ff7518c17a6c102771978ffb6ef1ef7f54c62ac98853a7390fc354 not found: ID does not exist" Nov 01 16:19:00.790707 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:00.788711 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6"] Nov 01 16:19:00.812412 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:00.807890 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-2ce08a/test-model-2ce08a-kserve-64c8c486fc-vlgr6"] Nov 01 16:19:00.973976 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:00.973450 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4"] Nov 01 16:19:00.973976 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:00.973533 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv"] Nov 01 16:19:00.973976 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:00.973733 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" podUID="05ec5955-5241-423b-8161-1ac7608f4fed" containerName="istio-proxy" containerID="cri-o://c3f9604d2929249e6d9581fe956ae902f56c348867bcbc7a8dc1422d707df3b4" gracePeriod=30 Nov 01 16:19:00.995623 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:00.985964 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" podUID="ef2387a8-1f76-4653-9047-0f5824e6cfbb" containerName="istio-proxy" containerID="cri-o://5b3486d4b53f8d69654824e4e3d3aa10bf2fbba2dab07c4b14a96e26c66ef5a0" gracePeriod=30 Nov 01 16:19:01.030742 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:01.030695 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8"] Nov 01 16:19:01.031231 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:01.031199 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" podUID="69f7030b-22ab-4f75-8705-af57a56c29f5" containerName="istio-proxy" containerID="cri-o://2936c71b4370d557c4253153a44fdc51748064e8a380ddce3c305747cba26b50" gracePeriod=30 Nov 01 16:19:01.036747 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:01.036645 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1c65e3-54b4ddd7d7-lbrcc" Nov 01 16:19:01.203063 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:01.203031 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85d2d15c-3c9a-4b88-89cb-5f5c06979aa9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85d2d15c-3c9a-4b88-89cb-5f5c06979aa9-plugins-config-volume\") pod \"85d2d15c-3c9a-4b88-89cb-5f5c06979aa9\" (UID: \"85d2d15c-3c9a-4b88-89cb-5f5c06979aa9\") " Nov 01 16:19:01.203676 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:01.203170 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85d2d15c-3c9a-4b88-89cb-5f5c06979aa9-kube-api-access-7kgn2\" (UniqueName: \"kubernetes.io/projected/85d2d15c-3c9a-4b88-89cb-5f5c06979aa9-kube-api-access-7kgn2\") pod \"85d2d15c-3c9a-4b88-89cb-5f5c06979aa9\" (UID: \"85d2d15c-3c9a-4b88-89cb-5f5c06979aa9\") " Nov 01 16:19:01.203676 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:01.203483 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85d2d15c-3c9a-4b88-89cb-5f5c06979aa9-plugins-config-volume" pod "85d2d15c-3c9a-4b88-89cb-5f5c06979aa9" (UID: "85d2d15c-3c9a-4b88-89cb-5f5c06979aa9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:19:01.204298 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:01.204278 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85d2d15c-3c9a-4b88-89cb-5f5c06979aa9-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:01.206072 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:01.206047 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85d2d15c-3c9a-4b88-89cb-5f5c06979aa9-kube-api-access-7kgn2" pod "85d2d15c-3c9a-4b88-89cb-5f5c06979aa9" (UID: "85d2d15c-3c9a-4b88-89cb-5f5c06979aa9"). InnerVolumeSpecName "kube-api-access-7kgn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:01.306248 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:01.306208 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kgn2\" (UniqueName: \"kubernetes.io/projected/85d2d15c-3c9a-4b88-89cb-5f5c06979aa9-kube-api-access-7kgn2\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:01.760486 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:01.760448 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1c65e3-54b4ddd7d7-lbrcc" event={"ID":"85d2d15c-3c9a-4b88-89cb-5f5c06979aa9","Type":"ContainerDied","Data":"271185efbb8a67a17bedc2623f53be76ada913267b58c0fdb746c7221d5ffe02"} Nov 01 16:19:01.760697 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:01.760502 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1c65e3-54b4ddd7d7-lbrcc" Nov 01 16:19:01.760697 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:01.760516 4363 scope.go:122] "RemoveContainer" containerID="ebca0458ff258072c27d4077a4ca73acb1ad6360cd26932e70e502456fd38405" Nov 01 16:19:01.793567 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:01.793535 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1c65e3-54b4ddd7d7-lbrcc"] Nov 01 16:19:01.800500 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:01.800466 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1c65e3-54b4ddd7d7-lbrcc"] Nov 01 16:19:01.908174 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:01.896922 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb"] Nov 01 16:19:02.096311 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:02.096264 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:19:02.096628 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:02.096583 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" podUID="69f7030b-22ab-4f75-8705-af57a56c29f5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:19:02.480392 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:02.480285 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-375350-b98cbd657-76rtq" podUID="7d0c63a3-8c1f-4c54-9373-afed7b61795c" containerName="payload-processing" containerID="cri-o://bc94780c2d6af818ef77a848f09edb05f03b31df70a630ae9e05d5b59175d469" gracePeriod=25 Nov 01 16:19:02.493380 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:02.493336 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-375350-68b888b95-4mx6d" podUID="eb084141-b900-462d-ac93-b1ce933d6f2e" containerName="payload-pre-processing" containerID="cri-o://6df9f030f44edd6d80b248963bafe3935cb735feb6ac14a5b00411804b593283" gracePeriod=25 Nov 01 16:19:02.801258 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:02.801171 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37773d39-9d52-4b1d-94c7-68e02714a476" path="/var/lib/kubelet/pods/37773d39-9d52-4b1d-94c7-68e02714a476/volumes" Nov 01 16:19:02.801792 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:02.801768 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85d2d15c-3c9a-4b88-89cb-5f5c06979aa9" path="/var/lib/kubelet/pods/85d2d15c-3c9a-4b88-89cb-5f5c06979aa9/volumes" Nov 01 16:19:06.335071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.335041 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" Nov 01 16:19:06.415034 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.413382 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05ec5955-5241-423b-8161-1ac7608f4fed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ec5955-5241-423b-8161-1ac7608f4fed-workload-socket\") pod \"05ec5955-5241-423b-8161-1ac7608f4fed\" (UID: \"05ec5955-5241-423b-8161-1ac7608f4fed\") " Nov 01 16:19:06.415034 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.413471 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05ec5955-5241-423b-8161-1ac7608f4fed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05ec5955-5241-423b-8161-1ac7608f4fed-istiod-ca-cert\") pod \"05ec5955-5241-423b-8161-1ac7608f4fed\" (UID: \"05ec5955-5241-423b-8161-1ac7608f4fed\") " Nov 01 16:19:06.415034 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.413598 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05ec5955-5241-423b-8161-1ac7608f4fed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ec5955-5241-423b-8161-1ac7608f4fed-credential-socket\") pod \"05ec5955-5241-423b-8161-1ac7608f4fed\" (UID: \"05ec5955-5241-423b-8161-1ac7608f4fed\") " Nov 01 16:19:06.415034 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.413634 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05ec5955-5241-423b-8161-1ac7608f4fed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05ec5955-5241-423b-8161-1ac7608f4fed-istio-data\") pod \"05ec5955-5241-423b-8161-1ac7608f4fed\" (UID: \"05ec5955-5241-423b-8161-1ac7608f4fed\") " Nov 01 16:19:06.415034 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.413763 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05ec5955-5241-423b-8161-1ac7608f4fed-kube-api-access-l525n\" (UniqueName: \"kubernetes.io/projected/05ec5955-5241-423b-8161-1ac7608f4fed-kube-api-access-l525n\") pod \"05ec5955-5241-423b-8161-1ac7608f4fed\" (UID: \"05ec5955-5241-423b-8161-1ac7608f4fed\") " Nov 01 16:19:06.415034 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.413833 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05ec5955-5241-423b-8161-1ac7608f4fed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05ec5955-5241-423b-8161-1ac7608f4fed-istio-podinfo\") pod \"05ec5955-5241-423b-8161-1ac7608f4fed\" (UID: \"05ec5955-5241-423b-8161-1ac7608f4fed\") " Nov 01 16:19:06.415034 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.414090 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05ec5955-5241-423b-8161-1ac7608f4fed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05ec5955-5241-423b-8161-1ac7608f4fed-istio-envoy\") pod \"05ec5955-5241-423b-8161-1ac7608f4fed\" (UID: \"05ec5955-5241-423b-8161-1ac7608f4fed\") " Nov 01 16:19:06.415034 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.414147 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05ec5955-5241-423b-8161-1ac7608f4fed-istio-token\" (UniqueName: \"kubernetes.io/projected/05ec5955-5241-423b-8161-1ac7608f4fed-istio-token\") pod \"05ec5955-5241-423b-8161-1ac7608f4fed\" (UID: \"05ec5955-5241-423b-8161-1ac7608f4fed\") " Nov 01 16:19:06.415034 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.414224 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05ec5955-5241-423b-8161-1ac7608f4fed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05ec5955-5241-423b-8161-1ac7608f4fed-workload-certs\") pod \"05ec5955-5241-423b-8161-1ac7608f4fed\" (UID: \"05ec5955-5241-423b-8161-1ac7608f4fed\") " Nov 01 16:19:06.415799 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.415531 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ec5955-5241-423b-8161-1ac7608f4fed-workload-certs" pod "05ec5955-5241-423b-8161-1ac7608f4fed" (UID: "05ec5955-5241-423b-8161-1ac7608f4fed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:06.415799 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.415738 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ec5955-5241-423b-8161-1ac7608f4fed-workload-socket" pod "05ec5955-5241-423b-8161-1ac7608f4fed" (UID: "05ec5955-5241-423b-8161-1ac7608f4fed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:06.424135 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.416036 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05ec5955-5241-423b-8161-1ac7608f4fed-istiod-ca-cert" pod "05ec5955-5241-423b-8161-1ac7608f4fed" (UID: "05ec5955-5241-423b-8161-1ac7608f4fed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:19:06.424135 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.416255 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ec5955-5241-423b-8161-1ac7608f4fed-credential-socket" pod "05ec5955-5241-423b-8161-1ac7608f4fed" (UID: "05ec5955-5241-423b-8161-1ac7608f4fed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:06.424135 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.416400 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ec5955-5241-423b-8161-1ac7608f4fed-istio-data" pod "05ec5955-5241-423b-8161-1ac7608f4fed" (UID: "05ec5955-5241-423b-8161-1ac7608f4fed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:06.431072 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.428488 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05ec5955-5241-423b-8161-1ac7608f4fed-istio-podinfo" pod "05ec5955-5241-423b-8161-1ac7608f4fed" (UID: "05ec5955-5241-423b-8161-1ac7608f4fed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:19:06.431072 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.429185 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05ec5955-5241-423b-8161-1ac7608f4fed-istio-token" pod "05ec5955-5241-423b-8161-1ac7608f4fed" (UID: "05ec5955-5241-423b-8161-1ac7608f4fed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:06.431072 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.429936 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ec5955-5241-423b-8161-1ac7608f4fed-istio-envoy" pod "05ec5955-5241-423b-8161-1ac7608f4fed" (UID: "05ec5955-5241-423b-8161-1ac7608f4fed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:06.457642 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.457608 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05ec5955-5241-423b-8161-1ac7608f4fed-kube-api-access-l525n" pod "05ec5955-5241-423b-8161-1ac7608f4fed" (UID: "05ec5955-5241-423b-8161-1ac7608f4fed"). InnerVolumeSpecName "kube-api-access-l525n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:06.467877 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.467782 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" Nov 01 16:19:06.478438 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.478359 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" Nov 01 16:19:06.518483 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.518447 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05ec5955-5241-423b-8161-1ac7608f4fed-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:06.518483 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.518482 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05ec5955-5241-423b-8161-1ac7608f4fed-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:06.518752 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.518501 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05ec5955-5241-423b-8161-1ac7608f4fed-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:06.518752 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.518515 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ec5955-5241-423b-8161-1ac7608f4fed-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:06.518752 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.518529 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05ec5955-5241-423b-8161-1ac7608f4fed-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:06.518752 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.518542 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ec5955-5241-423b-8161-1ac7608f4fed-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:06.518752 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.518556 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05ec5955-5241-423b-8161-1ac7608f4fed-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:06.518752 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.518570 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l525n\" (UniqueName: \"kubernetes.io/projected/05ec5955-5241-423b-8161-1ac7608f4fed-kube-api-access-l525n\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:06.518752 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.518585 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05ec5955-5241-423b-8161-1ac7608f4fed-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:06.560553 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.560468 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c65e3-77686b98cf-wzm9h" podUID="06fcb8d1-bff1-4182-92d2-98a57314f6fd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:19:06.620251 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.620217 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69f7030b-22ab-4f75-8705-af57a56c29f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69f7030b-22ab-4f75-8705-af57a56c29f5-istio-data\") pod \"69f7030b-22ab-4f75-8705-af57a56c29f5\" (UID: \"69f7030b-22ab-4f75-8705-af57a56c29f5\") " Nov 01 16:19:06.620451 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.620292 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef2387a8-1f76-4653-9047-0f5824e6cfbb-kube-api-access-z2l2k\" (UniqueName: \"kubernetes.io/projected/ef2387a8-1f76-4653-9047-0f5824e6cfbb-kube-api-access-z2l2k\") pod \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\" (UID: \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\") " Nov 01 16:19:06.620451 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.620344 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef2387a8-1f76-4653-9047-0f5824e6cfbb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef2387a8-1f76-4653-9047-0f5824e6cfbb-istio-podinfo\") pod \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\" (UID: \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\") " Nov 01 16:19:06.620451 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.620448 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69f7030b-22ab-4f75-8705-af57a56c29f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69f7030b-22ab-4f75-8705-af57a56c29f5-istiod-ca-cert\") pod \"69f7030b-22ab-4f75-8705-af57a56c29f5\" (UID: \"69f7030b-22ab-4f75-8705-af57a56c29f5\") " Nov 01 16:19:06.620628 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.620550 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef2387a8-1f76-4653-9047-0f5824e6cfbb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef2387a8-1f76-4653-9047-0f5824e6cfbb-workload-certs\") pod \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\" (UID: \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\") " Nov 01 16:19:06.620628 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.620547 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69f7030b-22ab-4f75-8705-af57a56c29f5-istio-data" pod "69f7030b-22ab-4f75-8705-af57a56c29f5" (UID: "69f7030b-22ab-4f75-8705-af57a56c29f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:06.620736 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.620663 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69f7030b-22ab-4f75-8705-af57a56c29f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69f7030b-22ab-4f75-8705-af57a56c29f5-workload-certs\") pod \"69f7030b-22ab-4f75-8705-af57a56c29f5\" (UID: \"69f7030b-22ab-4f75-8705-af57a56c29f5\") " Nov 01 16:19:06.620736 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.620690 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69f7030b-22ab-4f75-8705-af57a56c29f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69f7030b-22ab-4f75-8705-af57a56c29f5-credential-socket\") pod \"69f7030b-22ab-4f75-8705-af57a56c29f5\" (UID: \"69f7030b-22ab-4f75-8705-af57a56c29f5\") " Nov 01 16:19:06.620736 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.620721 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69f7030b-22ab-4f75-8705-af57a56c29f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69f7030b-22ab-4f75-8705-af57a56c29f5-workload-socket\") pod \"69f7030b-22ab-4f75-8705-af57a56c29f5\" (UID: \"69f7030b-22ab-4f75-8705-af57a56c29f5\") " Nov 01 16:19:06.620946 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.620760 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef2387a8-1f76-4653-9047-0f5824e6cfbb-istio-token\" (UniqueName: \"kubernetes.io/projected/ef2387a8-1f76-4653-9047-0f5824e6cfbb-istio-token\") pod \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\" (UID: \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\") " Nov 01 16:19:06.620946 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.620843 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef2387a8-1f76-4653-9047-0f5824e6cfbb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2387a8-1f76-4653-9047-0f5824e6cfbb-workload-socket\") pod \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\" (UID: \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\") " Nov 01 16:19:06.620946 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.620885 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69f7030b-22ab-4f75-8705-af57a56c29f5-istio-token\" (UniqueName: \"kubernetes.io/projected/69f7030b-22ab-4f75-8705-af57a56c29f5-istio-token\") pod \"69f7030b-22ab-4f75-8705-af57a56c29f5\" (UID: \"69f7030b-22ab-4f75-8705-af57a56c29f5\") " Nov 01 16:19:06.621120 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.620946 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef2387a8-1f76-4653-9047-0f5824e6cfbb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef2387a8-1f76-4653-9047-0f5824e6cfbb-istio-envoy\") pod \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\" (UID: \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\") " Nov 01 16:19:06.621749 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.621718 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef2387a8-1f76-4653-9047-0f5824e6cfbb-workload-socket" pod "ef2387a8-1f76-4653-9047-0f5824e6cfbb" (UID: "ef2387a8-1f76-4653-9047-0f5824e6cfbb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:06.621862 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.621789 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69f7030b-22ab-4f75-8705-af57a56c29f5-istiod-ca-cert" pod "69f7030b-22ab-4f75-8705-af57a56c29f5" (UID: "69f7030b-22ab-4f75-8705-af57a56c29f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:19:06.621862 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.621842 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef2387a8-1f76-4653-9047-0f5824e6cfbb-workload-certs" pod "ef2387a8-1f76-4653-9047-0f5824e6cfbb" (UID: "ef2387a8-1f76-4653-9047-0f5824e6cfbb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:06.621974 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.621954 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69f7030b-22ab-4f75-8705-af57a56c29f5-workload-socket" pod "69f7030b-22ab-4f75-8705-af57a56c29f5" (UID: "69f7030b-22ab-4f75-8705-af57a56c29f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:06.622182 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.622151 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69f7030b-22ab-4f75-8705-af57a56c29f5-workload-certs" pod "69f7030b-22ab-4f75-8705-af57a56c29f5" (UID: "69f7030b-22ab-4f75-8705-af57a56c29f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:06.622494 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.622476 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69f7030b-22ab-4f75-8705-af57a56c29f5-credential-socket" pod "69f7030b-22ab-4f75-8705-af57a56c29f5" (UID: "69f7030b-22ab-4f75-8705-af57a56c29f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:06.622689 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.622665 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69f7030b-22ab-4f75-8705-af57a56c29f5-kube-api-access-q9xdq\" (UniqueName: \"kubernetes.io/projected/69f7030b-22ab-4f75-8705-af57a56c29f5-kube-api-access-q9xdq\") pod \"69f7030b-22ab-4f75-8705-af57a56c29f5\" (UID: \"69f7030b-22ab-4f75-8705-af57a56c29f5\") " Nov 01 16:19:06.622812 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.622717 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef2387a8-1f76-4653-9047-0f5824e6cfbb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2387a8-1f76-4653-9047-0f5824e6cfbb-credential-socket\") pod \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\" (UID: \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\") " Nov 01 16:19:06.622812 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.622758 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69f7030b-22ab-4f75-8705-af57a56c29f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69f7030b-22ab-4f75-8705-af57a56c29f5-istio-envoy\") pod \"69f7030b-22ab-4f75-8705-af57a56c29f5\" (UID: \"69f7030b-22ab-4f75-8705-af57a56c29f5\") " Nov 01 16:19:06.622812 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.622804 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69f7030b-22ab-4f75-8705-af57a56c29f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69f7030b-22ab-4f75-8705-af57a56c29f5-istio-podinfo\") pod \"69f7030b-22ab-4f75-8705-af57a56c29f5\" (UID: \"69f7030b-22ab-4f75-8705-af57a56c29f5\") " Nov 01 16:19:06.623010 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.622852 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef2387a8-1f76-4653-9047-0f5824e6cfbb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef2387a8-1f76-4653-9047-0f5824e6cfbb-istiod-ca-cert\") pod \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\" (UID: \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\") " Nov 01 16:19:06.623010 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.622952 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef2387a8-1f76-4653-9047-0f5824e6cfbb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef2387a8-1f76-4653-9047-0f5824e6cfbb-istio-data\") pod \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\" (UID: \"ef2387a8-1f76-4653-9047-0f5824e6cfbb\") " Nov 01 16:19:06.624144 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.623941 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef2387a8-1f76-4653-9047-0f5824e6cfbb-istio-token" pod "ef2387a8-1f76-4653-9047-0f5824e6cfbb" (UID: "ef2387a8-1f76-4653-9047-0f5824e6cfbb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:06.624144 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.623961 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef2387a8-1f76-4653-9047-0f5824e6cfbb-credential-socket" pod "ef2387a8-1f76-4653-9047-0f5824e6cfbb" (UID: "ef2387a8-1f76-4653-9047-0f5824e6cfbb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:06.624144 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.624049 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef2387a8-1f76-4653-9047-0f5824e6cfbb-istio-podinfo" pod "ef2387a8-1f76-4653-9047-0f5824e6cfbb" (UID: "ef2387a8-1f76-4653-9047-0f5824e6cfbb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:19:06.624144 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.624118 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef2387a8-1f76-4653-9047-0f5824e6cfbb-istiod-ca-cert" pod "ef2387a8-1f76-4653-9047-0f5824e6cfbb" (UID: "ef2387a8-1f76-4653-9047-0f5824e6cfbb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:19:06.624421 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.624205 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef2387a8-1f76-4653-9047-0f5824e6cfbb-istio-data" pod "ef2387a8-1f76-4653-9047-0f5824e6cfbb" (UID: "ef2387a8-1f76-4653-9047-0f5824e6cfbb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:06.624421 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.624283 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef2387a8-1f76-4653-9047-0f5824e6cfbb-istio-envoy" pod "ef2387a8-1f76-4653-9047-0f5824e6cfbb" (UID: "ef2387a8-1f76-4653-9047-0f5824e6cfbb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:06.624743 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.624714 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2387a8-1f76-4653-9047-0f5824e6cfbb-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:06.624743 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.624738 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef2387a8-1f76-4653-9047-0f5824e6cfbb-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:06.624906 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.624753 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2387a8-1f76-4653-9047-0f5824e6cfbb-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:06.624906 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.624767 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef2387a8-1f76-4653-9047-0f5824e6cfbb-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:06.624906 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.624782 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef2387a8-1f76-4653-9047-0f5824e6cfbb-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:06.624906 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.624785 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69f7030b-22ab-4f75-8705-af57a56c29f5-istio-token" pod "69f7030b-22ab-4f75-8705-af57a56c29f5" (UID: "69f7030b-22ab-4f75-8705-af57a56c29f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:06.624906 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.624795 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69f7030b-22ab-4f75-8705-af57a56c29f5-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:06.624906 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.624831 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef2387a8-1f76-4653-9047-0f5824e6cfbb-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:06.624906 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.624850 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69f7030b-22ab-4f75-8705-af57a56c29f5-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:06.624906 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.624865 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef2387a8-1f76-4653-9047-0f5824e6cfbb-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:06.624906 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.624878 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69f7030b-22ab-4f75-8705-af57a56c29f5-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:06.624906 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.624893 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69f7030b-22ab-4f75-8705-af57a56c29f5-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:06.624906 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.624907 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69f7030b-22ab-4f75-8705-af57a56c29f5-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:06.625590 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.624920 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef2387a8-1f76-4653-9047-0f5824e6cfbb-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:06.625590 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.625374 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef2387a8-1f76-4653-9047-0f5824e6cfbb-kube-api-access-z2l2k" pod "ef2387a8-1f76-4653-9047-0f5824e6cfbb" (UID: "ef2387a8-1f76-4653-9047-0f5824e6cfbb"). InnerVolumeSpecName "kube-api-access-z2l2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:06.626395 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.626372 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69f7030b-22ab-4f75-8705-af57a56c29f5-kube-api-access-q9xdq" pod "69f7030b-22ab-4f75-8705-af57a56c29f5" (UID: "69f7030b-22ab-4f75-8705-af57a56c29f5"). InnerVolumeSpecName "kube-api-access-q9xdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:06.626691 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.626677 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69f7030b-22ab-4f75-8705-af57a56c29f5-istio-envoy" pod "69f7030b-22ab-4f75-8705-af57a56c29f5" (UID: "69f7030b-22ab-4f75-8705-af57a56c29f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:06.626888 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.626875 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69f7030b-22ab-4f75-8705-af57a56c29f5-istio-podinfo" pod "69f7030b-22ab-4f75-8705-af57a56c29f5" (UID: "69f7030b-22ab-4f75-8705-af57a56c29f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:19:06.741084 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.735449 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69f7030b-22ab-4f75-8705-af57a56c29f5-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:06.741084 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.735485 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9xdq\" (UniqueName: \"kubernetes.io/projected/69f7030b-22ab-4f75-8705-af57a56c29f5-kube-api-access-q9xdq\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:06.741084 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.735502 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69f7030b-22ab-4f75-8705-af57a56c29f5-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:06.741084 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.735515 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69f7030b-22ab-4f75-8705-af57a56c29f5-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:06.741084 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:06.735529 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2l2k\" (UniqueName: \"kubernetes.io/projected/ef2387a8-1f76-4653-9047-0f5824e6cfbb-kube-api-access-z2l2k\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:07.081393 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:07.081361 4363 generic.go:356] "Generic (PLEG): container finished" podID="05ec5955-5241-423b-8161-1ac7608f4fed" containerID="c3f9604d2929249e6d9581fe956ae902f56c348867bcbc7a8dc1422d707df3b4" exitCode=0 Nov 01 16:19:07.081882 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:07.081861 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" Nov 01 16:19:07.082677 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:07.082096 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" event={"ID":"05ec5955-5241-423b-8161-1ac7608f4fed","Type":"ContainerDied","Data":"c3f9604d2929249e6d9581fe956ae902f56c348867bcbc7a8dc1422d707df3b4"} Nov 01 16:19:07.082870 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:07.082847 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv" event={"ID":"05ec5955-5241-423b-8161-1ac7608f4fed","Type":"ContainerDied","Data":"6993cd40ba81d6c97fe770becac6e02a93d9df092fdb83388d08ec41d6c3da20"} Nov 01 16:19:07.084272 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:07.083696 4363 scope.go:122] "RemoveContainer" containerID="c3f9604d2929249e6d9581fe956ae902f56c348867bcbc7a8dc1422d707df3b4" Nov 01 16:19:07.086345 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:07.085602 4363 generic.go:356] "Generic (PLEG): container finished" podID="69f7030b-22ab-4f75-8705-af57a56c29f5" containerID="2936c71b4370d557c4253153a44fdc51748064e8a380ddce3c305747cba26b50" exitCode=0 Nov 01 16:19:07.086345 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:07.085652 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" Nov 01 16:19:07.086345 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:07.085675 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" event={"ID":"69f7030b-22ab-4f75-8705-af57a56c29f5","Type":"ContainerDied","Data":"2936c71b4370d557c4253153a44fdc51748064e8a380ddce3c305747cba26b50"} Nov 01 16:19:07.086345 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:07.086106 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8" event={"ID":"69f7030b-22ab-4f75-8705-af57a56c29f5","Type":"ContainerDied","Data":"e0b7c77b7346e5143be3e28b90579858554586f2a6b9d6dc33728260dfe13ede"} Nov 01 16:19:07.093136 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:07.091873 4363 generic.go:356] "Generic (PLEG): container finished" podID="ef2387a8-1f76-4653-9047-0f5824e6cfbb" containerID="5b3486d4b53f8d69654824e4e3d3aa10bf2fbba2dab07c4b14a96e26c66ef5a0" exitCode=0 Nov 01 16:19:07.093136 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:07.092047 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" event={"ID":"ef2387a8-1f76-4653-9047-0f5824e6cfbb","Type":"ContainerDied","Data":"5b3486d4b53f8d69654824e4e3d3aa10bf2fbba2dab07c4b14a96e26c66ef5a0"} Nov 01 16:19:07.093136 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:07.092087 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" event={"ID":"ef2387a8-1f76-4653-9047-0f5824e6cfbb","Type":"ContainerDied","Data":"3383aee1e0a81e8efd76a3f026fbbfadd19794f80feb4d65f8b418429a0bd7d8"} Nov 01 16:19:07.093136 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:07.092183 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4" Nov 01 16:19:07.101917 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:07.101888 4363 scope.go:122] "RemoveContainer" containerID="c3f9604d2929249e6d9581fe956ae902f56c348867bcbc7a8dc1422d707df3b4" Nov 01 16:19:07.102254 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:19:07.102234 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3f9604d2929249e6d9581fe956ae902f56c348867bcbc7a8dc1422d707df3b4\": container with ID starting with c3f9604d2929249e6d9581fe956ae902f56c348867bcbc7a8dc1422d707df3b4 not found: ID does not exist" containerID="c3f9604d2929249e6d9581fe956ae902f56c348867bcbc7a8dc1422d707df3b4" Nov 01 16:19:07.102343 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:07.102261 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3f9604d2929249e6d9581fe956ae902f56c348867bcbc7a8dc1422d707df3b4"} err="failed to get container status \"c3f9604d2929249e6d9581fe956ae902f56c348867bcbc7a8dc1422d707df3b4\": rpc error: code = NotFound desc = could not find container \"c3f9604d2929249e6d9581fe956ae902f56c348867bcbc7a8dc1422d707df3b4\": container with ID starting with c3f9604d2929249e6d9581fe956ae902f56c348867bcbc7a8dc1422d707df3b4 not found: ID does not exist" Nov 01 16:19:07.102343 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:07.102278 4363 scope.go:122] "RemoveContainer" containerID="2936c71b4370d557c4253153a44fdc51748064e8a380ddce3c305747cba26b50" Nov 01 16:19:07.114486 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:07.114463 4363 scope.go:122] "RemoveContainer" containerID="600118d493d6e661c338eef2b89235dfcee2bc96d4c1467a14e048f0881400f1" Nov 01 16:19:07.124444 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:07.124420 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8"] Nov 01 16:19:07.130351 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:07.130327 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-7s2p8"] Nov 01 16:19:07.137407 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:07.137389 4363 scope.go:122] "RemoveContainer" containerID="2936c71b4370d557c4253153a44fdc51748064e8a380ddce3c305747cba26b50" Nov 01 16:19:07.137891 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:19:07.137868 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2936c71b4370d557c4253153a44fdc51748064e8a380ddce3c305747cba26b50\": container with ID starting with 2936c71b4370d557c4253153a44fdc51748064e8a380ddce3c305747cba26b50 not found: ID does not exist" containerID="2936c71b4370d557c4253153a44fdc51748064e8a380ddce3c305747cba26b50" Nov 01 16:19:07.138449 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:07.138416 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2936c71b4370d557c4253153a44fdc51748064e8a380ddce3c305747cba26b50"} err="failed to get container status \"2936c71b4370d557c4253153a44fdc51748064e8a380ddce3c305747cba26b50\": rpc error: code = NotFound desc = could not find container \"2936c71b4370d557c4253153a44fdc51748064e8a380ddce3c305747cba26b50\": container with ID starting with 2936c71b4370d557c4253153a44fdc51748064e8a380ddce3c305747cba26b50 not found: ID does not exist" Nov 01 16:19:07.138578 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:07.138566 4363 scope.go:122] "RemoveContainer" containerID="600118d493d6e661c338eef2b89235dfcee2bc96d4c1467a14e048f0881400f1" Nov 01 16:19:07.138973 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:19:07.138953 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"600118d493d6e661c338eef2b89235dfcee2bc96d4c1467a14e048f0881400f1\": container with ID starting with 600118d493d6e661c338eef2b89235dfcee2bc96d4c1467a14e048f0881400f1 not found: ID does not exist" containerID="600118d493d6e661c338eef2b89235dfcee2bc96d4c1467a14e048f0881400f1" Nov 01 16:19:07.139231 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:07.139208 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"600118d493d6e661c338eef2b89235dfcee2bc96d4c1467a14e048f0881400f1"} err="failed to get container status \"600118d493d6e661c338eef2b89235dfcee2bc96d4c1467a14e048f0881400f1\": rpc error: code = NotFound desc = could not find container \"600118d493d6e661c338eef2b89235dfcee2bc96d4c1467a14e048f0881400f1\": container with ID starting with 600118d493d6e661c338eef2b89235dfcee2bc96d4c1467a14e048f0881400f1 not found: ID does not exist" Nov 01 16:19:07.139341 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:07.139330 4363 scope.go:122] "RemoveContainer" containerID="5b3486d4b53f8d69654824e4e3d3aa10bf2fbba2dab07c4b14a96e26c66ef5a0" Nov 01 16:19:07.143585 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:07.143566 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4"] Nov 01 16:19:07.151204 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:07.151180 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-rbqs4"] Nov 01 16:19:07.158793 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:07.158777 4363 scope.go:122] "RemoveContainer" containerID="5b3486d4b53f8d69654824e4e3d3aa10bf2fbba2dab07c4b14a96e26c66ef5a0" Nov 01 16:19:07.159675 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:19:07.159631 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b3486d4b53f8d69654824e4e3d3aa10bf2fbba2dab07c4b14a96e26c66ef5a0\": container with ID starting with 5b3486d4b53f8d69654824e4e3d3aa10bf2fbba2dab07c4b14a96e26c66ef5a0 not found: ID does not exist" containerID="5b3486d4b53f8d69654824e4e3d3aa10bf2fbba2dab07c4b14a96e26c66ef5a0" Nov 01 16:19:07.159856 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:07.159686 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b3486d4b53f8d69654824e4e3d3aa10bf2fbba2dab07c4b14a96e26c66ef5a0"} err="failed to get container status \"5b3486d4b53f8d69654824e4e3d3aa10bf2fbba2dab07c4b14a96e26c66ef5a0\": rpc error: code = NotFound desc = could not find container \"5b3486d4b53f8d69654824e4e3d3aa10bf2fbba2dab07c4b14a96e26c66ef5a0\": container with ID starting with 5b3486d4b53f8d69654824e4e3d3aa10bf2fbba2dab07c4b14a96e26c66ef5a0 not found: ID does not exist" Nov 01 16:19:07.161334 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:07.161312 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv"] Nov 01 16:19:07.169425 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:07.169396 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-zn5vv"] Nov 01 16:19:07.893865 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:07.893830 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-375350-76f869b4d8-ljvvj"] Nov 01 16:19:07.894425 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:07.894112 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-375350-76f869b4d8-ljvvj" podUID="cf5d34ea-8ba0-44e6-86f8-85e1dfc59412" containerName="maas-api" containerID="cri-o://369ab74a0e91a1d649356ffb030da0fdef8efde8cbb6a8f906c77e39c988f2ac" gracePeriod=30 Nov 01 16:19:08.571932 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:08.571680 4363 generic.go:356] "Generic (PLEG): container finished" podID="cf5d34ea-8ba0-44e6-86f8-85e1dfc59412" containerID="369ab74a0e91a1d649356ffb030da0fdef8efde8cbb6a8f906c77e39c988f2ac" exitCode=0 Nov 01 16:19:08.571932 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:08.571844 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-375350-76f869b4d8-ljvvj" event={"ID":"cf5d34ea-8ba0-44e6-86f8-85e1dfc59412","Type":"ContainerDied","Data":"369ab74a0e91a1d649356ffb030da0fdef8efde8cbb6a8f906c77e39c988f2ac"} Nov 01 16:19:08.616522 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:08.616495 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-375350-76f869b4d8-ljvvj" Nov 01 16:19:08.727046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:08.722990 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf5d34ea-8ba0-44e6-86f8-85e1dfc59412-kube-api-access-gns29\" (UniqueName: \"kubernetes.io/projected/cf5d34ea-8ba0-44e6-86f8-85e1dfc59412-kube-api-access-gns29\") pod \"cf5d34ea-8ba0-44e6-86f8-85e1dfc59412\" (UID: \"cf5d34ea-8ba0-44e6-86f8-85e1dfc59412\") " Nov 01 16:19:08.727046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:08.723341 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf5d34ea-8ba0-44e6-86f8-85e1dfc59412-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf5d34ea-8ba0-44e6-86f8-85e1dfc59412-metrics-certs\") pod \"cf5d34ea-8ba0-44e6-86f8-85e1dfc59412\" (UID: \"cf5d34ea-8ba0-44e6-86f8-85e1dfc59412\") " Nov 01 16:19:08.727046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:08.723410 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf5d34ea-8ba0-44e6-86f8-85e1dfc59412-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf5d34ea-8ba0-44e6-86f8-85e1dfc59412-maas-api-tls\") pod \"cf5d34ea-8ba0-44e6-86f8-85e1dfc59412\" (UID: \"cf5d34ea-8ba0-44e6-86f8-85e1dfc59412\") " Nov 01 16:19:08.727046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:08.723526 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf5d34ea-8ba0-44e6-86f8-85e1dfc59412-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf5d34ea-8ba0-44e6-86f8-85e1dfc59412-openshift-service-ca\") pod \"cf5d34ea-8ba0-44e6-86f8-85e1dfc59412\" (UID: \"cf5d34ea-8ba0-44e6-86f8-85e1dfc59412\") " Nov 01 16:19:08.739473 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:08.739159 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf5d34ea-8ba0-44e6-86f8-85e1dfc59412-metrics-certs" pod "cf5d34ea-8ba0-44e6-86f8-85e1dfc59412" (UID: "cf5d34ea-8ba0-44e6-86f8-85e1dfc59412"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:19:08.739771 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:08.739727 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf5d34ea-8ba0-44e6-86f8-85e1dfc59412-maas-api-tls" pod "cf5d34ea-8ba0-44e6-86f8-85e1dfc59412" (UID: "cf5d34ea-8ba0-44e6-86f8-85e1dfc59412"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:19:08.743151 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:08.742190 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf5d34ea-8ba0-44e6-86f8-85e1dfc59412-kube-api-access-gns29" pod "cf5d34ea-8ba0-44e6-86f8-85e1dfc59412" (UID: "cf5d34ea-8ba0-44e6-86f8-85e1dfc59412"). InnerVolumeSpecName "kube-api-access-gns29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:08.746047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:08.745342 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf5d34ea-8ba0-44e6-86f8-85e1dfc59412-openshift-service-ca" pod "cf5d34ea-8ba0-44e6-86f8-85e1dfc59412" (UID: "cf5d34ea-8ba0-44e6-86f8-85e1dfc59412"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:19:08.804557 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:08.804520 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05ec5955-5241-423b-8161-1ac7608f4fed" path="/var/lib/kubelet/pods/05ec5955-5241-423b-8161-1ac7608f4fed/volumes" Nov 01 16:19:08.805358 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:08.805338 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69f7030b-22ab-4f75-8705-af57a56c29f5" path="/var/lib/kubelet/pods/69f7030b-22ab-4f75-8705-af57a56c29f5/volumes" Nov 01 16:19:08.806057 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:08.806011 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef2387a8-1f76-4653-9047-0f5824e6cfbb" path="/var/lib/kubelet/pods/ef2387a8-1f76-4653-9047-0f5824e6cfbb/volumes" Nov 01 16:19:08.835960 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:08.835924 4363 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf5d34ea-8ba0-44e6-86f8-85e1dfc59412-metrics-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:08.835960 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:08.835956 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf5d34ea-8ba0-44e6-86f8-85e1dfc59412-maas-api-tls\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:08.836166 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:08.835970 4363 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf5d34ea-8ba0-44e6-86f8-85e1dfc59412-openshift-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:08.836166 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:08.835985 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gns29\" (UniqueName: \"kubernetes.io/projected/cf5d34ea-8ba0-44e6-86f8-85e1dfc59412-kube-api-access-gns29\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:09.110900 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:09.110869 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-375350-b98cbd657-76rtq" podUID="7d0c63a3-8c1f-4c54-9373-afed7b61795c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:19:09.111358 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:09.110902 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-375350-68b888b95-4mx6d" podUID="eb084141-b900-462d-ac93-b1ce933d6f2e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:19:09.585956 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:09.585876 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-375350-76f869b4d8-ljvvj" Nov 01 16:19:09.586152 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:09.585880 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-375350-76f869b4d8-ljvvj" event={"ID":"cf5d34ea-8ba0-44e6-86f8-85e1dfc59412","Type":"ContainerDied","Data":"3dad26764e931b9153e9a6b4cf212181b83c5b5055d04272db0de8d086ab6d4c"} Nov 01 16:19:09.586152 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:09.586053 4363 scope.go:122] "RemoveContainer" containerID="369ab74a0e91a1d649356ffb030da0fdef8efde8cbb6a8f906c77e39c988f2ac" Nov 01 16:19:09.752041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:09.742377 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-375350-76f869b4d8-ljvvj"] Nov 01 16:19:09.815419 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:09.812848 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-375350-76f869b4d8-ljvvj"] Nov 01 16:19:10.803530 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:10.803499 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf5d34ea-8ba0-44e6-86f8-85e1dfc59412" path="/var/lib/kubelet/pods/cf5d34ea-8ba0-44e6-86f8-85e1dfc59412/volumes" Nov 01 16:19:14.311237 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:14.310197 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m"] Nov 01 16:19:14.311237 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:14.310473 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" podUID="145ad297-6cf1-4c70-ba0e-db5047a25f20" containerName="istio-proxy" containerID="cri-o://304915312cbd6fcb813ed142f9f16e61ff87306bd617931ffe23646853f2b632" gracePeriod=30 Nov 01 16:19:15.440555 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:15.435676 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr"] Nov 01 16:19:15.448582 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:15.448539 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" podUID="9247119e-2d9d-4817-a84c-b48159236aed" containerName="istio-proxy" containerID="cri-o://6c6000120d9be4d34bec2684d7bc5fea5c34c5cb8d5b29f9933293430c2a2064" gracePeriod=30 Nov 01 16:19:15.565080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:15.565050 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw"] Nov 01 16:19:15.566276 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:15.566242 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" podUID="b3d831af-96a0-4040-8568-e092380a88c1" containerName="istio-proxy" containerID="cri-o://253b665ee5711dcb85fc8258bfbeba181f9ca1279a9748d8ae24d378a3ca5290" gracePeriod=30 Nov 01 16:19:16.172911 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:16.172874 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6"] Nov 01 16:19:16.173125 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:16.173095 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" podUID="ba075b49-a3d6-4813-bbcf-37f11e5550a1" containerName="istio-proxy" containerID="cri-o://41dcc89d213d23ad2ed60613e6b1a5f821c9952320dc43122e9f615a5e31ac94" gracePeriod=30 Nov 01 16:19:16.592223 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:16.592127 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c65e3-77686b98cf-wzm9h" podUID="06fcb8d1-bff1-4182-92d2-98a57314f6fd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:19:16.935523 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:16.935474 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" podUID="9cbd41ee-3134-4b54-9ea1-0c13921269bc" containerName="main" containerID="cri-o://9419b2e7d4db15972f5be522fe1bb744d2d77077fe7e3246499c67ca7d04f926" gracePeriod=45 Nov 01 16:19:17.309930 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:17.309900 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" Nov 01 16:19:17.407933 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:17.407890 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cbd41ee-3134-4b54-9ea1-0c13921269bc-tls-certs\" (UniqueName: \"kubernetes.io/secret/9cbd41ee-3134-4b54-9ea1-0c13921269bc-tls-certs\") pod \"9cbd41ee-3134-4b54-9ea1-0c13921269bc\" (UID: \"9cbd41ee-3134-4b54-9ea1-0c13921269bc\") " Nov 01 16:19:17.408164 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:17.408037 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cbd41ee-3134-4b54-9ea1-0c13921269bc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cbd41ee-3134-4b54-9ea1-0c13921269bc-model-cache\") pod \"9cbd41ee-3134-4b54-9ea1-0c13921269bc\" (UID: \"9cbd41ee-3134-4b54-9ea1-0c13921269bc\") " Nov 01 16:19:17.408164 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:17.408134 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cbd41ee-3134-4b54-9ea1-0c13921269bc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cbd41ee-3134-4b54-9ea1-0c13921269bc-dshm\") pod \"9cbd41ee-3134-4b54-9ea1-0c13921269bc\" (UID: \"9cbd41ee-3134-4b54-9ea1-0c13921269bc\") " Nov 01 16:19:17.408292 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:17.408270 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cbd41ee-3134-4b54-9ea1-0c13921269bc-kube-api-access-7nj2f\" (UniqueName: \"kubernetes.io/projected/9cbd41ee-3134-4b54-9ea1-0c13921269bc-kube-api-access-7nj2f\") pod \"9cbd41ee-3134-4b54-9ea1-0c13921269bc\" (UID: \"9cbd41ee-3134-4b54-9ea1-0c13921269bc\") " Nov 01 16:19:17.408352 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:17.408319 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cbd41ee-3134-4b54-9ea1-0c13921269bc-model-cache" pod "9cbd41ee-3134-4b54-9ea1-0c13921269bc" (UID: "9cbd41ee-3134-4b54-9ea1-0c13921269bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:17.408412 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:17.408330 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cbd41ee-3134-4b54-9ea1-0c13921269bc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cbd41ee-3134-4b54-9ea1-0c13921269bc-tmp-dir\") pod \"9cbd41ee-3134-4b54-9ea1-0c13921269bc\" (UID: \"9cbd41ee-3134-4b54-9ea1-0c13921269bc\") " Nov 01 16:19:17.408469 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:17.408406 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cbd41ee-3134-4b54-9ea1-0c13921269bc-home\" (UniqueName: \"kubernetes.io/empty-dir/9cbd41ee-3134-4b54-9ea1-0c13921269bc-home\") pod \"9cbd41ee-3134-4b54-9ea1-0c13921269bc\" (UID: \"9cbd41ee-3134-4b54-9ea1-0c13921269bc\") " Nov 01 16:19:17.408550 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:17.408511 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cbd41ee-3134-4b54-9ea1-0c13921269bc-tmp-dir" pod "9cbd41ee-3134-4b54-9ea1-0c13921269bc" (UID: "9cbd41ee-3134-4b54-9ea1-0c13921269bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:17.408792 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:17.408764 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cbd41ee-3134-4b54-9ea1-0c13921269bc-home" pod "9cbd41ee-3134-4b54-9ea1-0c13921269bc" (UID: "9cbd41ee-3134-4b54-9ea1-0c13921269bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:17.409317 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:17.409291 4363 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cbd41ee-3134-4b54-9ea1-0c13921269bc-tmp-dir\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:17.409317 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:17.409319 4363 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cbd41ee-3134-4b54-9ea1-0c13921269bc-home\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:17.409470 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:17.409332 4363 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cbd41ee-3134-4b54-9ea1-0c13921269bc-model-cache\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:17.410541 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:17.410518 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cbd41ee-3134-4b54-9ea1-0c13921269bc-dshm" pod "9cbd41ee-3134-4b54-9ea1-0c13921269bc" (UID: "9cbd41ee-3134-4b54-9ea1-0c13921269bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:17.410656 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:17.410636 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cbd41ee-3134-4b54-9ea1-0c13921269bc-tls-certs" pod "9cbd41ee-3134-4b54-9ea1-0c13921269bc" (UID: "9cbd41ee-3134-4b54-9ea1-0c13921269bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:19:17.410840 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:17.410820 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cbd41ee-3134-4b54-9ea1-0c13921269bc-kube-api-access-7nj2f" pod "9cbd41ee-3134-4b54-9ea1-0c13921269bc" (UID: "9cbd41ee-3134-4b54-9ea1-0c13921269bc"). InnerVolumeSpecName "kube-api-access-7nj2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:17.510671 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:17.510633 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nj2f\" (UniqueName: \"kubernetes.io/projected/9cbd41ee-3134-4b54-9ea1-0c13921269bc-kube-api-access-7nj2f\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:17.510671 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:17.510668 4363 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cbd41ee-3134-4b54-9ea1-0c13921269bc-tls-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:17.510671 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:17.510681 4363 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cbd41ee-3134-4b54-9ea1-0c13921269bc-dshm\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:17.776705 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:17.776674 4363 generic.go:356] "Generic (PLEG): container finished" podID="9cbd41ee-3134-4b54-9ea1-0c13921269bc" containerID="9419b2e7d4db15972f5be522fe1bb744d2d77077fe7e3246499c67ca7d04f926" exitCode=0 Nov 01 16:19:17.777228 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:17.776716 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" Nov 01 16:19:17.777228 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:17.776746 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" event={"ID":"9cbd41ee-3134-4b54-9ea1-0c13921269bc","Type":"ContainerDied","Data":"9419b2e7d4db15972f5be522fe1bb744d2d77077fe7e3246499c67ca7d04f926"} Nov 01 16:19:17.777228 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:17.776786 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb" event={"ID":"9cbd41ee-3134-4b54-9ea1-0c13921269bc","Type":"ContainerDied","Data":"3b6ec5f3f1103f055790ef90c637e1be10fcad9933804d999c07e4e473a3ab7e"} Nov 01 16:19:17.777228 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:17.776812 4363 scope.go:122] "RemoveContainer" containerID="9419b2e7d4db15972f5be522fe1bb744d2d77077fe7e3246499c67ca7d04f926" Nov 01 16:19:17.790828 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:17.790807 4363 scope.go:122] "RemoveContainer" containerID="9419b2e7d4db15972f5be522fe1bb744d2d77077fe7e3246499c67ca7d04f926" Nov 01 16:19:17.791216 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:19:17.791197 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9419b2e7d4db15972f5be522fe1bb744d2d77077fe7e3246499c67ca7d04f926\": container with ID starting with 9419b2e7d4db15972f5be522fe1bb744d2d77077fe7e3246499c67ca7d04f926 not found: ID does not exist" containerID="9419b2e7d4db15972f5be522fe1bb744d2d77077fe7e3246499c67ca7d04f926" Nov 01 16:19:17.791291 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:17.791222 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9419b2e7d4db15972f5be522fe1bb744d2d77077fe7e3246499c67ca7d04f926"} err="failed to get container status \"9419b2e7d4db15972f5be522fe1bb744d2d77077fe7e3246499c67ca7d04f926\": rpc error: code = NotFound desc = could not find container \"9419b2e7d4db15972f5be522fe1bb744d2d77077fe7e3246499c67ca7d04f926\": container with ID starting with 9419b2e7d4db15972f5be522fe1bb744d2d77077fe7e3246499c67ca7d04f926 not found: ID does not exist" Nov 01 16:19:17.801359 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:17.801333 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb"] Nov 01 16:19:17.809049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:17.806077 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-784444/test-model-784444-kserve-5679d76f49-vpwkb"] Nov 01 16:19:18.311980 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:19:18.311943 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:19:18.802214 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:18.802184 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cbd41ee-3134-4b54-9ea1-0c13921269bc" path="/var/lib/kubelet/pods/9cbd41ee-3134-4b54-9ea1-0c13921269bc/volumes" Nov 01 16:19:19.110416 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:19.110371 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-375350-68b888b95-4mx6d" podUID="eb084141-b900-462d-ac93-b1ce933d6f2e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:19:19.110620 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:19.110382 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-375350-b98cbd657-76rtq" podUID="7d0c63a3-8c1f-4c54-9373-afed7b61795c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:19:19.698042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:19.697997 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" Nov 01 16:19:19.800441 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:19.800340 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/145ad297-6cf1-4c70-ba0e-db5047a25f20-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/145ad297-6cf1-4c70-ba0e-db5047a25f20-credential-socket\") pod \"145ad297-6cf1-4c70-ba0e-db5047a25f20\" (UID: \"145ad297-6cf1-4c70-ba0e-db5047a25f20\") " Nov 01 16:19:19.800441 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:19.800389 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/145ad297-6cf1-4c70-ba0e-db5047a25f20-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/145ad297-6cf1-4c70-ba0e-db5047a25f20-istiod-ca-cert\") pod \"145ad297-6cf1-4c70-ba0e-db5047a25f20\" (UID: \"145ad297-6cf1-4c70-ba0e-db5047a25f20\") " Nov 01 16:19:19.800675 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:19.800494 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/145ad297-6cf1-4c70-ba0e-db5047a25f20-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/145ad297-6cf1-4c70-ba0e-db5047a25f20-istio-envoy\") pod \"145ad297-6cf1-4c70-ba0e-db5047a25f20\" (UID: \"145ad297-6cf1-4c70-ba0e-db5047a25f20\") " Nov 01 16:19:19.800675 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:19.800554 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/145ad297-6cf1-4c70-ba0e-db5047a25f20-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/145ad297-6cf1-4c70-ba0e-db5047a25f20-workload-socket\") pod \"145ad297-6cf1-4c70-ba0e-db5047a25f20\" (UID: \"145ad297-6cf1-4c70-ba0e-db5047a25f20\") " Nov 01 16:19:19.800675 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:19.800602 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/145ad297-6cf1-4c70-ba0e-db5047a25f20-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/145ad297-6cf1-4c70-ba0e-db5047a25f20-istio-data\") pod \"145ad297-6cf1-4c70-ba0e-db5047a25f20\" (UID: \"145ad297-6cf1-4c70-ba0e-db5047a25f20\") " Nov 01 16:19:19.800842 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:19.800682 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/145ad297-6cf1-4c70-ba0e-db5047a25f20-kube-api-access-v7frg\" (UniqueName: \"kubernetes.io/projected/145ad297-6cf1-4c70-ba0e-db5047a25f20-kube-api-access-v7frg\") pod \"145ad297-6cf1-4c70-ba0e-db5047a25f20\" (UID: \"145ad297-6cf1-4c70-ba0e-db5047a25f20\") " Nov 01 16:19:19.800842 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:19.800753 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/145ad297-6cf1-4c70-ba0e-db5047a25f20-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/145ad297-6cf1-4c70-ba0e-db5047a25f20-istio-podinfo\") pod \"145ad297-6cf1-4c70-ba0e-db5047a25f20\" (UID: \"145ad297-6cf1-4c70-ba0e-db5047a25f20\") " Nov 01 16:19:19.800842 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:19.800819 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/145ad297-6cf1-4c70-ba0e-db5047a25f20-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/145ad297-6cf1-4c70-ba0e-db5047a25f20-workload-certs\") pod \"145ad297-6cf1-4c70-ba0e-db5047a25f20\" (UID: \"145ad297-6cf1-4c70-ba0e-db5047a25f20\") " Nov 01 16:19:19.801001 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:19.800860 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/145ad297-6cf1-4c70-ba0e-db5047a25f20-istio-token\" (UniqueName: \"kubernetes.io/projected/145ad297-6cf1-4c70-ba0e-db5047a25f20-istio-token\") pod \"145ad297-6cf1-4c70-ba0e-db5047a25f20\" (UID: \"145ad297-6cf1-4c70-ba0e-db5047a25f20\") " Nov 01 16:19:19.801001 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:19.800971 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/145ad297-6cf1-4c70-ba0e-db5047a25f20-credential-socket" pod "145ad297-6cf1-4c70-ba0e-db5047a25f20" (UID: "145ad297-6cf1-4c70-ba0e-db5047a25f20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:19.803355 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:19.801347 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/145ad297-6cf1-4c70-ba0e-db5047a25f20-istiod-ca-cert" pod "145ad297-6cf1-4c70-ba0e-db5047a25f20" (UID: "145ad297-6cf1-4c70-ba0e-db5047a25f20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:19:19.803355 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:19.801818 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/145ad297-6cf1-4c70-ba0e-db5047a25f20-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:19.803355 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:19.801842 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/145ad297-6cf1-4c70-ba0e-db5047a25f20-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:19.803355 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:19.801845 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/145ad297-6cf1-4c70-ba0e-db5047a25f20-workload-certs" pod "145ad297-6cf1-4c70-ba0e-db5047a25f20" (UID: "145ad297-6cf1-4c70-ba0e-db5047a25f20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:19.805061 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:19.804181 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/145ad297-6cf1-4c70-ba0e-db5047a25f20-istio-envoy" pod "145ad297-6cf1-4c70-ba0e-db5047a25f20" (UID: "145ad297-6cf1-4c70-ba0e-db5047a25f20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:19.805061 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:19.804677 4363 generic.go:356] "Generic (PLEG): container finished" podID="145ad297-6cf1-4c70-ba0e-db5047a25f20" containerID="304915312cbd6fcb813ed142f9f16e61ff87306bd617931ffe23646853f2b632" exitCode=0 Nov 01 16:19:19.805061 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:19.804849 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/145ad297-6cf1-4c70-ba0e-db5047a25f20-istio-podinfo" pod "145ad297-6cf1-4c70-ba0e-db5047a25f20" (UID: "145ad297-6cf1-4c70-ba0e-db5047a25f20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:19:19.805061 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:19.804862 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" event={"ID":"145ad297-6cf1-4c70-ba0e-db5047a25f20","Type":"ContainerDied","Data":"304915312cbd6fcb813ed142f9f16e61ff87306bd617931ffe23646853f2b632"} Nov 01 16:19:19.805061 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:19.804909 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" event={"ID":"145ad297-6cf1-4c70-ba0e-db5047a25f20","Type":"ContainerDied","Data":"23446185d65106a65413c3b9102f936ba20845727e78b56a655e6d733e5794b2"} Nov 01 16:19:19.805061 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:19.804928 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m" Nov 01 16:19:19.805061 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:19.804938 4363 scope.go:122] "RemoveContainer" containerID="304915312cbd6fcb813ed142f9f16e61ff87306bd617931ffe23646853f2b632" Nov 01 16:19:19.805061 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:19.804852 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/145ad297-6cf1-4c70-ba0e-db5047a25f20-istio-token" pod "145ad297-6cf1-4c70-ba0e-db5047a25f20" (UID: "145ad297-6cf1-4c70-ba0e-db5047a25f20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:19.805549 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:19.805297 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/145ad297-6cf1-4c70-ba0e-db5047a25f20-kube-api-access-v7frg" pod "145ad297-6cf1-4c70-ba0e-db5047a25f20" (UID: "145ad297-6cf1-4c70-ba0e-db5047a25f20"). InnerVolumeSpecName "kube-api-access-v7frg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:19.805877 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:19.805853 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/145ad297-6cf1-4c70-ba0e-db5047a25f20-istio-data" pod "145ad297-6cf1-4c70-ba0e-db5047a25f20" (UID: "145ad297-6cf1-4c70-ba0e-db5047a25f20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:19.806070 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:19.806051 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/145ad297-6cf1-4c70-ba0e-db5047a25f20-workload-socket" pod "145ad297-6cf1-4c70-ba0e-db5047a25f20" (UID: "145ad297-6cf1-4c70-ba0e-db5047a25f20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:19.840294 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:19.840274 4363 scope.go:122] "RemoveContainer" containerID="2afc294017df9e90a1e8837a1cb74a7ff430675a829c41007f669d7b80469874" Nov 01 16:19:19.851130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:19.851107 4363 scope.go:122] "RemoveContainer" containerID="304915312cbd6fcb813ed142f9f16e61ff87306bd617931ffe23646853f2b632" Nov 01 16:19:19.851424 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:19:19.851404 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"304915312cbd6fcb813ed142f9f16e61ff87306bd617931ffe23646853f2b632\": container with ID starting with 304915312cbd6fcb813ed142f9f16e61ff87306bd617931ffe23646853f2b632 not found: ID does not exist" containerID="304915312cbd6fcb813ed142f9f16e61ff87306bd617931ffe23646853f2b632" Nov 01 16:19:19.851489 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:19.851432 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"304915312cbd6fcb813ed142f9f16e61ff87306bd617931ffe23646853f2b632"} err="failed to get container status \"304915312cbd6fcb813ed142f9f16e61ff87306bd617931ffe23646853f2b632\": rpc error: code = NotFound desc = could not find container \"304915312cbd6fcb813ed142f9f16e61ff87306bd617931ffe23646853f2b632\": container with ID starting with 304915312cbd6fcb813ed142f9f16e61ff87306bd617931ffe23646853f2b632 not found: ID does not exist" Nov 01 16:19:19.851489 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:19.851451 4363 scope.go:122] "RemoveContainer" containerID="2afc294017df9e90a1e8837a1cb74a7ff430675a829c41007f669d7b80469874" Nov 01 16:19:19.851693 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:19:19.851674 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2afc294017df9e90a1e8837a1cb74a7ff430675a829c41007f669d7b80469874\": container with ID starting with 2afc294017df9e90a1e8837a1cb74a7ff430675a829c41007f669d7b80469874 not found: ID does not exist" containerID="2afc294017df9e90a1e8837a1cb74a7ff430675a829c41007f669d7b80469874" Nov 01 16:19:19.851876 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:19.851699 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2afc294017df9e90a1e8837a1cb74a7ff430675a829c41007f669d7b80469874"} err="failed to get container status \"2afc294017df9e90a1e8837a1cb74a7ff430675a829c41007f669d7b80469874\": rpc error: code = NotFound desc = could not find container \"2afc294017df9e90a1e8837a1cb74a7ff430675a829c41007f669d7b80469874\": container with ID starting with 2afc294017df9e90a1e8837a1cb74a7ff430675a829c41007f669d7b80469874 not found: ID does not exist" Nov 01 16:19:19.903329 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:19.903300 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/145ad297-6cf1-4c70-ba0e-db5047a25f20-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:19.903467 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:19.903333 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/145ad297-6cf1-4c70-ba0e-db5047a25f20-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:19.903467 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:19.903348 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/145ad297-6cf1-4c70-ba0e-db5047a25f20-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:19.903467 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:19.903363 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7frg\" (UniqueName: \"kubernetes.io/projected/145ad297-6cf1-4c70-ba0e-db5047a25f20-kube-api-access-v7frg\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:19.903467 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:19.903379 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/145ad297-6cf1-4c70-ba0e-db5047a25f20-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:19.903467 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:19.903394 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/145ad297-6cf1-4c70-ba0e-db5047a25f20-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:19.903467 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:19.903407 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/145ad297-6cf1-4c70-ba0e-db5047a25f20-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:20.131158 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:20.131126 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m"] Nov 01 16:19:20.135977 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:20.135954 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1c65e3-openshift-default-95896db5c-5kw6m"] Nov 01 16:19:20.537094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:20.532106 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:19:20.537094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:20.532170 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" podUID="9247119e-2d9d-4817-a84c-b48159236aed" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:19:20.837771 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:20.836732 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="145ad297-6cf1-4c70-ba0e-db5047a25f20" path="/var/lib/kubelet/pods/145ad297-6cf1-4c70-ba0e-db5047a25f20/volumes" Nov 01 16:19:20.867447 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:20.867390 4363 generic.go:356] "Generic (PLEG): container finished" podID="06fcb8d1-bff1-4182-92d2-98a57314f6fd" containerID="e2f2704741032bd934dedbe00593467488a3e6e04e845624b1d3ee7d22b461f3" exitCode=0 Nov 01 16:19:20.867809 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:20.867551 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c65e3-77686b98cf-wzm9h" event={"ID":"06fcb8d1-bff1-4182-92d2-98a57314f6fd","Type":"ContainerDied","Data":"e2f2704741032bd934dedbe00593467488a3e6e04e845624b1d3ee7d22b461f3"} Nov 01 16:19:20.918040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:20.914126 4363 generic.go:356] "Generic (PLEG): container finished" podID="9247119e-2d9d-4817-a84c-b48159236aed" containerID="6c6000120d9be4d34bec2684d7bc5fea5c34c5cb8d5b29f9933293430c2a2064" exitCode=0 Nov 01 16:19:20.918040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:20.914287 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" event={"ID":"9247119e-2d9d-4817-a84c-b48159236aed","Type":"ContainerDied","Data":"6c6000120d9be4d34bec2684d7bc5fea5c34c5cb8d5b29f9933293430c2a2064"} Nov 01 16:19:20.921833 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:20.921802 4363 generic.go:356] "Generic (PLEG): container finished" podID="b3d831af-96a0-4040-8568-e092380a88c1" containerID="253b665ee5711dcb85fc8258bfbeba181f9ca1279a9748d8ae24d378a3ca5290" exitCode=0 Nov 01 16:19:20.922164 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:20.922139 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" event={"ID":"b3d831af-96a0-4040-8568-e092380a88c1","Type":"ContainerDied","Data":"253b665ee5711dcb85fc8258bfbeba181f9ca1279a9748d8ae24d378a3ca5290"} Nov 01 16:19:21.020511 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.020482 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" Nov 01 16:19:21.131231 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.131199 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" Nov 01 16:19:21.139577 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.139545 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3d831af-96a0-4040-8568-e092380a88c1-istio-token\" (UniqueName: \"kubernetes.io/projected/b3d831af-96a0-4040-8568-e092380a88c1-istio-token\") pod \"b3d831af-96a0-4040-8568-e092380a88c1\" (UID: \"b3d831af-96a0-4040-8568-e092380a88c1\") " Nov 01 16:19:21.144405 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.140036 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c65e3-77686b98cf-wzm9h" Nov 01 16:19:21.144405 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.144370 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3d831af-96a0-4040-8568-e092380a88c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3d831af-96a0-4040-8568-e092380a88c1-istio-data\") pod \"b3d831af-96a0-4040-8568-e092380a88c1\" (UID: \"b3d831af-96a0-4040-8568-e092380a88c1\") " Nov 01 16:19:21.144637 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.144521 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3d831af-96a0-4040-8568-e092380a88c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3d831af-96a0-4040-8568-e092380a88c1-credential-socket\") pod \"b3d831af-96a0-4040-8568-e092380a88c1\" (UID: \"b3d831af-96a0-4040-8568-e092380a88c1\") " Nov 01 16:19:21.144637 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.144577 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3d831af-96a0-4040-8568-e092380a88c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3d831af-96a0-4040-8568-e092380a88c1-istio-envoy\") pod \"b3d831af-96a0-4040-8568-e092380a88c1\" (UID: \"b3d831af-96a0-4040-8568-e092380a88c1\") " Nov 01 16:19:21.144637 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.144628 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3d831af-96a0-4040-8568-e092380a88c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3d831af-96a0-4040-8568-e092380a88c1-istio-podinfo\") pod \"b3d831af-96a0-4040-8568-e092380a88c1\" (UID: \"b3d831af-96a0-4040-8568-e092380a88c1\") " Nov 01 16:19:21.144788 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.144734 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3d831af-96a0-4040-8568-e092380a88c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3d831af-96a0-4040-8568-e092380a88c1-workload-certs\") pod \"b3d831af-96a0-4040-8568-e092380a88c1\" (UID: \"b3d831af-96a0-4040-8568-e092380a88c1\") " Nov 01 16:19:21.146360 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.144797 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3d831af-96a0-4040-8568-e092380a88c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3d831af-96a0-4040-8568-e092380a88c1-istiod-ca-cert\") pod \"b3d831af-96a0-4040-8568-e092380a88c1\" (UID: \"b3d831af-96a0-4040-8568-e092380a88c1\") " Nov 01 16:19:21.146360 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.144831 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3d831af-96a0-4040-8568-e092380a88c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3d831af-96a0-4040-8568-e092380a88c1-workload-socket\") pod \"b3d831af-96a0-4040-8568-e092380a88c1\" (UID: \"b3d831af-96a0-4040-8568-e092380a88c1\") " Nov 01 16:19:21.146360 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.144915 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3d831af-96a0-4040-8568-e092380a88c1-kube-api-access-sbnkf\" (UniqueName: \"kubernetes.io/projected/b3d831af-96a0-4040-8568-e092380a88c1-kube-api-access-sbnkf\") pod \"b3d831af-96a0-4040-8568-e092380a88c1\" (UID: \"b3d831af-96a0-4040-8568-e092380a88c1\") " Nov 01 16:19:21.146360 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.145429 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3d831af-96a0-4040-8568-e092380a88c1-credential-socket" pod "b3d831af-96a0-4040-8568-e092380a88c1" (UID: "b3d831af-96a0-4040-8568-e092380a88c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:21.146360 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.145647 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3d831af-96a0-4040-8568-e092380a88c1-istio-data" pod "b3d831af-96a0-4040-8568-e092380a88c1" (UID: "b3d831af-96a0-4040-8568-e092380a88c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:21.146360 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.146068 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3d831af-96a0-4040-8568-e092380a88c1-workload-certs" pod "b3d831af-96a0-4040-8568-e092380a88c1" (UID: "b3d831af-96a0-4040-8568-e092380a88c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:21.146360 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.146143 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3d831af-96a0-4040-8568-e092380a88c1-istiod-ca-cert" pod "b3d831af-96a0-4040-8568-e092380a88c1" (UID: "b3d831af-96a0-4040-8568-e092380a88c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:19:21.146360 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.146266 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3d831af-96a0-4040-8568-e092380a88c1-workload-socket" pod "b3d831af-96a0-4040-8568-e092380a88c1" (UID: "b3d831af-96a0-4040-8568-e092380a88c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:21.147510 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.147468 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3d831af-96a0-4040-8568-e092380a88c1-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.147510 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.147493 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3d831af-96a0-4040-8568-e092380a88c1-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.147510 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.147508 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3d831af-96a0-4040-8568-e092380a88c1-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.147729 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.147522 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3d831af-96a0-4040-8568-e092380a88c1-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.147729 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.147534 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3d831af-96a0-4040-8568-e092380a88c1-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.147729 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.147575 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3d831af-96a0-4040-8568-e092380a88c1-istio-podinfo" pod "b3d831af-96a0-4040-8568-e092380a88c1" (UID: "b3d831af-96a0-4040-8568-e092380a88c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:19:21.149000 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.148980 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3d831af-96a0-4040-8568-e092380a88c1-istio-envoy" pod "b3d831af-96a0-4040-8568-e092380a88c1" (UID: "b3d831af-96a0-4040-8568-e092380a88c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:21.150480 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.150455 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3d831af-96a0-4040-8568-e092380a88c1-istio-token" pod "b3d831af-96a0-4040-8568-e092380a88c1" (UID: "b3d831af-96a0-4040-8568-e092380a88c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:21.169451 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.169413 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3d831af-96a0-4040-8568-e092380a88c1-kube-api-access-sbnkf" pod "b3d831af-96a0-4040-8568-e092380a88c1" (UID: "b3d831af-96a0-4040-8568-e092380a88c1"). InnerVolumeSpecName "kube-api-access-sbnkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:21.248990 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.248799 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9247119e-2d9d-4817-a84c-b48159236aed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9247119e-2d9d-4817-a84c-b48159236aed-workload-socket\") pod \"9247119e-2d9d-4817-a84c-b48159236aed\" (UID: \"9247119e-2d9d-4817-a84c-b48159236aed\") " Nov 01 16:19:21.248990 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.248875 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06fcb8d1-bff1-4182-92d2-98a57314f6fd-kube-api-access-hjplw\" (UniqueName: \"kubernetes.io/projected/06fcb8d1-bff1-4182-92d2-98a57314f6fd-kube-api-access-hjplw\") pod \"06fcb8d1-bff1-4182-92d2-98a57314f6fd\" (UID: \"06fcb8d1-bff1-4182-92d2-98a57314f6fd\") " Nov 01 16:19:21.248990 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.248964 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9247119e-2d9d-4817-a84c-b48159236aed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9247119e-2d9d-4817-a84c-b48159236aed-workload-certs\") pod \"9247119e-2d9d-4817-a84c-b48159236aed\" (UID: \"9247119e-2d9d-4817-a84c-b48159236aed\") " Nov 01 16:19:21.249309 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.249008 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9247119e-2d9d-4817-a84c-b48159236aed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9247119e-2d9d-4817-a84c-b48159236aed-istio-podinfo\") pod \"9247119e-2d9d-4817-a84c-b48159236aed\" (UID: \"9247119e-2d9d-4817-a84c-b48159236aed\") " Nov 01 16:19:21.249309 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.249085 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9247119e-2d9d-4817-a84c-b48159236aed-istio-token\" (UniqueName: \"kubernetes.io/projected/9247119e-2d9d-4817-a84c-b48159236aed-istio-token\") pod \"9247119e-2d9d-4817-a84c-b48159236aed\" (UID: \"9247119e-2d9d-4817-a84c-b48159236aed\") " Nov 01 16:19:21.249309 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.249128 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9247119e-2d9d-4817-a84c-b48159236aed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9247119e-2d9d-4817-a84c-b48159236aed-istio-envoy\") pod \"9247119e-2d9d-4817-a84c-b48159236aed\" (UID: \"9247119e-2d9d-4817-a84c-b48159236aed\") " Nov 01 16:19:21.249309 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.249164 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9247119e-2d9d-4817-a84c-b48159236aed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9247119e-2d9d-4817-a84c-b48159236aed-credential-socket\") pod \"9247119e-2d9d-4817-a84c-b48159236aed\" (UID: \"9247119e-2d9d-4817-a84c-b48159236aed\") " Nov 01 16:19:21.249309 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.249227 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9247119e-2d9d-4817-a84c-b48159236aed-workload-socket" pod "9247119e-2d9d-4817-a84c-b48159236aed" (UID: "9247119e-2d9d-4817-a84c-b48159236aed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:21.249309 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.249250 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9247119e-2d9d-4817-a84c-b48159236aed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9247119e-2d9d-4817-a84c-b48159236aed-istiod-ca-cert\") pod \"9247119e-2d9d-4817-a84c-b48159236aed\" (UID: \"9247119e-2d9d-4817-a84c-b48159236aed\") " Nov 01 16:19:21.249649 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.249331 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9247119e-2d9d-4817-a84c-b48159236aed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9247119e-2d9d-4817-a84c-b48159236aed-istio-data\") pod \"9247119e-2d9d-4817-a84c-b48159236aed\" (UID: \"9247119e-2d9d-4817-a84c-b48159236aed\") " Nov 01 16:19:21.249649 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.249482 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9247119e-2d9d-4817-a84c-b48159236aed-workload-certs" pod "9247119e-2d9d-4817-a84c-b48159236aed" (UID: "9247119e-2d9d-4817-a84c-b48159236aed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:21.249649 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.249529 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06fcb8d1-bff1-4182-92d2-98a57314f6fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06fcb8d1-bff1-4182-92d2-98a57314f6fd-plugins-config-volume\") pod \"06fcb8d1-bff1-4182-92d2-98a57314f6fd\" (UID: \"06fcb8d1-bff1-4182-92d2-98a57314f6fd\") " Nov 01 16:19:21.249813 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.249676 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9247119e-2d9d-4817-a84c-b48159236aed-credential-socket" pod "9247119e-2d9d-4817-a84c-b48159236aed" (UID: "9247119e-2d9d-4817-a84c-b48159236aed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:21.249813 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.249684 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9247119e-2d9d-4817-a84c-b48159236aed-kube-api-access-w8fkl\" (UniqueName: \"kubernetes.io/projected/9247119e-2d9d-4817-a84c-b48159236aed-kube-api-access-w8fkl\") pod \"9247119e-2d9d-4817-a84c-b48159236aed\" (UID: \"9247119e-2d9d-4817-a84c-b48159236aed\") " Nov 01 16:19:21.250587 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.250063 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9247119e-2d9d-4817-a84c-b48159236aed-istio-data" pod "9247119e-2d9d-4817-a84c-b48159236aed" (UID: "9247119e-2d9d-4817-a84c-b48159236aed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:21.250587 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.250511 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06fcb8d1-bff1-4182-92d2-98a57314f6fd-plugins-config-volume" pod "06fcb8d1-bff1-4182-92d2-98a57314f6fd" (UID: "06fcb8d1-bff1-4182-92d2-98a57314f6fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:19:21.250587 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.250525 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9247119e-2d9d-4817-a84c-b48159236aed-istiod-ca-cert" pod "9247119e-2d9d-4817-a84c-b48159236aed" (UID: "9247119e-2d9d-4817-a84c-b48159236aed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:19:21.250587 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.250560 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9247119e-2d9d-4817-a84c-b48159236aed-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.250587 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.250581 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9247119e-2d9d-4817-a84c-b48159236aed-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.250874 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.250595 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3d831af-96a0-4040-8568-e092380a88c1-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.250874 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.250608 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9247119e-2d9d-4817-a84c-b48159236aed-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.250874 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.250621 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3d831af-96a0-4040-8568-e092380a88c1-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.250874 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.250632 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9247119e-2d9d-4817-a84c-b48159236aed-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.250874 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.250663 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbnkf\" (UniqueName: \"kubernetes.io/projected/b3d831af-96a0-4040-8568-e092380a88c1-kube-api-access-sbnkf\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.250874 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.250679 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3d831af-96a0-4040-8568-e092380a88c1-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.252176 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.252152 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06fcb8d1-bff1-4182-92d2-98a57314f6fd-kube-api-access-hjplw" pod "06fcb8d1-bff1-4182-92d2-98a57314f6fd" (UID: "06fcb8d1-bff1-4182-92d2-98a57314f6fd"). InnerVolumeSpecName "kube-api-access-hjplw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:21.252860 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.252839 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9247119e-2d9d-4817-a84c-b48159236aed-istio-envoy" pod "9247119e-2d9d-4817-a84c-b48159236aed" (UID: "9247119e-2d9d-4817-a84c-b48159236aed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:21.253148 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.253131 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9247119e-2d9d-4817-a84c-b48159236aed-istio-podinfo" pod "9247119e-2d9d-4817-a84c-b48159236aed" (UID: "9247119e-2d9d-4817-a84c-b48159236aed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:19:21.253588 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.253570 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9247119e-2d9d-4817-a84c-b48159236aed-kube-api-access-w8fkl" pod "9247119e-2d9d-4817-a84c-b48159236aed" (UID: "9247119e-2d9d-4817-a84c-b48159236aed"). InnerVolumeSpecName "kube-api-access-w8fkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:21.253865 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.253842 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9247119e-2d9d-4817-a84c-b48159236aed-istio-token" pod "9247119e-2d9d-4817-a84c-b48159236aed" (UID: "9247119e-2d9d-4817-a84c-b48159236aed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:21.370899 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.370309 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06fcb8d1-bff1-4182-92d2-98a57314f6fd-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.370899 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.370353 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8fkl\" (UniqueName: \"kubernetes.io/projected/9247119e-2d9d-4817-a84c-b48159236aed-kube-api-access-w8fkl\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.370899 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.370370 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjplw\" (UniqueName: \"kubernetes.io/projected/06fcb8d1-bff1-4182-92d2-98a57314f6fd-kube-api-access-hjplw\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.370899 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.370385 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9247119e-2d9d-4817-a84c-b48159236aed-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.370899 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.370398 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9247119e-2d9d-4817-a84c-b48159236aed-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.370899 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.370413 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9247119e-2d9d-4817-a84c-b48159236aed-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.370899 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.370426 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9247119e-2d9d-4817-a84c-b48159236aed-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.564281 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.564252 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" Nov 01 16:19:21.578462 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.578426 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba075b49-a3d6-4813-bbcf-37f11e5550a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba075b49-a3d6-4813-bbcf-37f11e5550a1-workload-certs\") pod \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\" (UID: \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\") " Nov 01 16:19:21.578631 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.578531 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba075b49-a3d6-4813-bbcf-37f11e5550a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba075b49-a3d6-4813-bbcf-37f11e5550a1-istiod-ca-cert\") pod \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\" (UID: \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\") " Nov 01 16:19:21.578719 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.578671 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba075b49-a3d6-4813-bbcf-37f11e5550a1-istio-token\" (UniqueName: \"kubernetes.io/projected/ba075b49-a3d6-4813-bbcf-37f11e5550a1-istio-token\") pod \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\" (UID: \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\") " Nov 01 16:19:21.578719 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.578694 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba075b49-a3d6-4813-bbcf-37f11e5550a1-workload-certs" pod "ba075b49-a3d6-4813-bbcf-37f11e5550a1" (UID: "ba075b49-a3d6-4813-bbcf-37f11e5550a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:21.578945 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.578883 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba075b49-a3d6-4813-bbcf-37f11e5550a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba075b49-a3d6-4813-bbcf-37f11e5550a1-istio-data\") pod \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\" (UID: \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\") " Nov 01 16:19:21.578945 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.578919 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba075b49-a3d6-4813-bbcf-37f11e5550a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba075b49-a3d6-4813-bbcf-37f11e5550a1-istio-podinfo\") pod \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\" (UID: \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\") " Nov 01 16:19:21.579173 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.579046 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba075b49-a3d6-4813-bbcf-37f11e5550a1-istiod-ca-cert" pod "ba075b49-a3d6-4813-bbcf-37f11e5550a1" (UID: "ba075b49-a3d6-4813-bbcf-37f11e5550a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:19:21.579173 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.579050 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba075b49-a3d6-4813-bbcf-37f11e5550a1-kube-api-access-68j26\" (UniqueName: \"kubernetes.io/projected/ba075b49-a3d6-4813-bbcf-37f11e5550a1-kube-api-access-68j26\") pod \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\" (UID: \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\") " Nov 01 16:19:21.579173 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.579130 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba075b49-a3d6-4813-bbcf-37f11e5550a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba075b49-a3d6-4813-bbcf-37f11e5550a1-workload-socket\") pod \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\" (UID: \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\") " Nov 01 16:19:21.579426 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.579178 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba075b49-a3d6-4813-bbcf-37f11e5550a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba075b49-a3d6-4813-bbcf-37f11e5550a1-credential-socket\") pod \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\" (UID: \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\") " Nov 01 16:19:21.579426 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.579250 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba075b49-a3d6-4813-bbcf-37f11e5550a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba075b49-a3d6-4813-bbcf-37f11e5550a1-istio-envoy\") pod \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\" (UID: \"ba075b49-a3d6-4813-bbcf-37f11e5550a1\") " Nov 01 16:19:21.579582 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.579509 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba075b49-a3d6-4813-bbcf-37f11e5550a1-istio-data" pod "ba075b49-a3d6-4813-bbcf-37f11e5550a1" (UID: "ba075b49-a3d6-4813-bbcf-37f11e5550a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:21.580135 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.580111 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba075b49-a3d6-4813-bbcf-37f11e5550a1-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.580300 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.580140 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba075b49-a3d6-4813-bbcf-37f11e5550a1-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.580300 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.580156 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba075b49-a3d6-4813-bbcf-37f11e5550a1-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.580856 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.580831 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba075b49-a3d6-4813-bbcf-37f11e5550a1-credential-socket" pod "ba075b49-a3d6-4813-bbcf-37f11e5550a1" (UID: "ba075b49-a3d6-4813-bbcf-37f11e5550a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:21.580949 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.580894 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba075b49-a3d6-4813-bbcf-37f11e5550a1-workload-socket" pod "ba075b49-a3d6-4813-bbcf-37f11e5550a1" (UID: "ba075b49-a3d6-4813-bbcf-37f11e5550a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:21.583673 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.583431 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba075b49-a3d6-4813-bbcf-37f11e5550a1-istio-envoy" pod "ba075b49-a3d6-4813-bbcf-37f11e5550a1" (UID: "ba075b49-a3d6-4813-bbcf-37f11e5550a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:21.584154 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.584121 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba075b49-a3d6-4813-bbcf-37f11e5550a1-istio-podinfo" pod "ba075b49-a3d6-4813-bbcf-37f11e5550a1" (UID: "ba075b49-a3d6-4813-bbcf-37f11e5550a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:19:21.584997 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.584970 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba075b49-a3d6-4813-bbcf-37f11e5550a1-kube-api-access-68j26" pod "ba075b49-a3d6-4813-bbcf-37f11e5550a1" (UID: "ba075b49-a3d6-4813-bbcf-37f11e5550a1"). InnerVolumeSpecName "kube-api-access-68j26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:21.586547 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.586525 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba075b49-a3d6-4813-bbcf-37f11e5550a1-istio-token" pod "ba075b49-a3d6-4813-bbcf-37f11e5550a1" (UID: "ba075b49-a3d6-4813-bbcf-37f11e5550a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:21.682057 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.682009 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba075b49-a3d6-4813-bbcf-37f11e5550a1-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.682057 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.682056 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba075b49-a3d6-4813-bbcf-37f11e5550a1-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.682227 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.682067 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68j26\" (UniqueName: \"kubernetes.io/projected/ba075b49-a3d6-4813-bbcf-37f11e5550a1-kube-api-access-68j26\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.682227 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.682077 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba075b49-a3d6-4813-bbcf-37f11e5550a1-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.682227 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.682091 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba075b49-a3d6-4813-bbcf-37f11e5550a1-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.682227 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.682102 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba075b49-a3d6-4813-bbcf-37f11e5550a1-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.933909 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.933823 4363 generic.go:356] "Generic (PLEG): container finished" podID="ba075b49-a3d6-4813-bbcf-37f11e5550a1" containerID="41dcc89d213d23ad2ed60613e6b1a5f821c9952320dc43122e9f615a5e31ac94" exitCode=0 Nov 01 16:19:21.933909 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.933873 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" Nov 01 16:19:21.934434 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.933914 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" event={"ID":"ba075b49-a3d6-4813-bbcf-37f11e5550a1","Type":"ContainerDied","Data":"41dcc89d213d23ad2ed60613e6b1a5f821c9952320dc43122e9f615a5e31ac94"} Nov 01 16:19:21.934434 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.933963 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6" event={"ID":"ba075b49-a3d6-4813-bbcf-37f11e5550a1","Type":"ContainerDied","Data":"49e844f5036b6ee011b7e64272efc058b14de10edfc0c3d2c558194ea382e764"} Nov 01 16:19:21.934434 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.933987 4363 scope.go:122] "RemoveContainer" containerID="41dcc89d213d23ad2ed60613e6b1a5f821c9952320dc43122e9f615a5e31ac94" Nov 01 16:19:21.935529 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.935507 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c65e3-77686b98cf-wzm9h" Nov 01 16:19:21.935640 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.935511 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1c65e3-77686b98cf-wzm9h" event={"ID":"06fcb8d1-bff1-4182-92d2-98a57314f6fd","Type":"ContainerDied","Data":"c52624cd13d965b8ea0121410aa8e16bc01dc724392f3b2fcb7a772fbb62d745"} Nov 01 16:19:21.937276 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.937257 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" Nov 01 16:19:21.937395 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.937253 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr" event={"ID":"9247119e-2d9d-4817-a84c-b48159236aed","Type":"ContainerDied","Data":"23f255fabb151c52c879a9f6b3920f73c0b459eb9bb435ea694dea0ca0e77063"} Nov 01 16:19:21.939306 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.939281 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" event={"ID":"b3d831af-96a0-4040-8568-e092380a88c1","Type":"ContainerDied","Data":"103d4f786d309019628abecfbf9735da1ba2f29738fe240270b977bb0d838153"} Nov 01 16:19:21.939306 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.939303 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw" Nov 01 16:19:21.949682 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.949660 4363 scope.go:122] "RemoveContainer" containerID="57d261958ee14cf1fb89882b6a7e49c3f10510781bfa7b4b0ce85668171dae1f" Nov 01 16:19:21.961548 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.961530 4363 scope.go:122] "RemoveContainer" containerID="41dcc89d213d23ad2ed60613e6b1a5f821c9952320dc43122e9f615a5e31ac94" Nov 01 16:19:21.961821 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:19:21.961797 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41dcc89d213d23ad2ed60613e6b1a5f821c9952320dc43122e9f615a5e31ac94\": container with ID starting with 41dcc89d213d23ad2ed60613e6b1a5f821c9952320dc43122e9f615a5e31ac94 not found: ID does not exist" containerID="41dcc89d213d23ad2ed60613e6b1a5f821c9952320dc43122e9f615a5e31ac94" Nov 01 16:19:21.961911 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.961828 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41dcc89d213d23ad2ed60613e6b1a5f821c9952320dc43122e9f615a5e31ac94"} err="failed to get container status \"41dcc89d213d23ad2ed60613e6b1a5f821c9952320dc43122e9f615a5e31ac94\": rpc error: code = NotFound desc = could not find container \"41dcc89d213d23ad2ed60613e6b1a5f821c9952320dc43122e9f615a5e31ac94\": container with ID starting with 41dcc89d213d23ad2ed60613e6b1a5f821c9952320dc43122e9f615a5e31ac94 not found: ID does not exist" Nov 01 16:19:21.961911 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.961846 4363 scope.go:122] "RemoveContainer" containerID="57d261958ee14cf1fb89882b6a7e49c3f10510781bfa7b4b0ce85668171dae1f" Nov 01 16:19:21.962132 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:19:21.962117 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57d261958ee14cf1fb89882b6a7e49c3f10510781bfa7b4b0ce85668171dae1f\": container with ID starting with 57d261958ee14cf1fb89882b6a7e49c3f10510781bfa7b4b0ce85668171dae1f not found: ID does not exist" containerID="57d261958ee14cf1fb89882b6a7e49c3f10510781bfa7b4b0ce85668171dae1f" Nov 01 16:19:21.962185 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.962137 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57d261958ee14cf1fb89882b6a7e49c3f10510781bfa7b4b0ce85668171dae1f"} err="failed to get container status \"57d261958ee14cf1fb89882b6a7e49c3f10510781bfa7b4b0ce85668171dae1f\": rpc error: code = NotFound desc = could not find container \"57d261958ee14cf1fb89882b6a7e49c3f10510781bfa7b4b0ce85668171dae1f\": container with ID starting with 57d261958ee14cf1fb89882b6a7e49c3f10510781bfa7b4b0ce85668171dae1f not found: ID does not exist" Nov 01 16:19:21.962185 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.962150 4363 scope.go:122] "RemoveContainer" containerID="e2f2704741032bd934dedbe00593467488a3e6e04e845624b1d3ee7d22b461f3" Nov 01 16:19:21.971561 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.971536 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1c65e3-77686b98cf-wzm9h"] Nov 01 16:19:21.976634 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.976605 4363 scope.go:122] "RemoveContainer" containerID="6c6000120d9be4d34bec2684d7bc5fea5c34c5cb8d5b29f9933293430c2a2064" Nov 01 16:19:21.978044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.978010 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1c65e3-77686b98cf-wzm9h"] Nov 01 16:19:21.983446 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.983424 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6"] Nov 01 16:19:21.991374 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.991349 4363 scope.go:122] "RemoveContainer" containerID="253b665ee5711dcb85fc8258bfbeba181f9ca1279a9748d8ae24d378a3ca5290" Nov 01 16:19:21.994947 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:21.994923 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-b7gw6"] Nov 01 16:19:22.006453 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:22.006424 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw"] Nov 01 16:19:22.011241 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:22.011215 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-k66xw"] Nov 01 16:19:22.022176 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:22.022147 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr"] Nov 01 16:19:22.026408 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:22.026386 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-n6vmr"] Nov 01 16:19:22.845721 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:22.845690 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06fcb8d1-bff1-4182-92d2-98a57314f6fd" path="/var/lib/kubelet/pods/06fcb8d1-bff1-4182-92d2-98a57314f6fd/volumes" Nov 01 16:19:22.846848 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:22.846799 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9247119e-2d9d-4817-a84c-b48159236aed" path="/var/lib/kubelet/pods/9247119e-2d9d-4817-a84c-b48159236aed/volumes" Nov 01 16:19:22.853505 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:22.853482 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3d831af-96a0-4040-8568-e092380a88c1" path="/var/lib/kubelet/pods/b3d831af-96a0-4040-8568-e092380a88c1/volumes" Nov 01 16:19:22.854870 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:22.854846 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba075b49-a3d6-4813-bbcf-37f11e5550a1" path="/var/lib/kubelet/pods/ba075b49-a3d6-4813-bbcf-37f11e5550a1/volumes" Nov 01 16:19:22.964067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:22.964003 4363 generic.go:356] "Generic (PLEG): container finished" podID="7d0c63a3-8c1f-4c54-9373-afed7b61795c" containerID="bc94780c2d6af818ef77a848f09edb05f03b31df70a630ae9e05d5b59175d469" exitCode=0 Nov 01 16:19:22.964523 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:22.964123 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-375350-b98cbd657-76rtq" event={"ID":"7d0c63a3-8c1f-4c54-9373-afed7b61795c","Type":"ContainerDied","Data":"bc94780c2d6af818ef77a848f09edb05f03b31df70a630ae9e05d5b59175d469"} Nov 01 16:19:22.964523 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:22.964204 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-375350-b98cbd657-76rtq" event={"ID":"7d0c63a3-8c1f-4c54-9373-afed7b61795c","Type":"ContainerDied","Data":"697b4e4255ad1855f8aea8ec3a19a1819201ac59311c119915864c2a96bd5b72"} Nov 01 16:19:22.964523 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:22.964237 4363 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="697b4e4255ad1855f8aea8ec3a19a1819201ac59311c119915864c2a96bd5b72" Nov 01 16:19:22.970740 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:22.970716 4363 generic.go:356] "Generic (PLEG): container finished" podID="eb084141-b900-462d-ac93-b1ce933d6f2e" containerID="6df9f030f44edd6d80b248963bafe3935cb735feb6ac14a5b00411804b593283" exitCode=0 Nov 01 16:19:22.970897 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:22.970827 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-375350-68b888b95-4mx6d" event={"ID":"eb084141-b900-462d-ac93-b1ce933d6f2e","Type":"ContainerDied","Data":"6df9f030f44edd6d80b248963bafe3935cb735feb6ac14a5b00411804b593283"} Nov 01 16:19:22.971008 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:22.970990 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-375350-b98cbd657-76rtq" Nov 01 16:19:23.027902 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.027873 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-375350-68b888b95-4mx6d" Nov 01 16:19:23.064332 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.064305 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d0c63a3-8c1f-4c54-9373-afed7b61795c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d0c63a3-8c1f-4c54-9373-afed7b61795c-plugins-config-volume\") pod \"7d0c63a3-8c1f-4c54-9373-afed7b61795c\" (UID: \"7d0c63a3-8c1f-4c54-9373-afed7b61795c\") " Nov 01 16:19:23.064463 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.064417 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb084141-b900-462d-ac93-b1ce933d6f2e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb084141-b900-462d-ac93-b1ce933d6f2e-plugins-config-volume\") pod \"eb084141-b900-462d-ac93-b1ce933d6f2e\" (UID: \"eb084141-b900-462d-ac93-b1ce933d6f2e\") " Nov 01 16:19:23.064536 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.064518 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb084141-b900-462d-ac93-b1ce933d6f2e-kube-api-access-hvs54\" (UniqueName: \"kubernetes.io/projected/eb084141-b900-462d-ac93-b1ce933d6f2e-kube-api-access-hvs54\") pod \"eb084141-b900-462d-ac93-b1ce933d6f2e\" (UID: \"eb084141-b900-462d-ac93-b1ce933d6f2e\") " Nov 01 16:19:23.064829 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.064799 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d0c63a3-8c1f-4c54-9373-afed7b61795c-plugins-config-volume" pod "7d0c63a3-8c1f-4c54-9373-afed7b61795c" (UID: "7d0c63a3-8c1f-4c54-9373-afed7b61795c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:19:23.065055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.064818 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d0c63a3-8c1f-4c54-9373-afed7b61795c-kube-api-access-d6g7z\" (UniqueName: \"kubernetes.io/projected/7d0c63a3-8c1f-4c54-9373-afed7b61795c-kube-api-access-d6g7z\") pod \"7d0c63a3-8c1f-4c54-9373-afed7b61795c\" (UID: \"7d0c63a3-8c1f-4c54-9373-afed7b61795c\") " Nov 01 16:19:23.065188 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.064819 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb084141-b900-462d-ac93-b1ce933d6f2e-plugins-config-volume" pod "eb084141-b900-462d-ac93-b1ce933d6f2e" (UID: "eb084141-b900-462d-ac93-b1ce933d6f2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:19:23.066145 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.065947 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d0c63a3-8c1f-4c54-9373-afed7b61795c-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:23.066145 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.065980 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb084141-b900-462d-ac93-b1ce933d6f2e-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:23.066861 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.066823 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d0c63a3-8c1f-4c54-9373-afed7b61795c-kube-api-access-d6g7z" pod "7d0c63a3-8c1f-4c54-9373-afed7b61795c" (UID: "7d0c63a3-8c1f-4c54-9373-afed7b61795c"). InnerVolumeSpecName "kube-api-access-d6g7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:23.067064 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.067032 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb084141-b900-462d-ac93-b1ce933d6f2e-kube-api-access-hvs54" pod "eb084141-b900-462d-ac93-b1ce933d6f2e" (UID: "eb084141-b900-462d-ac93-b1ce933d6f2e"). InnerVolumeSpecName "kube-api-access-hvs54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:23.160999 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.160918 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-5e2d03/auth-test-model-5e2d03-kserve-665774f749-75t2g"] Nov 01 16:19:23.162167 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162136 4363 cpu_manager.go:414] "containerMap: removing container" podUID="b3d831af-96a0-4040-8568-e092380a88c1" containerName="istio-proxy" Nov 01 16:19:23.162167 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162158 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3d831af-96a0-4040-8568-e092380a88c1" containerName="istio-proxy" Nov 01 16:19:23.162379 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162183 4363 cpu_manager.go:414] "containerMap: removing container" podUID="ef2387a8-1f76-4653-9047-0f5824e6cfbb" containerName="istio-proxy" Nov 01 16:19:23.162379 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162190 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef2387a8-1f76-4653-9047-0f5824e6cfbb" containerName="istio-proxy" Nov 01 16:19:23.162379 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162216 4363 cpu_manager.go:414] "containerMap: removing container" podUID="ef9cc82b-caf7-4108-b80b-d10135f512b1" containerName="istio-proxy" Nov 01 16:19:23.162379 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162223 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef9cc82b-caf7-4108-b80b-d10135f512b1" containerName="istio-proxy" Nov 01 16:19:23.162379 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162239 4363 cpu_manager.go:414] "containerMap: removing container" podUID="9247119e-2d9d-4817-a84c-b48159236aed" containerName="istio-proxy" Nov 01 16:19:23.162379 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162246 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9247119e-2d9d-4817-a84c-b48159236aed" containerName="istio-proxy" Nov 01 16:19:23.162379 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162257 4363 cpu_manager.go:414] "containerMap: removing container" podUID="145ad297-6cf1-4c70-ba0e-db5047a25f20" containerName="istio-proxy" Nov 01 16:19:23.162379 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162264 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="145ad297-6cf1-4c70-ba0e-db5047a25f20" containerName="istio-proxy" Nov 01 16:19:23.162379 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162282 4363 cpu_manager.go:414] "containerMap: removing container" podUID="d92f7196-153d-46b0-9e5b-51667329cd02" containerName="main" Nov 01 16:19:23.162379 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162288 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d92f7196-153d-46b0-9e5b-51667329cd02" containerName="main" Nov 01 16:19:23.162379 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162307 4363 cpu_manager.go:414] "containerMap: removing container" podUID="ba075b49-a3d6-4813-bbcf-37f11e5550a1" containerName="istio-proxy" Nov 01 16:19:23.162379 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162315 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba075b49-a3d6-4813-bbcf-37f11e5550a1" containerName="istio-proxy" Nov 01 16:19:23.162379 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162327 4363 cpu_manager.go:414] "containerMap: removing container" podUID="69f7030b-22ab-4f75-8705-af57a56c29f5" containerName="istio-proxy" Nov 01 16:19:23.162379 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162334 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69f7030b-22ab-4f75-8705-af57a56c29f5" containerName="istio-proxy" Nov 01 16:19:23.162379 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162344 4363 cpu_manager.go:414] "containerMap: removing container" podUID="05ec5955-5241-423b-8161-1ac7608f4fed" containerName="istio-proxy" Nov 01 16:19:23.162379 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162350 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05ec5955-5241-423b-8161-1ac7608f4fed" containerName="istio-proxy" Nov 01 16:19:23.162379 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162367 4363 cpu_manager.go:414] "containerMap: removing container" podUID="a15aa191-e376-490a-8763-16cdd7685b5c" containerName="main" Nov 01 16:19:23.162379 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162374 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a15aa191-e376-490a-8763-16cdd7685b5c" containerName="main" Nov 01 16:19:23.163390 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162391 4363 cpu_manager.go:414] "containerMap: removing container" podUID="eb084141-b900-462d-ac93-b1ce933d6f2e" containerName="payload-pre-processing" Nov 01 16:19:23.163390 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162400 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb084141-b900-462d-ac93-b1ce933d6f2e" containerName="payload-pre-processing" Nov 01 16:19:23.163390 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162413 4363 cpu_manager.go:414] "containerMap: removing container" podUID="06fcb8d1-bff1-4182-92d2-98a57314f6fd" containerName="payload-pre-processing" Nov 01 16:19:23.163390 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162420 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06fcb8d1-bff1-4182-92d2-98a57314f6fd" containerName="payload-pre-processing" Nov 01 16:19:23.163390 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162433 4363 cpu_manager.go:414] "containerMap: removing container" podUID="37773d39-9d52-4b1d-94c7-68e02714a476" containerName="main" Nov 01 16:19:23.163390 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162440 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37773d39-9d52-4b1d-94c7-68e02714a476" containerName="main" Nov 01 16:19:23.163390 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162458 4363 cpu_manager.go:414] "containerMap: removing container" podUID="85d2d15c-3c9a-4b88-89cb-5f5c06979aa9" containerName="payload-processing" Nov 01 16:19:23.163390 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162466 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85d2d15c-3c9a-4b88-89cb-5f5c06979aa9" containerName="payload-processing" Nov 01 16:19:23.163390 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162478 4363 cpu_manager.go:414] "containerMap: removing container" podUID="9cbd41ee-3134-4b54-9ea1-0c13921269bc" containerName="main" Nov 01 16:19:23.163390 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162485 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cbd41ee-3134-4b54-9ea1-0c13921269bc" containerName="main" Nov 01 16:19:23.163390 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162502 4363 cpu_manager.go:414] "containerMap: removing container" podUID="7d0c63a3-8c1f-4c54-9373-afed7b61795c" containerName="payload-processing" Nov 01 16:19:23.163390 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162509 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d0c63a3-8c1f-4c54-9373-afed7b61795c" containerName="payload-processing" Nov 01 16:19:23.163390 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162534 4363 cpu_manager.go:414] "containerMap: removing container" podUID="ba075b49-a3d6-4813-bbcf-37f11e5550a1" containerName="istio-proxy" Nov 01 16:19:23.163390 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162541 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba075b49-a3d6-4813-bbcf-37f11e5550a1" containerName="istio-proxy" Nov 01 16:19:23.163390 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162559 4363 cpu_manager.go:414] "containerMap: removing container" podUID="cf5d34ea-8ba0-44e6-86f8-85e1dfc59412" containerName="maas-api" Nov 01 16:19:23.163390 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162565 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf5d34ea-8ba0-44e6-86f8-85e1dfc59412" containerName="maas-api" Nov 01 16:19:23.163390 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162589 4363 cpu_manager.go:414] "containerMap: removing container" podUID="69f7030b-22ab-4f75-8705-af57a56c29f5" containerName="istio-proxy" Nov 01 16:19:23.163390 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162596 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69f7030b-22ab-4f75-8705-af57a56c29f5" containerName="istio-proxy" Nov 01 16:19:23.163390 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162804 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef2387a8-1f76-4653-9047-0f5824e6cfbb" containerName="istio-proxy" Nov 01 16:19:23.163390 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162820 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef9cc82b-caf7-4108-b80b-d10135f512b1" containerName="istio-proxy" Nov 01 16:19:23.163390 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162831 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="9247119e-2d9d-4817-a84c-b48159236aed" containerName="istio-proxy" Nov 01 16:19:23.163390 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162844 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="145ad297-6cf1-4c70-ba0e-db5047a25f20" containerName="istio-proxy" Nov 01 16:19:23.163390 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162855 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="d92f7196-153d-46b0-9e5b-51667329cd02" containerName="main" Nov 01 16:19:23.163390 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162868 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba075b49-a3d6-4813-bbcf-37f11e5550a1" containerName="istio-proxy" Nov 01 16:19:23.163390 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162878 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="69f7030b-22ab-4f75-8705-af57a56c29f5" containerName="istio-proxy" Nov 01 16:19:23.163390 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162886 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="05ec5955-5241-423b-8161-1ac7608f4fed" containerName="istio-proxy" Nov 01 16:19:23.163390 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162898 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="a15aa191-e376-490a-8763-16cdd7685b5c" containerName="main" Nov 01 16:19:23.163390 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162910 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb084141-b900-462d-ac93-b1ce933d6f2e" containerName="payload-pre-processing" Nov 01 16:19:23.163390 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162921 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="06fcb8d1-bff1-4182-92d2-98a57314f6fd" containerName="payload-pre-processing" Nov 01 16:19:23.163390 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162928 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="37773d39-9d52-4b1d-94c7-68e02714a476" containerName="main" Nov 01 16:19:23.163390 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162940 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="85d2d15c-3c9a-4b88-89cb-5f5c06979aa9" containerName="payload-processing" Nov 01 16:19:23.163390 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162951 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cbd41ee-3134-4b54-9ea1-0c13921269bc" containerName="main" Nov 01 16:19:23.163390 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162961 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d0c63a3-8c1f-4c54-9373-afed7b61795c" containerName="payload-processing" Nov 01 16:19:23.163390 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162973 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba075b49-a3d6-4813-bbcf-37f11e5550a1" containerName="istio-proxy" Nov 01 16:19:23.163390 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162984 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf5d34ea-8ba0-44e6-86f8-85e1dfc59412" containerName="maas-api" Nov 01 16:19:23.163390 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.162997 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="69f7030b-22ab-4f75-8705-af57a56c29f5" containerName="istio-proxy" Nov 01 16:19:23.163390 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.163008 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3d831af-96a0-4040-8568-e092380a88c1" containerName="istio-proxy" Nov 01 16:19:23.167658 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.167630 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6g7z\" (UniqueName: \"kubernetes.io/projected/7d0c63a3-8c1f-4c54-9373-afed7b61795c-kube-api-access-d6g7z\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:23.167769 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.167660 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvs54\" (UniqueName: \"kubernetes.io/projected/eb084141-b900-462d-ac93-b1ce933d6f2e-kube-api-access-hvs54\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:23.168369 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.168350 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-5e2d03/auth-test-model-5e2d03-kserve-665774f749-75t2g" Nov 01 16:19:23.170138 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.170113 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-5e2d03/auth-test-model-5e2d03-kserve-665774f749-75t2g"] Nov 01 16:19:23.176298 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.176269 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-5e2d03\"/\"openshift-service-ca.crt\"" Nov 01 16:19:23.176420 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.176394 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-5e2d03\"/\"kube-root-ca.crt\"" Nov 01 16:19:23.176517 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.176501 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-5e2d03\"/\"default-dockercfg-jdl4m\"" Nov 01 16:19:23.176627 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.176585 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-5e2d03\"/\"auth-test-model-5e2d03-kserve-self-signed-certs\"" Nov 01 16:19:23.268882 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.268851 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e66518b6-c569-4afd-9464-ba934cccbcf7-model-cache\") pod \"auth-test-model-5e2d03-kserve-665774f749-75t2g\" (UID: \"e66518b6-c569-4afd-9464-ba934cccbcf7\") " pod="ai-tenant-e2e-shared-a-w0-5e2d03/auth-test-model-5e2d03-kserve-665774f749-75t2g" Nov 01 16:19:23.269089 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.268910 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e66518b6-c569-4afd-9464-ba934cccbcf7-home\") pod \"auth-test-model-5e2d03-kserve-665774f749-75t2g\" (UID: \"e66518b6-c569-4afd-9464-ba934cccbcf7\") " pod="ai-tenant-e2e-shared-a-w0-5e2d03/auth-test-model-5e2d03-kserve-665774f749-75t2g" Nov 01 16:19:23.269089 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.268989 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e66518b6-c569-4afd-9464-ba934cccbcf7-dshm\") pod \"auth-test-model-5e2d03-kserve-665774f749-75t2g\" (UID: \"e66518b6-c569-4afd-9464-ba934cccbcf7\") " pod="ai-tenant-e2e-shared-a-w0-5e2d03/auth-test-model-5e2d03-kserve-665774f749-75t2g" Nov 01 16:19:23.269089 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.269046 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e66518b6-c569-4afd-9464-ba934cccbcf7-tls-certs\") pod \"auth-test-model-5e2d03-kserve-665774f749-75t2g\" (UID: \"e66518b6-c569-4afd-9464-ba934cccbcf7\") " pod="ai-tenant-e2e-shared-a-w0-5e2d03/auth-test-model-5e2d03-kserve-665774f749-75t2g" Nov 01 16:19:23.269227 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.269136 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e66518b6-c569-4afd-9464-ba934cccbcf7-tmp-dir\") pod \"auth-test-model-5e2d03-kserve-665774f749-75t2g\" (UID: \"e66518b6-c569-4afd-9464-ba934cccbcf7\") " pod="ai-tenant-e2e-shared-a-w0-5e2d03/auth-test-model-5e2d03-kserve-665774f749-75t2g" Nov 01 16:19:23.269363 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.269348 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42xz4\" (UniqueName: \"kubernetes.io/projected/e66518b6-c569-4afd-9464-ba934cccbcf7-kube-api-access-42xz4\") pod \"auth-test-model-5e2d03-kserve-665774f749-75t2g\" (UID: \"e66518b6-c569-4afd-9464-ba934cccbcf7\") " pod="ai-tenant-e2e-shared-a-w0-5e2d03/auth-test-model-5e2d03-kserve-665774f749-75t2g" Nov 01 16:19:23.371436 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.371393 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e66518b6-c569-4afd-9464-ba934cccbcf7-home\") pod \"auth-test-model-5e2d03-kserve-665774f749-75t2g\" (UID: \"e66518b6-c569-4afd-9464-ba934cccbcf7\") " pod="ai-tenant-e2e-shared-a-w0-5e2d03/auth-test-model-5e2d03-kserve-665774f749-75t2g" Nov 01 16:19:23.372690 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.372662 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e66518b6-c569-4afd-9464-ba934cccbcf7-dshm\") pod \"auth-test-model-5e2d03-kserve-665774f749-75t2g\" (UID: \"e66518b6-c569-4afd-9464-ba934cccbcf7\") " pod="ai-tenant-e2e-shared-a-w0-5e2d03/auth-test-model-5e2d03-kserve-665774f749-75t2g" Nov 01 16:19:23.382309 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.382282 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e66518b6-c569-4afd-9464-ba934cccbcf7-tls-certs\") pod \"auth-test-model-5e2d03-kserve-665774f749-75t2g\" (UID: \"e66518b6-c569-4afd-9464-ba934cccbcf7\") " pod="ai-tenant-e2e-shared-a-w0-5e2d03/auth-test-model-5e2d03-kserve-665774f749-75t2g" Nov 01 16:19:23.383284 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.383258 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e66518b6-c569-4afd-9464-ba934cccbcf7-tmp-dir\") pod \"auth-test-model-5e2d03-kserve-665774f749-75t2g\" (UID: \"e66518b6-c569-4afd-9464-ba934cccbcf7\") " pod="ai-tenant-e2e-shared-a-w0-5e2d03/auth-test-model-5e2d03-kserve-665774f749-75t2g" Nov 01 16:19:23.384226 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.384200 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42xz4\" (UniqueName: \"kubernetes.io/projected/e66518b6-c569-4afd-9464-ba934cccbcf7-kube-api-access-42xz4\") pod \"auth-test-model-5e2d03-kserve-665774f749-75t2g\" (UID: \"e66518b6-c569-4afd-9464-ba934cccbcf7\") " pod="ai-tenant-e2e-shared-a-w0-5e2d03/auth-test-model-5e2d03-kserve-665774f749-75t2g" Nov 01 16:19:23.391895 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.391858 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e66518b6-c569-4afd-9464-ba934cccbcf7-model-cache\") pod \"auth-test-model-5e2d03-kserve-665774f749-75t2g\" (UID: \"e66518b6-c569-4afd-9464-ba934cccbcf7\") " pod="ai-tenant-e2e-shared-a-w0-5e2d03/auth-test-model-5e2d03-kserve-665774f749-75t2g" Nov 01 16:19:23.392615 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.383728 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e66518b6-c569-4afd-9464-ba934cccbcf7-dshm\") pod \"auth-test-model-5e2d03-kserve-665774f749-75t2g\" (UID: \"e66518b6-c569-4afd-9464-ba934cccbcf7\") " pod="ai-tenant-e2e-shared-a-w0-5e2d03/auth-test-model-5e2d03-kserve-665774f749-75t2g" Nov 01 16:19:23.392745 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.372118 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e66518b6-c569-4afd-9464-ba934cccbcf7-home\") pod \"auth-test-model-5e2d03-kserve-665774f749-75t2g\" (UID: \"e66518b6-c569-4afd-9464-ba934cccbcf7\") " pod="ai-tenant-e2e-shared-a-w0-5e2d03/auth-test-model-5e2d03-kserve-665774f749-75t2g" Nov 01 16:19:23.392828 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.386875 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e66518b6-c569-4afd-9464-ba934cccbcf7-tmp-dir\") pod \"auth-test-model-5e2d03-kserve-665774f749-75t2g\" (UID: \"e66518b6-c569-4afd-9464-ba934cccbcf7\") " pod="ai-tenant-e2e-shared-a-w0-5e2d03/auth-test-model-5e2d03-kserve-665774f749-75t2g" Nov 01 16:19:23.414149 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.393380 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e66518b6-c569-4afd-9464-ba934cccbcf7-model-cache\") pod \"auth-test-model-5e2d03-kserve-665774f749-75t2g\" (UID: \"e66518b6-c569-4afd-9464-ba934cccbcf7\") " pod="ai-tenant-e2e-shared-a-w0-5e2d03/auth-test-model-5e2d03-kserve-665774f749-75t2g" Nov 01 16:19:23.414413 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.412769 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e66518b6-c569-4afd-9464-ba934cccbcf7-tls-certs\") pod \"auth-test-model-5e2d03-kserve-665774f749-75t2g\" (UID: \"e66518b6-c569-4afd-9464-ba934cccbcf7\") " pod="ai-tenant-e2e-shared-a-w0-5e2d03/auth-test-model-5e2d03-kserve-665774f749-75t2g" Nov 01 16:19:23.457533 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.456162 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42xz4\" (UniqueName: \"kubernetes.io/projected/e66518b6-c569-4afd-9464-ba934cccbcf7-kube-api-access-42xz4\") pod \"auth-test-model-5e2d03-kserve-665774f749-75t2g\" (UID: \"e66518b6-c569-4afd-9464-ba934cccbcf7\") " pod="ai-tenant-e2e-shared-a-w0-5e2d03/auth-test-model-5e2d03-kserve-665774f749-75t2g" Nov 01 16:19:23.490188 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.490160 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-5e2d03/auth-test-model-5e2d03-kserve-665774f749-75t2g" Nov 01 16:19:23.848055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.847986 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-5e2d03/auth-test-model-5e2d03-kserve-665774f749-75t2g"] Nov 01 16:19:23.849055 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:19:23.849006 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode66518b6_c569_4afd_9464_ba934cccbcf7.slice/crio-259259e3cec6cec93fbca18b5e8255e7132c82e8ddb457f8098959ba40416aea WatchSource:0}: Error finding container 259259e3cec6cec93fbca18b5e8255e7132c82e8ddb457f8098959ba40416aea: Status 404 returned error can't find the container with id 259259e3cec6cec93fbca18b5e8255e7132c82e8ddb457f8098959ba40416aea Nov 01 16:19:23.988553 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.988508 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-375350-68b888b95-4mx6d" event={"ID":"eb084141-b900-462d-ac93-b1ce933d6f2e","Type":"ContainerDied","Data":"1f01067561241c49907a3d32702e867139f8f926410ed989c06bb4e255680ac9"} Nov 01 16:19:23.989060 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.988589 4363 scope.go:122] "RemoveContainer" containerID="6df9f030f44edd6d80b248963bafe3935cb735feb6ac14a5b00411804b593283" Nov 01 16:19:23.989060 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.988525 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-375350-68b888b95-4mx6d" Nov 01 16:19:23.992869 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.992546 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-375350-b98cbd657-76rtq" Nov 01 16:19:23.994267 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.994097 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-5e2d03/auth-test-model-5e2d03-kserve-665774f749-75t2g" event={"ID":"e66518b6-c569-4afd-9464-ba934cccbcf7","Type":"ContainerStarted","Data":"a31747fe9aba851c57ca451ab9361a562b2aa34e14e1a923aca1efa0ab59efd2"} Nov 01 16:19:23.994267 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:23.994149 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-5e2d03/auth-test-model-5e2d03-kserve-665774f749-75t2g" event={"ID":"e66518b6-c569-4afd-9464-ba934cccbcf7","Type":"ContainerStarted","Data":"259259e3cec6cec93fbca18b5e8255e7132c82e8ddb457f8098959ba40416aea"} Nov 01 16:19:24.034528 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:24.034479 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-5e2d03/auth-test-model-5e2d03-kserve-665774f749-75t2g" podStartSLOduration=1.034464349 podStartE2EDuration="1.034464349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:19:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:19:24.030430178 +0000 UTC m=+1897.362167829" watchObservedRunningTime="2026-11-01 16:19:24.034464349 +0000 UTC m=+1897.366202032" Nov 01 16:19:24.054211 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:24.054170 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-375350-68b888b95-4mx6d"] Nov 01 16:19:24.060421 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:24.060396 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-375350-68b888b95-4mx6d"] Nov 01 16:19:24.072417 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:24.072394 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-375350-b98cbd657-76rtq"] Nov 01 16:19:24.076379 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:24.076355 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-375350-b98cbd657-76rtq"] Nov 01 16:19:24.154096 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:24.150110 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-99e837/test-model-99e837-kserve-68448d5ff7-hfsbc"] Nov 01 16:19:24.498273 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:24.498172 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-5e2d03/auth-test-model-5e2d03-kserve-665774f749-75t2g" Nov 01 16:19:24.806877 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:24.806782 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d0c63a3-8c1f-4c54-9373-afed7b61795c" path="/var/lib/kubelet/pods/7d0c63a3-8c1f-4c54-9373-afed7b61795c/volumes" Nov 01 16:19:24.807462 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:24.807437 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb084141-b900-462d-ac93-b1ce933d6f2e" path="/var/lib/kubelet/pods/eb084141-b900-462d-ac93-b1ce933d6f2e/volumes" Nov 01 16:19:25.005743 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:25.005709 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2ce08a-5bc49b65bf-qn2rr"] Nov 01 16:19:25.013268 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:25.013241 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2ce08a-69744fddc5-cfpqd"] Nov 01 16:19:30.013039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:30.010102 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-2ce08a-5bc49b65bf-qn2rr" podUID="51ac18a7-1176-48ff-9a97-c5b1dd3d490b" containerName="payload-processing" containerID="cri-o://16ba0ca48bb170bcdf14b41f60ebd4e462b2a55dca2edb4c69733bd3630a1a46" gracePeriod=25 Nov 01 16:19:30.017095 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:30.014070 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2ce08a-69744fddc5-cfpqd" podUID="1b44172c-d6bb-4766-b1da-3343a3b46cc6" containerName="payload-pre-processing" containerID="cri-o://5e11a0c126638284753185ee7abfb5336ef8e421c3331469958393c8d1859cfa" gracePeriod=25 Nov 01 16:19:30.358801 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:30.358776 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2ce08a-5bc49b65bf-qn2rr" Nov 01 16:19:30.499578 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:30.499540 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51ac18a7-1176-48ff-9a97-c5b1dd3d490b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51ac18a7-1176-48ff-9a97-c5b1dd3d490b-plugins-config-volume\") pod \"51ac18a7-1176-48ff-9a97-c5b1dd3d490b\" (UID: \"51ac18a7-1176-48ff-9a97-c5b1dd3d490b\") " Nov 01 16:19:30.499781 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:30.499610 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51ac18a7-1176-48ff-9a97-c5b1dd3d490b-kube-api-access-6xf65\" (UniqueName: \"kubernetes.io/projected/51ac18a7-1176-48ff-9a97-c5b1dd3d490b-kube-api-access-6xf65\") pod \"51ac18a7-1176-48ff-9a97-c5b1dd3d490b\" (UID: \"51ac18a7-1176-48ff-9a97-c5b1dd3d490b\") " Nov 01 16:19:30.500115 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:30.500080 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51ac18a7-1176-48ff-9a97-c5b1dd3d490b-plugins-config-volume" pod "51ac18a7-1176-48ff-9a97-c5b1dd3d490b" (UID: "51ac18a7-1176-48ff-9a97-c5b1dd3d490b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:19:30.500571 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:30.500550 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51ac18a7-1176-48ff-9a97-c5b1dd3d490b-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:30.502304 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:30.502276 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51ac18a7-1176-48ff-9a97-c5b1dd3d490b-kube-api-access-6xf65" pod "51ac18a7-1176-48ff-9a97-c5b1dd3d490b" (UID: "51ac18a7-1176-48ff-9a97-c5b1dd3d490b"). InnerVolumeSpecName "kube-api-access-6xf65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:30.602718 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:30.602687 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xf65\" (UniqueName: \"kubernetes.io/projected/51ac18a7-1176-48ff-9a97-c5b1dd3d490b-kube-api-access-6xf65\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:30.839254 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:30.839220 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-784444-55c46475-d78lt"] Nov 01 16:19:30.876217 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:30.876133 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-784444-79c887957f-gsjc5"] Nov 01 16:19:31.159765 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:31.159676 4363 generic.go:356] "Generic (PLEG): container finished" podID="51ac18a7-1176-48ff-9a97-c5b1dd3d490b" containerID="16ba0ca48bb170bcdf14b41f60ebd4e462b2a55dca2edb4c69733bd3630a1a46" exitCode=0 Nov 01 16:19:31.160552 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:31.160505 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2ce08a-5bc49b65bf-qn2rr" event={"ID":"51ac18a7-1176-48ff-9a97-c5b1dd3d490b","Type":"ContainerDied","Data":"16ba0ca48bb170bcdf14b41f60ebd4e462b2a55dca2edb4c69733bd3630a1a46"} Nov 01 16:19:31.160871 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:31.160852 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2ce08a-5bc49b65bf-qn2rr" event={"ID":"51ac18a7-1176-48ff-9a97-c5b1dd3d490b","Type":"ContainerDied","Data":"dcf84b2acb3dd454d9207daedb321c2867c201d86af133b1771e29349099b448"} Nov 01 16:19:31.161186 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:31.160762 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2ce08a-5bc49b65bf-qn2rr" Nov 01 16:19:31.166205 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:31.160998 4363 scope.go:122] "RemoveContainer" containerID="16ba0ca48bb170bcdf14b41f60ebd4e462b2a55dca2edb4c69733bd3630a1a46" Nov 01 16:19:31.295742 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:31.295719 4363 scope.go:122] "RemoveContainer" containerID="16ba0ca48bb170bcdf14b41f60ebd4e462b2a55dca2edb4c69733bd3630a1a46" Nov 01 16:19:31.296335 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:19:31.296308 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16ba0ca48bb170bcdf14b41f60ebd4e462b2a55dca2edb4c69733bd3630a1a46\": container with ID starting with 16ba0ca48bb170bcdf14b41f60ebd4e462b2a55dca2edb4c69733bd3630a1a46 not found: ID does not exist" containerID="16ba0ca48bb170bcdf14b41f60ebd4e462b2a55dca2edb4c69733bd3630a1a46" Nov 01 16:19:31.296510 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:31.296485 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16ba0ca48bb170bcdf14b41f60ebd4e462b2a55dca2edb4c69733bd3630a1a46"} err="failed to get container status \"16ba0ca48bb170bcdf14b41f60ebd4e462b2a55dca2edb4c69733bd3630a1a46\": rpc error: code = NotFound desc = could not find container \"16ba0ca48bb170bcdf14b41f60ebd4e462b2a55dca2edb4c69733bd3630a1a46\": container with ID starting with 16ba0ca48bb170bcdf14b41f60ebd4e462b2a55dca2edb4c69733bd3630a1a46 not found: ID does not exist" Nov 01 16:19:31.359730 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:31.359672 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2ce08a-5bc49b65bf-qn2rr"] Nov 01 16:19:31.366973 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:31.365373 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2ce08a-5bc49b65bf-qn2rr"] Nov 01 16:19:32.801270 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:32.801236 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51ac18a7-1176-48ff-9a97-c5b1dd3d490b" path="/var/lib/kubelet/pods/51ac18a7-1176-48ff-9a97-c5b1dd3d490b/volumes" Nov 01 16:19:33.491165 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:33.491133 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-5e2d03/auth-test-model-5e2d03-kserve-665774f749-75t2g" Nov 01 16:19:33.509630 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:33.509601 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-5e2d03/auth-test-model-5e2d03-kserve-665774f749-75t2g" Nov 01 16:19:34.277149 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:34.277109 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-5e2d03/auth-test-model-5e2d03-kserve-665774f749-75t2g" Nov 01 16:19:35.840552 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:35.840509 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-784444-55c46475-d78lt" podUID="59ebabe0-34d2-4102-86de-52bb4d6d04e2" containerName="payload-processing" containerID="cri-o://0ade98c8ab5aecb81daf070285e586f52de6816ce0969851c7eed7e06c8cff1e" gracePeriod=25 Nov 01 16:19:35.876774 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:35.876743 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-784444-79c887957f-gsjc5" podUID="f796692a-c6ab-48b9-a227-00e8af4f24df" containerName="payload-pre-processing" containerID="cri-o://10dca15fcaf3a9014dfc285d4a55df18db1ca6cfe936347bfb3c4698f038915c" gracePeriod=25 Nov 01 16:19:36.186188 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:36.186149 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2ce08a-69744fddc5-cfpqd" podUID="1b44172c-d6bb-4766-b1da-3343a3b46cc6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:19:36.761256 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:36.761208 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2001b0-678dffcb5f-c94ht"] Nov 01 16:19:36.761520 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:36.761494 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2001b0-678dffcb5f-c94ht" podUID="9a8db0d8-0d46-4003-9dc9-679dae13b8a1" containerName="maas-api" containerID="cri-o://c42381096a57c0eb99a00a5c101b7a6f0cd6235fbb4998e913b4bf5182a4ba01" gracePeriod=30 Nov 01 16:19:37.160042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:37.154437 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2001b0-678dffcb5f-c94ht" Nov 01 16:19:37.249571 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:37.248079 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a8db0d8-0d46-4003-9dc9-679dae13b8a1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a8db0d8-0d46-4003-9dc9-679dae13b8a1-metrics-certs\") pod \"9a8db0d8-0d46-4003-9dc9-679dae13b8a1\" (UID: \"9a8db0d8-0d46-4003-9dc9-679dae13b8a1\") " Nov 01 16:19:37.249571 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:37.248402 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a8db0d8-0d46-4003-9dc9-679dae13b8a1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a8db0d8-0d46-4003-9dc9-679dae13b8a1-maas-api-tls\") pod \"9a8db0d8-0d46-4003-9dc9-679dae13b8a1\" (UID: \"9a8db0d8-0d46-4003-9dc9-679dae13b8a1\") " Nov 01 16:19:37.249571 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:37.248482 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a8db0d8-0d46-4003-9dc9-679dae13b8a1-kube-api-access-lxwx8\" (UniqueName: \"kubernetes.io/projected/9a8db0d8-0d46-4003-9dc9-679dae13b8a1-kube-api-access-lxwx8\") pod \"9a8db0d8-0d46-4003-9dc9-679dae13b8a1\" (UID: \"9a8db0d8-0d46-4003-9dc9-679dae13b8a1\") " Nov 01 16:19:37.249571 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:37.248516 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a8db0d8-0d46-4003-9dc9-679dae13b8a1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a8db0d8-0d46-4003-9dc9-679dae13b8a1-openshift-service-ca\") pod \"9a8db0d8-0d46-4003-9dc9-679dae13b8a1\" (UID: \"9a8db0d8-0d46-4003-9dc9-679dae13b8a1\") " Nov 01 16:19:37.250707 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:37.250421 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a8db0d8-0d46-4003-9dc9-679dae13b8a1-openshift-service-ca" pod "9a8db0d8-0d46-4003-9dc9-679dae13b8a1" (UID: "9a8db0d8-0d46-4003-9dc9-679dae13b8a1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:19:37.274587 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:37.274518 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a8db0d8-0d46-4003-9dc9-679dae13b8a1-maas-api-tls" pod "9a8db0d8-0d46-4003-9dc9-679dae13b8a1" (UID: "9a8db0d8-0d46-4003-9dc9-679dae13b8a1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:19:37.287562 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:37.282268 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a8db0d8-0d46-4003-9dc9-679dae13b8a1-kube-api-access-lxwx8" pod "9a8db0d8-0d46-4003-9dc9-679dae13b8a1" (UID: "9a8db0d8-0d46-4003-9dc9-679dae13b8a1"). InnerVolumeSpecName "kube-api-access-lxwx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:37.287562 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:37.282370 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a8db0d8-0d46-4003-9dc9-679dae13b8a1-metrics-certs" pod "9a8db0d8-0d46-4003-9dc9-679dae13b8a1" (UID: "9a8db0d8-0d46-4003-9dc9-679dae13b8a1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:19:37.375100 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:37.360330 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a8db0d8-0d46-4003-9dc9-679dae13b8a1-maas-api-tls\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:37.375100 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:37.360369 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxwx8\" (UniqueName: \"kubernetes.io/projected/9a8db0d8-0d46-4003-9dc9-679dae13b8a1-kube-api-access-lxwx8\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:37.375100 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:37.360386 4363 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a8db0d8-0d46-4003-9dc9-679dae13b8a1-openshift-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:37.375100 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:37.360401 4363 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a8db0d8-0d46-4003-9dc9-679dae13b8a1-metrics-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:37.405321 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:37.378594 4363 generic.go:356] "Generic (PLEG): container finished" podID="9a8db0d8-0d46-4003-9dc9-679dae13b8a1" containerID="c42381096a57c0eb99a00a5c101b7a6f0cd6235fbb4998e913b4bf5182a4ba01" exitCode=0 Nov 01 16:19:37.405321 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:37.378667 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2001b0-678dffcb5f-c94ht" Nov 01 16:19:37.405321 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:37.378698 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2001b0-678dffcb5f-c94ht" event={"ID":"9a8db0d8-0d46-4003-9dc9-679dae13b8a1","Type":"ContainerDied","Data":"c42381096a57c0eb99a00a5c101b7a6f0cd6235fbb4998e913b4bf5182a4ba01"} Nov 01 16:19:37.405321 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:37.378744 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2001b0-678dffcb5f-c94ht" event={"ID":"9a8db0d8-0d46-4003-9dc9-679dae13b8a1","Type":"ContainerDied","Data":"326ed11e67d7428c99c123c3e9e32e5046ebded06b8f07514bd1eaf06bf7c359"} Nov 01 16:19:37.405321 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:37.378773 4363 scope.go:122] "RemoveContainer" containerID="c42381096a57c0eb99a00a5c101b7a6f0cd6235fbb4998e913b4bf5182a4ba01" Nov 01 16:19:37.451526 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:37.451416 4363 scope.go:122] "RemoveContainer" containerID="c42381096a57c0eb99a00a5c101b7a6f0cd6235fbb4998e913b4bf5182a4ba01" Nov 01 16:19:37.452330 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:19:37.451962 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c42381096a57c0eb99a00a5c101b7a6f0cd6235fbb4998e913b4bf5182a4ba01\": container with ID starting with c42381096a57c0eb99a00a5c101b7a6f0cd6235fbb4998e913b4bf5182a4ba01 not found: ID does not exist" containerID="c42381096a57c0eb99a00a5c101b7a6f0cd6235fbb4998e913b4bf5182a4ba01" Nov 01 16:19:37.452330 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:37.451991 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c42381096a57c0eb99a00a5c101b7a6f0cd6235fbb4998e913b4bf5182a4ba01"} err="failed to get container status \"c42381096a57c0eb99a00a5c101b7a6f0cd6235fbb4998e913b4bf5182a4ba01\": rpc error: code = NotFound desc = could not find container \"c42381096a57c0eb99a00a5c101b7a6f0cd6235fbb4998e913b4bf5182a4ba01\": container with ID starting with c42381096a57c0eb99a00a5c101b7a6f0cd6235fbb4998e913b4bf5182a4ba01 not found: ID does not exist" Nov 01 16:19:37.459372 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:37.459352 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2001b0-678dffcb5f-c94ht"] Nov 01 16:19:37.464065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:37.464035 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2001b0-678dffcb5f-c94ht"] Nov 01 16:19:38.804527 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:38.804490 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a8db0d8-0d46-4003-9dc9-679dae13b8a1" path="/var/lib/kubelet/pods/9a8db0d8-0d46-4003-9dc9-679dae13b8a1/volumes" Nov 01 16:19:39.188364 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:39.188322 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-99e837/test-model-99e837-kserve-68448d5ff7-hfsbc" podUID="62fde750-084e-488b-9d65-24b38a323cb0" containerName="main" containerID="cri-o://6de671ef5553e04c15a2ef15397dadd2c9b689d54a98fc913b176186551708e3" gracePeriod=45 Nov 01 16:19:39.400729 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:39.400703 4363 generic.go:356] "Generic (PLEG): container finished" podID="62fde750-084e-488b-9d65-24b38a323cb0" containerID="6de671ef5553e04c15a2ef15397dadd2c9b689d54a98fc913b176186551708e3" exitCode=0 Nov 01 16:19:39.401086 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:39.400921 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-99e837/test-model-99e837-kserve-68448d5ff7-hfsbc" event={"ID":"62fde750-084e-488b-9d65-24b38a323cb0","Type":"ContainerDied","Data":"6de671ef5553e04c15a2ef15397dadd2c9b689d54a98fc913b176186551708e3"} Nov 01 16:19:39.519471 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:39.519445 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-99e837/test-model-99e837-kserve-68448d5ff7-hfsbc" Nov 01 16:19:39.668465 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:39.639566 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62fde750-084e-488b-9d65-24b38a323cb0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62fde750-084e-488b-9d65-24b38a323cb0-tmp-dir\") pod \"62fde750-084e-488b-9d65-24b38a323cb0\" (UID: \"62fde750-084e-488b-9d65-24b38a323cb0\") " Nov 01 16:19:39.668465 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:39.639617 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62fde750-084e-488b-9d65-24b38a323cb0-tls-certs\" (UniqueName: \"kubernetes.io/secret/62fde750-084e-488b-9d65-24b38a323cb0-tls-certs\") pod \"62fde750-084e-488b-9d65-24b38a323cb0\" (UID: \"62fde750-084e-488b-9d65-24b38a323cb0\") " Nov 01 16:19:39.668465 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:39.639806 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62fde750-084e-488b-9d65-24b38a323cb0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/62fde750-084e-488b-9d65-24b38a323cb0-dshm\") pod \"62fde750-084e-488b-9d65-24b38a323cb0\" (UID: \"62fde750-084e-488b-9d65-24b38a323cb0\") " Nov 01 16:19:39.668465 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:39.639915 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62fde750-084e-488b-9d65-24b38a323cb0-kube-api-access-ktnn4\" (UniqueName: \"kubernetes.io/projected/62fde750-084e-488b-9d65-24b38a323cb0-kube-api-access-ktnn4\") pod \"62fde750-084e-488b-9d65-24b38a323cb0\" (UID: \"62fde750-084e-488b-9d65-24b38a323cb0\") " Nov 01 16:19:39.668465 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:39.639992 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62fde750-084e-488b-9d65-24b38a323cb0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62fde750-084e-488b-9d65-24b38a323cb0-model-cache\") pod \"62fde750-084e-488b-9d65-24b38a323cb0\" (UID: \"62fde750-084e-488b-9d65-24b38a323cb0\") " Nov 01 16:19:39.668465 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:39.640063 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62fde750-084e-488b-9d65-24b38a323cb0-home\" (UniqueName: \"kubernetes.io/empty-dir/62fde750-084e-488b-9d65-24b38a323cb0-home\") pod \"62fde750-084e-488b-9d65-24b38a323cb0\" (UID: \"62fde750-084e-488b-9d65-24b38a323cb0\") " Nov 01 16:19:39.718060 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:39.700392 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62fde750-084e-488b-9d65-24b38a323cb0-home" pod "62fde750-084e-488b-9d65-24b38a323cb0" (UID: "62fde750-084e-488b-9d65-24b38a323cb0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:39.718060 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:39.704099 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62fde750-084e-488b-9d65-24b38a323cb0-tmp-dir" pod "62fde750-084e-488b-9d65-24b38a323cb0" (UID: "62fde750-084e-488b-9d65-24b38a323cb0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:39.718060 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:39.708080 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62fde750-084e-488b-9d65-24b38a323cb0-model-cache" pod "62fde750-084e-488b-9d65-24b38a323cb0" (UID: "62fde750-084e-488b-9d65-24b38a323cb0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:39.762221 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:39.748420 4363 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62fde750-084e-488b-9d65-24b38a323cb0-tmp-dir\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:39.762221 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:39.748451 4363 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62fde750-084e-488b-9d65-24b38a323cb0-model-cache\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:39.762221 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:39.748461 4363 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62fde750-084e-488b-9d65-24b38a323cb0-home\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:39.762221 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:39.759556 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62fde750-084e-488b-9d65-24b38a323cb0-dshm" pod "62fde750-084e-488b-9d65-24b38a323cb0" (UID: "62fde750-084e-488b-9d65-24b38a323cb0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:39.762221 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:39.760998 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62fde750-084e-488b-9d65-24b38a323cb0-tls-certs" pod "62fde750-084e-488b-9d65-24b38a323cb0" (UID: "62fde750-084e-488b-9d65-24b38a323cb0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:19:39.763502 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:39.763398 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62fde750-084e-488b-9d65-24b38a323cb0-kube-api-access-ktnn4" pod "62fde750-084e-488b-9d65-24b38a323cb0" (UID: "62fde750-084e-488b-9d65-24b38a323cb0"). InnerVolumeSpecName "kube-api-access-ktnn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:39.849922 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:39.849892 4363 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62fde750-084e-488b-9d65-24b38a323cb0-tls-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:39.849922 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:39.849920 4363 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62fde750-084e-488b-9d65-24b38a323cb0-dshm\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:39.849922 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:39.849930 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktnn4\" (UniqueName: \"kubernetes.io/projected/62fde750-084e-488b-9d65-24b38a323cb0-kube-api-access-ktnn4\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:40.412336 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:40.412302 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-99e837/test-model-99e837-kserve-68448d5ff7-hfsbc" Nov 01 16:19:40.412336 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:40.412320 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-99e837/test-model-99e837-kserve-68448d5ff7-hfsbc" event={"ID":"62fde750-084e-488b-9d65-24b38a323cb0","Type":"ContainerDied","Data":"dbcd73ccc58809ce57365a88b9581ab3ab459a046206dfbf5b167f3afcfa2b32"} Nov 01 16:19:40.412620 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:40.412375 4363 scope.go:122] "RemoveContainer" containerID="6de671ef5553e04c15a2ef15397dadd2c9b689d54a98fc913b176186551708e3" Nov 01 16:19:40.440782 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:40.440750 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-99e837/test-model-99e837-kserve-68448d5ff7-hfsbc"] Nov 01 16:19:40.445084 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:40.445060 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-99e837/test-model-99e837-kserve-68448d5ff7-hfsbc"] Nov 01 16:19:40.802216 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:40.802132 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62fde750-084e-488b-9d65-24b38a323cb0" path="/var/lib/kubelet/pods/62fde750-084e-488b-9d65-24b38a323cb0/volumes" Nov 01 16:19:44.312035 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:44.311958 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-784444-79c887957f-gsjc5" podUID="f796692a-c6ab-48b9-a227-00e8af4f24df" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:19:45.334097 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:45.334060 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-784444-55c46475-d78lt" podUID="59ebabe0-34d2-4102-86de-52bb4d6d04e2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:19:46.186322 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:46.186278 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2ce08a-69744fddc5-cfpqd" podUID="1b44172c-d6bb-4766-b1da-3343a3b46cc6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:19:46.241679 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:46.241639 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246"] Nov 01 16:19:46.242377 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:46.241942 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" podUID="387bf77d-04f3-4915-926c-96bdb07374e6" containerName="istio-proxy" containerID="cri-o://155a7788883e55c3998cd540714dfbe616961a4b54eed9ac14bf144d0a501bbf" gracePeriod=30 Nov 01 16:19:46.521343 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:46.521258 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg"] Nov 01 16:19:46.523233 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:46.523206 4363 cpu_manager.go:414] "containerMap: removing container" podUID="9a8db0d8-0d46-4003-9dc9-679dae13b8a1" containerName="maas-api" Nov 01 16:19:46.533395 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:46.533360 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a8db0d8-0d46-4003-9dc9-679dae13b8a1" containerName="maas-api" Nov 01 16:19:46.546533 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:46.546503 4363 cpu_manager.go:414] "containerMap: removing container" podUID="62fde750-084e-488b-9d65-24b38a323cb0" containerName="main" Nov 01 16:19:46.582612 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:46.582578 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62fde750-084e-488b-9d65-24b38a323cb0" containerName="main" Nov 01 16:19:46.588231 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:46.588186 4363 cpu_manager.go:414] "containerMap: removing container" podUID="51ac18a7-1176-48ff-9a97-c5b1dd3d490b" containerName="payload-processing" Nov 01 16:19:46.588642 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:46.588621 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51ac18a7-1176-48ff-9a97-c5b1dd3d490b" containerName="payload-processing" Nov 01 16:19:46.588789 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:46.588775 4363 cpu_manager.go:414] "containerMap: removing container" podUID="ef9cc82b-caf7-4108-b80b-d10135f512b1" containerName="istio-proxy" Nov 01 16:19:46.595086 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:46.595056 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef9cc82b-caf7-4108-b80b-d10135f512b1" containerName="istio-proxy" Nov 01 16:19:46.595319 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:46.595299 4363 cpu_manager.go:414] "containerMap: removing container" podUID="145ad297-6cf1-4c70-ba0e-db5047a25f20" containerName="istio-proxy" Nov 01 16:19:46.595500 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:46.595479 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="145ad297-6cf1-4c70-ba0e-db5047a25f20" containerName="istio-proxy" Nov 01 16:19:46.620040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:46.595894 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a8db0d8-0d46-4003-9dc9-679dae13b8a1" containerName="maas-api" Nov 01 16:19:46.722772 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:46.722744 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="62fde750-084e-488b-9d65-24b38a323cb0" containerName="main" Nov 01 16:19:46.723034 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:46.722999 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="51ac18a7-1176-48ff-9a97-c5b1dd3d490b" containerName="payload-processing" Nov 01 16:19:46.723164 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:46.723150 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef9cc82b-caf7-4108-b80b-d10135f512b1" containerName="istio-proxy" Nov 01 16:19:46.723258 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:46.723247 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="145ad297-6cf1-4c70-ba0e-db5047a25f20" containerName="istio-proxy" Nov 01 16:19:47.097694 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:47.097666 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" Nov 01 16:19:47.102366 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:47.101210 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-7225e5-openshift-default-dockercfg-bgkdt\"" Nov 01 16:19:47.110053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:47.110011 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg"] Nov 01 16:19:47.182986 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:47.182950 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4b349f5-acb2-42b2-8d90-01b2029167f7-istio-podinfo\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg\" (UID: \"f4b349f5-acb2-42b2-8d90-01b2029167f7\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" Nov 01 16:19:47.183204 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:47.183131 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4b349f5-acb2-42b2-8d90-01b2029167f7-istiod-ca-cert\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg\" (UID: \"f4b349f5-acb2-42b2-8d90-01b2029167f7\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" Nov 01 16:19:47.183264 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:47.183237 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b349f5-acb2-42b2-8d90-01b2029167f7-workload-socket\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg\" (UID: \"f4b349f5-acb2-42b2-8d90-01b2029167f7\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" Nov 01 16:19:47.183367 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:47.183352 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4b349f5-acb2-42b2-8d90-01b2029167f7-istio-data\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg\" (UID: \"f4b349f5-acb2-42b2-8d90-01b2029167f7\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" Nov 01 16:19:47.183451 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:47.183437 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mmk5\" (UniqueName: \"kubernetes.io/projected/f4b349f5-acb2-42b2-8d90-01b2029167f7-kube-api-access-5mmk5\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg\" (UID: \"f4b349f5-acb2-42b2-8d90-01b2029167f7\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" Nov 01 16:19:47.183596 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:47.183578 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4b349f5-acb2-42b2-8d90-01b2029167f7-workload-certs\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg\" (UID: \"f4b349f5-acb2-42b2-8d90-01b2029167f7\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" Nov 01 16:19:47.183649 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:47.183622 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4b349f5-acb2-42b2-8d90-01b2029167f7-istio-token\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg\" (UID: \"f4b349f5-acb2-42b2-8d90-01b2029167f7\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" Nov 01 16:19:47.183762 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:47.183747 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4b349f5-acb2-42b2-8d90-01b2029167f7-istio-envoy\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg\" (UID: \"f4b349f5-acb2-42b2-8d90-01b2029167f7\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" Nov 01 16:19:47.183858 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:47.183846 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b349f5-acb2-42b2-8d90-01b2029167f7-credential-socket\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg\" (UID: \"f4b349f5-acb2-42b2-8d90-01b2029167f7\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" Nov 01 16:19:47.285862 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:47.285825 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4b349f5-acb2-42b2-8d90-01b2029167f7-workload-certs\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg\" (UID: \"f4b349f5-acb2-42b2-8d90-01b2029167f7\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" Nov 01 16:19:47.286084 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:47.285870 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4b349f5-acb2-42b2-8d90-01b2029167f7-istio-token\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg\" (UID: \"f4b349f5-acb2-42b2-8d90-01b2029167f7\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" Nov 01 16:19:47.286084 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:47.285930 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4b349f5-acb2-42b2-8d90-01b2029167f7-istio-envoy\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg\" (UID: \"f4b349f5-acb2-42b2-8d90-01b2029167f7\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" Nov 01 16:19:47.286084 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:47.285996 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b349f5-acb2-42b2-8d90-01b2029167f7-credential-socket\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg\" (UID: \"f4b349f5-acb2-42b2-8d90-01b2029167f7\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" Nov 01 16:19:47.286084 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:47.286082 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4b349f5-acb2-42b2-8d90-01b2029167f7-istio-podinfo\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg\" (UID: \"f4b349f5-acb2-42b2-8d90-01b2029167f7\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" Nov 01 16:19:47.286335 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:47.286136 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4b349f5-acb2-42b2-8d90-01b2029167f7-istiod-ca-cert\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg\" (UID: \"f4b349f5-acb2-42b2-8d90-01b2029167f7\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" Nov 01 16:19:47.286335 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:47.286192 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b349f5-acb2-42b2-8d90-01b2029167f7-workload-socket\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg\" (UID: \"f4b349f5-acb2-42b2-8d90-01b2029167f7\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" Nov 01 16:19:47.286335 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:47.286264 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4b349f5-acb2-42b2-8d90-01b2029167f7-workload-certs\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg\" (UID: \"f4b349f5-acb2-42b2-8d90-01b2029167f7\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" Nov 01 16:19:47.286335 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:47.286273 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4b349f5-acb2-42b2-8d90-01b2029167f7-istio-data\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg\" (UID: \"f4b349f5-acb2-42b2-8d90-01b2029167f7\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" Nov 01 16:19:47.286557 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:47.286405 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mmk5\" (UniqueName: \"kubernetes.io/projected/f4b349f5-acb2-42b2-8d90-01b2029167f7-kube-api-access-5mmk5\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg\" (UID: \"f4b349f5-acb2-42b2-8d90-01b2029167f7\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" Nov 01 16:19:47.286557 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:47.286453 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b349f5-acb2-42b2-8d90-01b2029167f7-credential-socket\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg\" (UID: \"f4b349f5-acb2-42b2-8d90-01b2029167f7\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" Nov 01 16:19:47.286557 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:47.286522 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b349f5-acb2-42b2-8d90-01b2029167f7-workload-socket\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg\" (UID: \"f4b349f5-acb2-42b2-8d90-01b2029167f7\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" Nov 01 16:19:47.286731 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:47.286571 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4b349f5-acb2-42b2-8d90-01b2029167f7-istio-data\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg\" (UID: \"f4b349f5-acb2-42b2-8d90-01b2029167f7\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" Nov 01 16:19:47.287151 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:47.287126 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4b349f5-acb2-42b2-8d90-01b2029167f7-istiod-ca-cert\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg\" (UID: \"f4b349f5-acb2-42b2-8d90-01b2029167f7\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" Nov 01 16:19:47.288543 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:47.288524 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4b349f5-acb2-42b2-8d90-01b2029167f7-istio-envoy\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg\" (UID: \"f4b349f5-acb2-42b2-8d90-01b2029167f7\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" Nov 01 16:19:47.289334 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:47.289316 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4b349f5-acb2-42b2-8d90-01b2029167f7-istio-podinfo\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg\" (UID: \"f4b349f5-acb2-42b2-8d90-01b2029167f7\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" Nov 01 16:19:47.294337 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:47.294310 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mmk5\" (UniqueName: \"kubernetes.io/projected/f4b349f5-acb2-42b2-8d90-01b2029167f7-kube-api-access-5mmk5\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg\" (UID: \"f4b349f5-acb2-42b2-8d90-01b2029167f7\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" Nov 01 16:19:47.294570 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:47.294551 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4b349f5-acb2-42b2-8d90-01b2029167f7-istio-token\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg\" (UID: \"f4b349f5-acb2-42b2-8d90-01b2029167f7\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" Nov 01 16:19:47.417187 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:47.417102 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-7225e5-openshift-default-dockercfg-bgkdt\"" Nov 01 16:19:47.425236 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:47.425207 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" Nov 01 16:19:47.475225 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:47.475194 4363 patch_prober.go:29] interesting pod/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:19:47.475479 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:47.475450 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" podUID="387bf77d-04f3-4915-926c-96bdb07374e6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:19:47.715798 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:47.715765 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg"] Nov 01 16:19:47.716877 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:19:47.716849 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b349f5_acb2_42b2_8d90_01b2029167f7.slice/crio-67969328a1c7f2040039b4136efd9a6065a0e8ec3ae5dbe6c9782340c444130a WatchSource:0}: Error finding container 67969328a1c7f2040039b4136efd9a6065a0e8ec3ae5dbe6c9782340c444130a: Status 404 returned error can't find the container with id 67969328a1c7f2040039b4136efd9a6065a0e8ec3ae5dbe6c9782340c444130a Nov 01 16:19:47.720009 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:47.719974 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:19:47.720145 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:47.720077 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:19:47.720145 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:47.720125 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:19:47.740902 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:47.740875 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" event={"ID":"f4b349f5-acb2-42b2-8d90-01b2029167f7","Type":"ContainerStarted","Data":"67969328a1c7f2040039b4136efd9a6065a0e8ec3ae5dbe6c9782340c444130a"} Nov 01 16:19:48.314668 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:19:48.314633 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:19:48.760418 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:48.760390 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" event={"ID":"f4b349f5-acb2-42b2-8d90-01b2029167f7","Type":"ContainerStarted","Data":"db8bd7437ddcdc6617b7e6c3dc9d860fab8ad93befb0669687b273a8d20eed82"} Nov 01 16:19:48.784423 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:48.784370 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" podStartSLOduration=2.784355941 podStartE2EDuration="2.784355941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:19:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:19:48.78122185 +0000 UTC m=+1922.112959500" watchObservedRunningTime="2026-11-01 16:19:48.784355941 +0000 UTC m=+1922.116093591" Nov 01 16:19:49.428453 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:49.428416 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" Nov 01 16:19:49.435351 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:49.435315 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" Nov 01 16:19:49.454410 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:49.454373 4363 scope.go:122] "RemoveContainer" containerID="bc94780c2d6af818ef77a848f09edb05f03b31df70a630ae9e05d5b59175d469" Nov 01 16:19:49.768727 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:49.768695 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" Nov 01 16:19:49.769712 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:49.769690 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" Nov 01 16:19:50.254988 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:50.254961 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2ce08a-69744fddc5-cfpqd" Nov 01 16:19:50.357204 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:50.357175 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b44172c-d6bb-4766-b1da-3343a3b46cc6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b44172c-d6bb-4766-b1da-3343a3b46cc6-plugins-config-volume\") pod \"1b44172c-d6bb-4766-b1da-3343a3b46cc6\" (UID: \"1b44172c-d6bb-4766-b1da-3343a3b46cc6\") " Nov 01 16:19:50.357381 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:50.357245 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b44172c-d6bb-4766-b1da-3343a3b46cc6-kube-api-access-5r7tp\" (UniqueName: \"kubernetes.io/projected/1b44172c-d6bb-4766-b1da-3343a3b46cc6-kube-api-access-5r7tp\") pod \"1b44172c-d6bb-4766-b1da-3343a3b46cc6\" (UID: \"1b44172c-d6bb-4766-b1da-3343a3b46cc6\") " Nov 01 16:19:50.357559 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:50.357537 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b44172c-d6bb-4766-b1da-3343a3b46cc6-plugins-config-volume" pod "1b44172c-d6bb-4766-b1da-3343a3b46cc6" (UID: "1b44172c-d6bb-4766-b1da-3343a3b46cc6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:19:50.357835 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:50.357820 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b44172c-d6bb-4766-b1da-3343a3b46cc6-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:50.359433 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:50.359413 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b44172c-d6bb-4766-b1da-3343a3b46cc6-kube-api-access-5r7tp" pod "1b44172c-d6bb-4766-b1da-3343a3b46cc6" (UID: "1b44172c-d6bb-4766-b1da-3343a3b46cc6"). InnerVolumeSpecName "kube-api-access-5r7tp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:50.459466 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:50.459433 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r7tp\" (UniqueName: \"kubernetes.io/projected/1b44172c-d6bb-4766-b1da-3343a3b46cc6-kube-api-access-5r7tp\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:50.779307 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:50.779217 4363 generic.go:356] "Generic (PLEG): container finished" podID="1b44172c-d6bb-4766-b1da-3343a3b46cc6" containerID="5e11a0c126638284753185ee7abfb5336ef8e421c3331469958393c8d1859cfa" exitCode=0 Nov 01 16:19:50.780237 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:50.780215 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2ce08a-69744fddc5-cfpqd" Nov 01 16:19:50.781147 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:50.781121 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2ce08a-69744fddc5-cfpqd" event={"ID":"1b44172c-d6bb-4766-b1da-3343a3b46cc6","Type":"ContainerDied","Data":"5e11a0c126638284753185ee7abfb5336ef8e421c3331469958393c8d1859cfa"} Nov 01 16:19:50.781337 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:50.781319 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2ce08a-69744fddc5-cfpqd" event={"ID":"1b44172c-d6bb-4766-b1da-3343a3b46cc6","Type":"ContainerDied","Data":"e5c32e592988463703bb6327a13e7dc6543d7caa6e89901e931671f9d6402c6d"} Nov 01 16:19:50.781457 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:50.781346 4363 scope.go:122] "RemoveContainer" containerID="5e11a0c126638284753185ee7abfb5336ef8e421c3331469958393c8d1859cfa" Nov 01 16:19:50.802168 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:50.801706 4363 scope.go:122] "RemoveContainer" containerID="5e11a0c126638284753185ee7abfb5336ef8e421c3331469958393c8d1859cfa" Nov 01 16:19:50.811118 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:19:50.810096 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e11a0c126638284753185ee7abfb5336ef8e421c3331469958393c8d1859cfa\": container with ID starting with 5e11a0c126638284753185ee7abfb5336ef8e421c3331469958393c8d1859cfa not found: ID does not exist" containerID="5e11a0c126638284753185ee7abfb5336ef8e421c3331469958393c8d1859cfa" Nov 01 16:19:50.811118 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:50.810137 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e11a0c126638284753185ee7abfb5336ef8e421c3331469958393c8d1859cfa"} err="failed to get container status \"5e11a0c126638284753185ee7abfb5336ef8e421c3331469958393c8d1859cfa\": rpc error: code = NotFound desc = could not find container \"5e11a0c126638284753185ee7abfb5336ef8e421c3331469958393c8d1859cfa\": container with ID starting with 5e11a0c126638284753185ee7abfb5336ef8e421c3331469958393c8d1859cfa not found: ID does not exist" Nov 01 16:19:50.817978 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:50.817954 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2ce08a-69744fddc5-cfpqd"] Nov 01 16:19:50.822320 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:50.822297 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2ce08a-69744fddc5-cfpqd"] Nov 01 16:19:51.618040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:51.609596 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" Nov 01 16:19:51.696540 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:51.696507 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/387bf77d-04f3-4915-926c-96bdb07374e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/387bf77d-04f3-4915-926c-96bdb07374e6-istio-podinfo\") pod \"387bf77d-04f3-4915-926c-96bdb07374e6\" (UID: \"387bf77d-04f3-4915-926c-96bdb07374e6\") " Nov 01 16:19:51.696766 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:51.696752 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/387bf77d-04f3-4915-926c-96bdb07374e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/387bf77d-04f3-4915-926c-96bdb07374e6-credential-socket\") pod \"387bf77d-04f3-4915-926c-96bdb07374e6\" (UID: \"387bf77d-04f3-4915-926c-96bdb07374e6\") " Nov 01 16:19:51.696884 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:51.696873 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/387bf77d-04f3-4915-926c-96bdb07374e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/387bf77d-04f3-4915-926c-96bdb07374e6-workload-socket\") pod \"387bf77d-04f3-4915-926c-96bdb07374e6\" (UID: \"387bf77d-04f3-4915-926c-96bdb07374e6\") " Nov 01 16:19:51.697158 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:51.697096 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/387bf77d-04f3-4915-926c-96bdb07374e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/387bf77d-04f3-4915-926c-96bdb07374e6-workload-certs\") pod \"387bf77d-04f3-4915-926c-96bdb07374e6\" (UID: \"387bf77d-04f3-4915-926c-96bdb07374e6\") " Nov 01 16:19:51.706175 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:51.704095 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/387bf77d-04f3-4915-926c-96bdb07374e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/387bf77d-04f3-4915-926c-96bdb07374e6-istiod-ca-cert\") pod \"387bf77d-04f3-4915-926c-96bdb07374e6\" (UID: \"387bf77d-04f3-4915-926c-96bdb07374e6\") " Nov 01 16:19:51.706175 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:51.704174 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/387bf77d-04f3-4915-926c-96bdb07374e6-istio-token\" (UniqueName: \"kubernetes.io/projected/387bf77d-04f3-4915-926c-96bdb07374e6-istio-token\") pod \"387bf77d-04f3-4915-926c-96bdb07374e6\" (UID: \"387bf77d-04f3-4915-926c-96bdb07374e6\") " Nov 01 16:19:51.706175 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:51.704314 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/387bf77d-04f3-4915-926c-96bdb07374e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/387bf77d-04f3-4915-926c-96bdb07374e6-istio-envoy\") pod \"387bf77d-04f3-4915-926c-96bdb07374e6\" (UID: \"387bf77d-04f3-4915-926c-96bdb07374e6\") " Nov 01 16:19:51.706175 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:51.704432 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/387bf77d-04f3-4915-926c-96bdb07374e6-kube-api-access-rb6mp\" (UniqueName: \"kubernetes.io/projected/387bf77d-04f3-4915-926c-96bdb07374e6-kube-api-access-rb6mp\") pod \"387bf77d-04f3-4915-926c-96bdb07374e6\" (UID: \"387bf77d-04f3-4915-926c-96bdb07374e6\") " Nov 01 16:19:51.706175 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:51.704478 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/387bf77d-04f3-4915-926c-96bdb07374e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/387bf77d-04f3-4915-926c-96bdb07374e6-istio-data\") pod \"387bf77d-04f3-4915-926c-96bdb07374e6\" (UID: \"387bf77d-04f3-4915-926c-96bdb07374e6\") " Nov 01 16:19:51.706175 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:51.705612 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/387bf77d-04f3-4915-926c-96bdb07374e6-istio-data" pod "387bf77d-04f3-4915-926c-96bdb07374e6" (UID: "387bf77d-04f3-4915-926c-96bdb07374e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:51.708282 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:51.708242 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/387bf77d-04f3-4915-926c-96bdb07374e6-workload-socket" pod "387bf77d-04f3-4915-926c-96bdb07374e6" (UID: "387bf77d-04f3-4915-926c-96bdb07374e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:51.708630 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:51.708594 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/387bf77d-04f3-4915-926c-96bdb07374e6-credential-socket" pod "387bf77d-04f3-4915-926c-96bdb07374e6" (UID: "387bf77d-04f3-4915-926c-96bdb07374e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:51.709869 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:51.709843 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/387bf77d-04f3-4915-926c-96bdb07374e6-workload-certs" pod "387bf77d-04f3-4915-926c-96bdb07374e6" (UID: "387bf77d-04f3-4915-926c-96bdb07374e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:51.710411 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:51.710385 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/387bf77d-04f3-4915-926c-96bdb07374e6-istiod-ca-cert" pod "387bf77d-04f3-4915-926c-96bdb07374e6" (UID: "387bf77d-04f3-4915-926c-96bdb07374e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:19:51.711438 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:51.711412 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/387bf77d-04f3-4915-926c-96bdb07374e6-istio-podinfo" pod "387bf77d-04f3-4915-926c-96bdb07374e6" (UID: "387bf77d-04f3-4915-926c-96bdb07374e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:19:51.716237 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:51.715839 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/387bf77d-04f3-4915-926c-96bdb07374e6-istio-token" pod "387bf77d-04f3-4915-926c-96bdb07374e6" (UID: "387bf77d-04f3-4915-926c-96bdb07374e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:51.716237 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:51.716043 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/387bf77d-04f3-4915-926c-96bdb07374e6-kube-api-access-rb6mp" pod "387bf77d-04f3-4915-926c-96bdb07374e6" (UID: "387bf77d-04f3-4915-926c-96bdb07374e6"). InnerVolumeSpecName "kube-api-access-rb6mp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:51.716237 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:51.716185 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/387bf77d-04f3-4915-926c-96bdb07374e6-istio-envoy" pod "387bf77d-04f3-4915-926c-96bdb07374e6" (UID: "387bf77d-04f3-4915-926c-96bdb07374e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:51.789419 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:51.789332 4363 generic.go:356] "Generic (PLEG): container finished" podID="387bf77d-04f3-4915-926c-96bdb07374e6" containerID="155a7788883e55c3998cd540714dfbe616961a4b54eed9ac14bf144d0a501bbf" exitCode=0 Nov 01 16:19:51.789872 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:51.789447 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" event={"ID":"387bf77d-04f3-4915-926c-96bdb07374e6","Type":"ContainerDied","Data":"155a7788883e55c3998cd540714dfbe616961a4b54eed9ac14bf144d0a501bbf"} Nov 01 16:19:51.789872 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:51.789507 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" event={"ID":"387bf77d-04f3-4915-926c-96bdb07374e6","Type":"ContainerDied","Data":"62aba302bd119edec1826677b4a6e631e1992e91d8061d7ace3f7148ff9ed0a4"} Nov 01 16:19:51.789872 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:51.789537 4363 scope.go:122] "RemoveContainer" containerID="155a7788883e55c3998cd540714dfbe616961a4b54eed9ac14bf144d0a501bbf" Nov 01 16:19:51.790084 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:51.790052 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246" Nov 01 16:19:51.808421 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:51.808352 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/387bf77d-04f3-4915-926c-96bdb07374e6-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:51.808421 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:51.808383 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb6mp\" (UniqueName: \"kubernetes.io/projected/387bf77d-04f3-4915-926c-96bdb07374e6-kube-api-access-rb6mp\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:51.808421 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:51.808411 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/387bf77d-04f3-4915-926c-96bdb07374e6-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:51.808421 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:51.808424 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/387bf77d-04f3-4915-926c-96bdb07374e6-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:51.808773 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:51.808438 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/387bf77d-04f3-4915-926c-96bdb07374e6-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:51.808773 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:51.808451 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/387bf77d-04f3-4915-926c-96bdb07374e6-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:51.808773 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:51.808466 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/387bf77d-04f3-4915-926c-96bdb07374e6-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:51.808773 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:51.808480 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/387bf77d-04f3-4915-926c-96bdb07374e6-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:51.808773 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:51.808492 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/387bf77d-04f3-4915-926c-96bdb07374e6-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:51.811390 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:51.811358 4363 scope.go:122] "RemoveContainer" containerID="155a7788883e55c3998cd540714dfbe616961a4b54eed9ac14bf144d0a501bbf" Nov 01 16:19:51.811773 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:19:51.811736 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"155a7788883e55c3998cd540714dfbe616961a4b54eed9ac14bf144d0a501bbf\": container with ID starting with 155a7788883e55c3998cd540714dfbe616961a4b54eed9ac14bf144d0a501bbf not found: ID does not exist" containerID="155a7788883e55c3998cd540714dfbe616961a4b54eed9ac14bf144d0a501bbf" Nov 01 16:19:51.811894 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:51.811779 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"155a7788883e55c3998cd540714dfbe616961a4b54eed9ac14bf144d0a501bbf"} err="failed to get container status \"155a7788883e55c3998cd540714dfbe616961a4b54eed9ac14bf144d0a501bbf\": rpc error: code = NotFound desc = could not find container \"155a7788883e55c3998cd540714dfbe616961a4b54eed9ac14bf144d0a501bbf\": container with ID starting with 155a7788883e55c3998cd540714dfbe616961a4b54eed9ac14bf144d0a501bbf not found: ID does not exist" Nov 01 16:19:51.827782 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:51.827742 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246"] Nov 01 16:19:51.832189 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:51.832167 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-hs246"] Nov 01 16:19:52.211193 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:52.211163 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7225e5-db455946c-psv2h"] Nov 01 16:19:52.212104 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:52.212084 4363 cpu_manager.go:414] "containerMap: removing container" podUID="387bf77d-04f3-4915-926c-96bdb07374e6" containerName="istio-proxy" Nov 01 16:19:52.212104 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:52.212103 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="387bf77d-04f3-4915-926c-96bdb07374e6" containerName="istio-proxy" Nov 01 16:19:52.212285 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:52.212123 4363 cpu_manager.go:414] "containerMap: removing container" podUID="1b44172c-d6bb-4766-b1da-3343a3b46cc6" containerName="payload-pre-processing" Nov 01 16:19:52.212285 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:52.212129 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b44172c-d6bb-4766-b1da-3343a3b46cc6" containerName="payload-pre-processing" Nov 01 16:19:52.212285 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:52.212252 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="387bf77d-04f3-4915-926c-96bdb07374e6" containerName="istio-proxy" Nov 01 16:19:52.212285 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:52.212263 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b44172c-d6bb-4766-b1da-3343a3b46cc6" containerName="payload-pre-processing" Nov 01 16:19:52.217777 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:52.217753 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7225e5-db455946c-psv2h" Nov 01 16:19:52.223152 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:52.222829 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-7225e5\"" Nov 01 16:19:52.226097 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:52.223748 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-7225e5-dockercfg-jmr9g\"" Nov 01 16:19:52.226996 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:52.226973 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7225e5-db455946c-psv2h"] Nov 01 16:19:52.262106 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:52.262076 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7225e5-d8c8dcb65-ts6np"] Nov 01 16:19:52.270570 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:52.267648 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7225e5-d8c8dcb65-ts6np" Nov 01 16:19:52.299443 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:52.290407 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7225e5-d8c8dcb65-ts6np"] Nov 01 16:19:52.381040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:52.319570 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlnqz\" (UniqueName: \"kubernetes.io/projected/bfe20137-e20c-49c2-a99a-8cc8d517003a-kube-api-access-rlnqz\") pod \"payload-pre-processing-e2e-shared-a-w2-7225e5-db455946c-psv2h\" (UID: \"bfe20137-e20c-49c2-a99a-8cc8d517003a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7225e5-db455946c-psv2h" Nov 01 16:19:52.381040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:52.319688 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe20137-e20c-49c2-a99a-8cc8d517003a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-7225e5-db455946c-psv2h\" (UID: \"bfe20137-e20c-49c2-a99a-8cc8d517003a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7225e5-db455946c-psv2h" Nov 01 16:19:52.381040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:52.319725 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3899829-a9d3-470c-8d39-befbe630cb96-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-7225e5-d8c8dcb65-ts6np\" (UID: \"f3899829-a9d3-470c-8d39-befbe630cb96\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7225e5-d8c8dcb65-ts6np" Nov 01 16:19:52.381040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:52.319832 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg4rx\" (UniqueName: \"kubernetes.io/projected/f3899829-a9d3-470c-8d39-befbe630cb96-kube-api-access-sg4rx\") pod \"payload-processing-e2e-shared-a-w2-7225e5-d8c8dcb65-ts6np\" (UID: \"f3899829-a9d3-470c-8d39-befbe630cb96\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7225e5-d8c8dcb65-ts6np" Nov 01 16:19:52.545087 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:52.529577 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe20137-e20c-49c2-a99a-8cc8d517003a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-7225e5-db455946c-psv2h\" (UID: \"bfe20137-e20c-49c2-a99a-8cc8d517003a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7225e5-db455946c-psv2h" Nov 01 16:19:52.545087 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:52.529647 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3899829-a9d3-470c-8d39-befbe630cb96-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-7225e5-d8c8dcb65-ts6np\" (UID: \"f3899829-a9d3-470c-8d39-befbe630cb96\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7225e5-d8c8dcb65-ts6np" Nov 01 16:19:52.545087 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:52.529790 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg4rx\" (UniqueName: \"kubernetes.io/projected/f3899829-a9d3-470c-8d39-befbe630cb96-kube-api-access-sg4rx\") pod \"payload-processing-e2e-shared-a-w2-7225e5-d8c8dcb65-ts6np\" (UID: \"f3899829-a9d3-470c-8d39-befbe630cb96\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7225e5-d8c8dcb65-ts6np" Nov 01 16:19:52.545087 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:52.530150 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlnqz\" (UniqueName: \"kubernetes.io/projected/bfe20137-e20c-49c2-a99a-8cc8d517003a-kube-api-access-rlnqz\") pod \"payload-pre-processing-e2e-shared-a-w2-7225e5-db455946c-psv2h\" (UID: \"bfe20137-e20c-49c2-a99a-8cc8d517003a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7225e5-db455946c-psv2h" Nov 01 16:19:52.545087 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:52.542190 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe20137-e20c-49c2-a99a-8cc8d517003a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-7225e5-db455946c-psv2h\" (UID: \"bfe20137-e20c-49c2-a99a-8cc8d517003a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7225e5-db455946c-psv2h" Nov 01 16:19:52.551620 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:52.546926 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3899829-a9d3-470c-8d39-befbe630cb96-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-7225e5-d8c8dcb65-ts6np\" (UID: \"f3899829-a9d3-470c-8d39-befbe630cb96\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7225e5-d8c8dcb65-ts6np" Nov 01 16:19:52.641815 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:52.641778 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlnqz\" (UniqueName: \"kubernetes.io/projected/bfe20137-e20c-49c2-a99a-8cc8d517003a-kube-api-access-rlnqz\") pod \"payload-pre-processing-e2e-shared-a-w2-7225e5-db455946c-psv2h\" (UID: \"bfe20137-e20c-49c2-a99a-8cc8d517003a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7225e5-db455946c-psv2h" Nov 01 16:19:52.643920 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:52.643890 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg4rx\" (UniqueName: \"kubernetes.io/projected/f3899829-a9d3-470c-8d39-befbe630cb96-kube-api-access-sg4rx\") pod \"payload-processing-e2e-shared-a-w2-7225e5-d8c8dcb65-ts6np\" (UID: \"f3899829-a9d3-470c-8d39-befbe630cb96\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7225e5-d8c8dcb65-ts6np" Nov 01 16:19:52.803304 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:52.803211 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b44172c-d6bb-4766-b1da-3343a3b46cc6" path="/var/lib/kubelet/pods/1b44172c-d6bb-4766-b1da-3343a3b46cc6/volumes" Nov 01 16:19:52.804050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:52.803995 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="387bf77d-04f3-4915-926c-96bdb07374e6" path="/var/lib/kubelet/pods/387bf77d-04f3-4915-926c-96bdb07374e6/volumes" Nov 01 16:19:52.848260 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:52.848229 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7225e5-db455946c-psv2h" Nov 01 16:19:52.878898 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:52.878862 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7225e5-d8c8dcb65-ts6np" Nov 01 16:19:53.154558 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:19:53.154521 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfe20137_e20c_49c2_a99a_8cc8d517003a.slice/crio-c6f4abdf0744a785578a003cb875ebf56eacbd579a6a8f129a571734b046f318 WatchSource:0}: Error finding container c6f4abdf0744a785578a003cb875ebf56eacbd579a6a8f129a571734b046f318: Status 404 returned error can't find the container with id c6f4abdf0744a785578a003cb875ebf56eacbd579a6a8f129a571734b046f318 Nov 01 16:19:53.156069 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:53.156039 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7225e5-db455946c-psv2h"] Nov 01 16:19:53.205072 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:53.205042 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7225e5-d8c8dcb65-ts6np"] Nov 01 16:19:53.815118 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:53.815077 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7225e5-db455946c-psv2h" event={"ID":"bfe20137-e20c-49c2-a99a-8cc8d517003a","Type":"ContainerStarted","Data":"0f8fdaa78b6a45c03910823d799944cc8ee1aaaecbee652ec1443ff4a06291fc"} Nov 01 16:19:53.815616 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:53.815145 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7225e5-db455946c-psv2h" event={"ID":"bfe20137-e20c-49c2-a99a-8cc8d517003a","Type":"ContainerStarted","Data":"c6f4abdf0744a785578a003cb875ebf56eacbd579a6a8f129a571734b046f318"} Nov 01 16:19:53.815616 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:53.815173 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7225e5-db455946c-psv2h" Nov 01 16:19:53.816875 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:53.816851 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7225e5-d8c8dcb65-ts6np" event={"ID":"f3899829-a9d3-470c-8d39-befbe630cb96","Type":"ContainerStarted","Data":"a18883ebb081f95e165615cf0f1830fd5fb5d86d36a3d8b9cdccb209921558d0"} Nov 01 16:19:53.817003 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:53.816894 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7225e5-d8c8dcb65-ts6np" event={"ID":"f3899829-a9d3-470c-8d39-befbe630cb96","Type":"ContainerStarted","Data":"3d90fc0a9a1e18a3699d38e329105518445506801f6c8d5c2e434fc2aaf73864"} Nov 01 16:19:53.817003 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:53.816978 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7225e5-d8c8dcb65-ts6np" Nov 01 16:19:53.834295 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:53.834258 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7225e5-db455946c-psv2h" podStartSLOduration=1.8342449790000002 podStartE2EDuration="1.834244979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:19:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:19:53.829921076 +0000 UTC m=+1927.161658726" watchObservedRunningTime="2026-11-01 16:19:53.834244979 +0000 UTC m=+1927.165982662" Nov 01 16:19:53.848130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:53.848087 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7225e5-d8c8dcb65-ts6np" podStartSLOduration=1.8480756550000002 podStartE2EDuration="1.848075655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:19:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:19:53.845130122 +0000 UTC m=+1927.176867771" watchObservedRunningTime="2026-11-01 16:19:53.848075655 +0000 UTC m=+1927.179813327" Nov 01 16:19:54.313266 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:54.313224 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-784444-79c887957f-gsjc5" podUID="f796692a-c6ab-48b9-a227-00e8af4f24df" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:19:55.335609 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:55.335575 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-784444-55c46475-d78lt" podUID="59ebabe0-34d2-4102-86de-52bb4d6d04e2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:19:55.817215 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:55.817179 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-99afca-5bc4ccf5cc-qs8gg"] Nov 01 16:19:55.829679 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:55.829650 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-99afca-6df5cd4d75-kqpvz"] Nov 01 16:19:56.425854 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:56.425828 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-784444-79c887957f-gsjc5" Nov 01 16:19:56.436860 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:56.436831 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f796692a-c6ab-48b9-a227-00e8af4f24df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f796692a-c6ab-48b9-a227-00e8af4f24df-plugins-config-volume\") pod \"f796692a-c6ab-48b9-a227-00e8af4f24df\" (UID: \"f796692a-c6ab-48b9-a227-00e8af4f24df\") " Nov 01 16:19:56.437203 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:56.437149 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f796692a-c6ab-48b9-a227-00e8af4f24df-kube-api-access-96mfm\" (UniqueName: \"kubernetes.io/projected/f796692a-c6ab-48b9-a227-00e8af4f24df-kube-api-access-96mfm\") pod \"f796692a-c6ab-48b9-a227-00e8af4f24df\" (UID: \"f796692a-c6ab-48b9-a227-00e8af4f24df\") " Nov 01 16:19:56.437336 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:56.437298 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f796692a-c6ab-48b9-a227-00e8af4f24df-plugins-config-volume" pod "f796692a-c6ab-48b9-a227-00e8af4f24df" (UID: "f796692a-c6ab-48b9-a227-00e8af4f24df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:19:56.438266 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:56.438244 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f796692a-c6ab-48b9-a227-00e8af4f24df-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:56.440073 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:56.440038 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f796692a-c6ab-48b9-a227-00e8af4f24df-kube-api-access-96mfm" pod "f796692a-c6ab-48b9-a227-00e8af4f24df" (UID: "f796692a-c6ab-48b9-a227-00e8af4f24df"). InnerVolumeSpecName "kube-api-access-96mfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:56.477934 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:56.477909 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-784444-55c46475-d78lt" Nov 01 16:19:56.543204 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:56.543153 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96mfm\" (UniqueName: \"kubernetes.io/projected/f796692a-c6ab-48b9-a227-00e8af4f24df-kube-api-access-96mfm\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:56.644618 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:56.644532 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59ebabe0-34d2-4102-86de-52bb4d6d04e2-kube-api-access-8bkkq\" (UniqueName: \"kubernetes.io/projected/59ebabe0-34d2-4102-86de-52bb4d6d04e2-kube-api-access-8bkkq\") pod \"59ebabe0-34d2-4102-86de-52bb4d6d04e2\" (UID: \"59ebabe0-34d2-4102-86de-52bb4d6d04e2\") " Nov 01 16:19:56.644778 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:56.644620 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59ebabe0-34d2-4102-86de-52bb4d6d04e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ebabe0-34d2-4102-86de-52bb4d6d04e2-plugins-config-volume\") pod \"59ebabe0-34d2-4102-86de-52bb4d6d04e2\" (UID: \"59ebabe0-34d2-4102-86de-52bb4d6d04e2\") " Nov 01 16:19:56.645238 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:56.645215 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59ebabe0-34d2-4102-86de-52bb4d6d04e2-plugins-config-volume" pod "59ebabe0-34d2-4102-86de-52bb4d6d04e2" (UID: "59ebabe0-34d2-4102-86de-52bb4d6d04e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:19:56.647196 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:56.647156 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59ebabe0-34d2-4102-86de-52bb4d6d04e2-kube-api-access-8bkkq" pod "59ebabe0-34d2-4102-86de-52bb4d6d04e2" (UID: "59ebabe0-34d2-4102-86de-52bb4d6d04e2"). InnerVolumeSpecName "kube-api-access-8bkkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:56.746339 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:56.746305 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bkkq\" (UniqueName: \"kubernetes.io/projected/59ebabe0-34d2-4102-86de-52bb4d6d04e2-kube-api-access-8bkkq\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:56.746339 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:56.746335 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ebabe0-34d2-4102-86de-52bb4d6d04e2-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:56.900915 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:56.900833 4363 generic.go:356] "Generic (PLEG): container finished" podID="59ebabe0-34d2-4102-86de-52bb4d6d04e2" containerID="0ade98c8ab5aecb81daf070285e586f52de6816ce0969851c7eed7e06c8cff1e" exitCode=0 Nov 01 16:19:56.900915 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:56.900875 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-784444-55c46475-d78lt" event={"ID":"59ebabe0-34d2-4102-86de-52bb4d6d04e2","Type":"ContainerDied","Data":"0ade98c8ab5aecb81daf070285e586f52de6816ce0969851c7eed7e06c8cff1e"} Nov 01 16:19:56.901171 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:56.900890 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-784444-55c46475-d78lt" Nov 01 16:19:56.901171 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:56.900946 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-784444-55c46475-d78lt" event={"ID":"59ebabe0-34d2-4102-86de-52bb4d6d04e2","Type":"ContainerDied","Data":"26a8fe815d9f28ea406fa7d5f3a8ed9782d2d58c0892d282833d7859ad67545b"} Nov 01 16:19:56.901171 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:56.900978 4363 scope.go:122] "RemoveContainer" containerID="0ade98c8ab5aecb81daf070285e586f52de6816ce0969851c7eed7e06c8cff1e" Nov 01 16:19:56.903214 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:56.902966 4363 generic.go:356] "Generic (PLEG): container finished" podID="f796692a-c6ab-48b9-a227-00e8af4f24df" containerID="10dca15fcaf3a9014dfc285d4a55df18db1ca6cfe936347bfb3c4698f038915c" exitCode=0 Nov 01 16:19:56.903214 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:56.903097 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-784444-79c887957f-gsjc5" event={"ID":"f796692a-c6ab-48b9-a227-00e8af4f24df","Type":"ContainerDied","Data":"10dca15fcaf3a9014dfc285d4a55df18db1ca6cfe936347bfb3c4698f038915c"} Nov 01 16:19:56.903214 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:56.903185 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-784444-79c887957f-gsjc5" event={"ID":"f796692a-c6ab-48b9-a227-00e8af4f24df","Type":"ContainerDied","Data":"29fcdac271afd3010ef9d7b196fef903a3ba58fff531ebd753b8bd92b0bd5245"} Nov 01 16:19:56.903492 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:56.903220 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-784444-79c887957f-gsjc5" Nov 01 16:19:56.918535 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:56.918515 4363 scope.go:122] "RemoveContainer" containerID="0ade98c8ab5aecb81daf070285e586f52de6816ce0969851c7eed7e06c8cff1e" Nov 01 16:19:56.919210 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:19:56.919165 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ade98c8ab5aecb81daf070285e586f52de6816ce0969851c7eed7e06c8cff1e\": container with ID starting with 0ade98c8ab5aecb81daf070285e586f52de6816ce0969851c7eed7e06c8cff1e not found: ID does not exist" containerID="0ade98c8ab5aecb81daf070285e586f52de6816ce0969851c7eed7e06c8cff1e" Nov 01 16:19:56.919380 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:56.919205 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ade98c8ab5aecb81daf070285e586f52de6816ce0969851c7eed7e06c8cff1e"} err="failed to get container status \"0ade98c8ab5aecb81daf070285e586f52de6816ce0969851c7eed7e06c8cff1e\": rpc error: code = NotFound desc = could not find container \"0ade98c8ab5aecb81daf070285e586f52de6816ce0969851c7eed7e06c8cff1e\": container with ID starting with 0ade98c8ab5aecb81daf070285e586f52de6816ce0969851c7eed7e06c8cff1e not found: ID does not exist" Nov 01 16:19:56.919380 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:56.919227 4363 scope.go:122] "RemoveContainer" containerID="10dca15fcaf3a9014dfc285d4a55df18db1ca6cfe936347bfb3c4698f038915c" Nov 01 16:19:56.927921 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:56.927901 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-784444-55c46475-d78lt"] Nov 01 16:19:56.933868 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:56.932945 4363 scope.go:122] "RemoveContainer" containerID="10dca15fcaf3a9014dfc285d4a55df18db1ca6cfe936347bfb3c4698f038915c" Nov 01 16:19:56.934692 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:56.934670 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-784444-55c46475-d78lt"] Nov 01 16:19:56.934877 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:19:56.934850 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10dca15fcaf3a9014dfc285d4a55df18db1ca6cfe936347bfb3c4698f038915c\": container with ID starting with 10dca15fcaf3a9014dfc285d4a55df18db1ca6cfe936347bfb3c4698f038915c not found: ID does not exist" containerID="10dca15fcaf3a9014dfc285d4a55df18db1ca6cfe936347bfb3c4698f038915c" Nov 01 16:19:56.934967 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:56.934889 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10dca15fcaf3a9014dfc285d4a55df18db1ca6cfe936347bfb3c4698f038915c"} err="failed to get container status \"10dca15fcaf3a9014dfc285d4a55df18db1ca6cfe936347bfb3c4698f038915c\": rpc error: code = NotFound desc = could not find container \"10dca15fcaf3a9014dfc285d4a55df18db1ca6cfe936347bfb3c4698f038915c\": container with ID starting with 10dca15fcaf3a9014dfc285d4a55df18db1ca6cfe936347bfb3c4698f038915c not found: ID does not exist" Nov 01 16:19:56.940032 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:56.940001 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-784444-79c887957f-gsjc5"] Nov 01 16:19:56.943455 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:56.943436 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-784444-79c887957f-gsjc5"] Nov 01 16:19:58.804829 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:58.804798 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59ebabe0-34d2-4102-86de-52bb4d6d04e2" path="/var/lib/kubelet/pods/59ebabe0-34d2-4102-86de-52bb4d6d04e2/volumes" Nov 01 16:19:58.805414 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:19:58.805396 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f796692a-c6ab-48b9-a227-00e8af4f24df" path="/var/lib/kubelet/pods/f796692a-c6ab-48b9-a227-00e8af4f24df/volumes" Nov 01 16:20:00.823093 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:00.818091 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-99afca-5bc4ccf5cc-qs8gg" podUID="15978381-170d-44fd-bfe3-725b8e60060d" containerName="payload-processing" containerID="cri-o://b02783d7c63304189e747bc4fe3c899f80c592dd857f7209b748bfb4f317f784" gracePeriod=25 Nov 01 16:20:00.830334 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:00.830192 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99afca-6df5cd4d75-kqpvz" podUID="cdd9e45d-cd61-4a97-9d05-839b5cb4698b" containerName="payload-pre-processing" containerID="cri-o://69f66ffc632eed9f5f72736c3230bb932dcc7cb07b8adacc8b51062186bb88b8" gracePeriod=25 Nov 01 16:20:00.987540 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:00.987372 4363 generic.go:356] "Generic (PLEG): container finished" podID="15978381-170d-44fd-bfe3-725b8e60060d" containerID="b02783d7c63304189e747bc4fe3c899f80c592dd857f7209b748bfb4f317f784" exitCode=0 Nov 01 16:20:00.987748 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:00.987545 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-99afca-5bc4ccf5cc-qs8gg" event={"ID":"15978381-170d-44fd-bfe3-725b8e60060d","Type":"ContainerDied","Data":"b02783d7c63304189e747bc4fe3c899f80c592dd857f7209b748bfb4f317f784"} Nov 01 16:20:01.142575 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:01.142546 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-99afca-5bc4ccf5cc-qs8gg" Nov 01 16:20:01.241421 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:01.241389 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15978381-170d-44fd-bfe3-725b8e60060d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15978381-170d-44fd-bfe3-725b8e60060d-plugins-config-volume\") pod \"15978381-170d-44fd-bfe3-725b8e60060d\" (UID: \"15978381-170d-44fd-bfe3-725b8e60060d\") " Nov 01 16:20:01.241592 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:01.241581 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15978381-170d-44fd-bfe3-725b8e60060d-kube-api-access-kz4rw\" (UniqueName: \"kubernetes.io/projected/15978381-170d-44fd-bfe3-725b8e60060d-kube-api-access-kz4rw\") pod \"15978381-170d-44fd-bfe3-725b8e60060d\" (UID: \"15978381-170d-44fd-bfe3-725b8e60060d\") " Nov 01 16:20:01.241773 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:01.241749 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15978381-170d-44fd-bfe3-725b8e60060d-plugins-config-volume" pod "15978381-170d-44fd-bfe3-725b8e60060d" (UID: "15978381-170d-44fd-bfe3-725b8e60060d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:20:01.242144 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:01.242114 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15978381-170d-44fd-bfe3-725b8e60060d-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:01.243752 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:01.243728 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15978381-170d-44fd-bfe3-725b8e60060d-kube-api-access-kz4rw" pod "15978381-170d-44fd-bfe3-725b8e60060d" (UID: "15978381-170d-44fd-bfe3-725b8e60060d"). InnerVolumeSpecName "kube-api-access-kz4rw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:20:01.343696 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:01.343665 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz4rw\" (UniqueName: \"kubernetes.io/projected/15978381-170d-44fd-bfe3-725b8e60060d-kube-api-access-kz4rw\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:01.997773 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:01.997741 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-99afca-5bc4ccf5cc-qs8gg" Nov 01 16:20:01.998301 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:01.997783 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-99afca-5bc4ccf5cc-qs8gg" event={"ID":"15978381-170d-44fd-bfe3-725b8e60060d","Type":"ContainerDied","Data":"8508a05f7958c7bf0bba748eaf96c5fb543807abc53260bed444a30fe6b15fec"} Nov 01 16:20:01.998301 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:01.997853 4363 scope.go:122] "RemoveContainer" containerID="b02783d7c63304189e747bc4fe3c899f80c592dd857f7209b748bfb4f317f784" Nov 01 16:20:02.027550 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:02.027510 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-99afca-5bc4ccf5cc-qs8gg"] Nov 01 16:20:02.031683 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:02.031659 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-99afca-5bc4ccf5cc-qs8gg"] Nov 01 16:20:02.802123 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:02.802090 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15978381-170d-44fd-bfe3-725b8e60060d" path="/var/lib/kubelet/pods/15978381-170d-44fd-bfe3-725b8e60060d/volumes" Nov 01 16:20:04.917041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:04.896323 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7225e5-d8c8dcb65-ts6np" Nov 01 16:20:04.917041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:04.913915 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7225e5-db455946c-psv2h" Nov 01 16:20:05.801573 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:05.801534 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99afca-6df5cd4d75-kqpvz" podUID="cdd9e45d-cd61-4a97-9d05-839b5cb4698b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:20:08.036758 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:08.036718 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7225e5-646b876456-9wqwp"] Nov 01 16:20:08.037989 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:08.037963 4363 cpu_manager.go:414] "containerMap: removing container" podUID="f796692a-c6ab-48b9-a227-00e8af4f24df" containerName="payload-pre-processing" Nov 01 16:20:08.038126 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:08.037990 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f796692a-c6ab-48b9-a227-00e8af4f24df" containerName="payload-pre-processing" Nov 01 16:20:08.038126 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:08.038076 4363 cpu_manager.go:414] "containerMap: removing container" podUID="15978381-170d-44fd-bfe3-725b8e60060d" containerName="payload-processing" Nov 01 16:20:08.038126 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:08.038085 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15978381-170d-44fd-bfe3-725b8e60060d" containerName="payload-processing" Nov 01 16:20:08.038126 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:08.038119 4363 cpu_manager.go:414] "containerMap: removing container" podUID="59ebabe0-34d2-4102-86de-52bb4d6d04e2" containerName="payload-processing" Nov 01 16:20:08.038126 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:08.038127 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59ebabe0-34d2-4102-86de-52bb4d6d04e2" containerName="payload-processing" Nov 01 16:20:08.038407 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:08.038258 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="f796692a-c6ab-48b9-a227-00e8af4f24df" containerName="payload-pre-processing" Nov 01 16:20:08.038407 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:08.038272 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="15978381-170d-44fd-bfe3-725b8e60060d" containerName="payload-processing" Nov 01 16:20:08.038407 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:08.038283 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="59ebabe0-34d2-4102-86de-52bb4d6d04e2" containerName="payload-processing" Nov 01 16:20:08.046192 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:08.046160 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7225e5-646b876456-9wqwp" Nov 01 16:20:08.052828 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:08.052795 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-7225e5-serving-cert\"" Nov 01 16:20:08.071134 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:08.071107 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7225e5-646b876456-9wqwp"] Nov 01 16:20:08.140069 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:08.140028 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/033f03c6-c73b-45e7-8567-fdb94c4e5b8d-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-7225e5-646b876456-9wqwp\" (UID: \"033f03c6-c73b-45e7-8567-fdb94c4e5b8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7225e5-646b876456-9wqwp" Nov 01 16:20:08.140514 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:08.140482 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/033f03c6-c73b-45e7-8567-fdb94c4e5b8d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-7225e5-646b876456-9wqwp\" (UID: \"033f03c6-c73b-45e7-8567-fdb94c4e5b8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7225e5-646b876456-9wqwp" Nov 01 16:20:08.140726 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:08.140706 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q29nn\" (UniqueName: \"kubernetes.io/projected/033f03c6-c73b-45e7-8567-fdb94c4e5b8d-kube-api-access-q29nn\") pod \"maas-api-e2e-shared-a-w2-7225e5-646b876456-9wqwp\" (UID: \"033f03c6-c73b-45e7-8567-fdb94c4e5b8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7225e5-646b876456-9wqwp" Nov 01 16:20:08.140771 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:08.140760 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/033f03c6-c73b-45e7-8567-fdb94c4e5b8d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-7225e5-646b876456-9wqwp\" (UID: \"033f03c6-c73b-45e7-8567-fdb94c4e5b8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7225e5-646b876456-9wqwp" Nov 01 16:20:08.242856 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:08.242821 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/033f03c6-c73b-45e7-8567-fdb94c4e5b8d-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-7225e5-646b876456-9wqwp\" (UID: \"033f03c6-c73b-45e7-8567-fdb94c4e5b8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7225e5-646b876456-9wqwp" Nov 01 16:20:08.243080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:08.242903 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/033f03c6-c73b-45e7-8567-fdb94c4e5b8d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-7225e5-646b876456-9wqwp\" (UID: \"033f03c6-c73b-45e7-8567-fdb94c4e5b8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7225e5-646b876456-9wqwp" Nov 01 16:20:08.243080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:08.243041 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q29nn\" (UniqueName: \"kubernetes.io/projected/033f03c6-c73b-45e7-8567-fdb94c4e5b8d-kube-api-access-q29nn\") pod \"maas-api-e2e-shared-a-w2-7225e5-646b876456-9wqwp\" (UID: \"033f03c6-c73b-45e7-8567-fdb94c4e5b8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7225e5-646b876456-9wqwp" Nov 01 16:20:08.243080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:08.243070 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/033f03c6-c73b-45e7-8567-fdb94c4e5b8d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-7225e5-646b876456-9wqwp\" (UID: \"033f03c6-c73b-45e7-8567-fdb94c4e5b8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7225e5-646b876456-9wqwp" Nov 01 16:20:08.243666 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:08.243634 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/033f03c6-c73b-45e7-8567-fdb94c4e5b8d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-7225e5-646b876456-9wqwp\" (UID: \"033f03c6-c73b-45e7-8567-fdb94c4e5b8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7225e5-646b876456-9wqwp" Nov 01 16:20:08.245596 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:08.245564 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/033f03c6-c73b-45e7-8567-fdb94c4e5b8d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-7225e5-646b876456-9wqwp\" (UID: \"033f03c6-c73b-45e7-8567-fdb94c4e5b8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7225e5-646b876456-9wqwp" Nov 01 16:20:08.245698 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:08.245632 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/033f03c6-c73b-45e7-8567-fdb94c4e5b8d-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-7225e5-646b876456-9wqwp\" (UID: \"033f03c6-c73b-45e7-8567-fdb94c4e5b8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7225e5-646b876456-9wqwp" Nov 01 16:20:08.251549 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:08.251523 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q29nn\" (UniqueName: \"kubernetes.io/projected/033f03c6-c73b-45e7-8567-fdb94c4e5b8d-kube-api-access-q29nn\") pod \"maas-api-e2e-shared-a-w2-7225e5-646b876456-9wqwp\" (UID: \"033f03c6-c73b-45e7-8567-fdb94c4e5b8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7225e5-646b876456-9wqwp" Nov 01 16:20:08.376713 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:08.376680 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7225e5-646b876456-9wqwp" Nov 01 16:20:08.785359 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:20:08.781271 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod033f03c6_c73b_45e7_8567_fdb94c4e5b8d.slice/crio-91c26d3e03f7bee7741266bae6b5c75674a7bef0af85c17243f93410187e75e2 WatchSource:0}: Error finding container 91c26d3e03f7bee7741266bae6b5c75674a7bef0af85c17243f93410187e75e2: Status 404 returned error can't find the container with id 91c26d3e03f7bee7741266bae6b5c75674a7bef0af85c17243f93410187e75e2 Nov 01 16:20:08.785359 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:08.781767 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7225e5-646b876456-9wqwp"] Nov 01 16:20:09.086804 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:09.086725 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7225e5-646b876456-9wqwp" event={"ID":"033f03c6-c73b-45e7-8567-fdb94c4e5b8d","Type":"ContainerStarted","Data":"91c26d3e03f7bee7741266bae6b5c75674a7bef0af85c17243f93410187e75e2"} Nov 01 16:20:10.096590 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:10.096557 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7225e5-646b876456-9wqwp" event={"ID":"033f03c6-c73b-45e7-8567-fdb94c4e5b8d","Type":"ContainerStarted","Data":"c822381d2995d721c3155ff3dd17bd4fa1c36a9234e073fe7042b14394e1d57a"} Nov 01 16:20:10.097073 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:10.096636 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7225e5-646b876456-9wqwp" Nov 01 16:20:10.111428 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:10.111378 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7225e5-646b876456-9wqwp" podStartSLOduration=1.65150131 podStartE2EDuration="2.111364204s" totalImagesPullingTime="459.862894ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:20:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:20:10.110325536 +0000 UTC m=+1943.442063186" watchObservedRunningTime="2026-11-01 16:20:10.111364204 +0000 UTC m=+1943.443101984" Nov 01 16:20:11.586438 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:11.586404 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn"] Nov 01 16:20:11.586929 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:11.586718 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" podUID="b6716c2c-b539-4e3c-ac09-7f2d1426af74" containerName="istio-proxy" containerID="cri-o://992c76fb0b84fab448f1f2f44a73f0b15590dd6a7359a3a7dd6c038e52feacce" gracePeriod=30 Nov 01 16:20:11.595265 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:11.595239 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv"] Nov 01 16:20:11.595647 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:11.595619 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" podUID="768d7927-13bd-4fe9-984e-c5ac56bc4fdb" containerName="istio-proxy" containerID="cri-o://2e2c499dc24759b887be8c6339f77a616ee4e1ef1bc78d5eba9416ecbc58fe5d" gracePeriod=30 Nov 01 16:20:12.644041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:12.643258 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w5-383fde/auth-test-model-383fde-kserve-57c55ccb8f-rqp6x"] Nov 01 16:20:12.650399 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:12.650371 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-383fde/auth-test-model-383fde-kserve-57c55ccb8f-rqp6x" Nov 01 16:20:12.657768 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:12.657662 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-383fde\"/\"default-dockercfg-55w5d\"" Nov 01 16:20:12.657768 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:12.657689 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-383fde\"/\"kube-root-ca.crt\"" Nov 01 16:20:12.657768 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:12.657735 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-383fde\"/\"openshift-service-ca.crt\"" Nov 01 16:20:12.657768 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:12.657748 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-383fde\"/\"auth-test-model-383fde-kserve-self-signed-certs\"" Nov 01 16:20:12.658778 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:12.658749 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-383fde/auth-test-model-383fde-kserve-57c55ccb8f-rqp6x"] Nov 01 16:20:12.772117 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:12.772079 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spgll\" (UniqueName: \"kubernetes.io/projected/36f441c7-2672-4581-9a0e-7142326e172b-kube-api-access-spgll\") pod \"auth-test-model-383fde-kserve-57c55ccb8f-rqp6x\" (UID: \"36f441c7-2672-4581-9a0e-7142326e172b\") " pod="ai-tenant-e2e-shared-a-w5-383fde/auth-test-model-383fde-kserve-57c55ccb8f-rqp6x" Nov 01 16:20:12.772312 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:12.772289 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36f441c7-2672-4581-9a0e-7142326e172b-home\") pod \"auth-test-model-383fde-kserve-57c55ccb8f-rqp6x\" (UID: \"36f441c7-2672-4581-9a0e-7142326e172b\") " pod="ai-tenant-e2e-shared-a-w5-383fde/auth-test-model-383fde-kserve-57c55ccb8f-rqp6x" Nov 01 16:20:12.772410 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:12.772394 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36f441c7-2672-4581-9a0e-7142326e172b-tls-certs\") pod \"auth-test-model-383fde-kserve-57c55ccb8f-rqp6x\" (UID: \"36f441c7-2672-4581-9a0e-7142326e172b\") " pod="ai-tenant-e2e-shared-a-w5-383fde/auth-test-model-383fde-kserve-57c55ccb8f-rqp6x" Nov 01 16:20:12.772471 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:12.772458 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36f441c7-2672-4581-9a0e-7142326e172b-dshm\") pod \"auth-test-model-383fde-kserve-57c55ccb8f-rqp6x\" (UID: \"36f441c7-2672-4581-9a0e-7142326e172b\") " pod="ai-tenant-e2e-shared-a-w5-383fde/auth-test-model-383fde-kserve-57c55ccb8f-rqp6x" Nov 01 16:20:12.772531 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:12.772504 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36f441c7-2672-4581-9a0e-7142326e172b-model-cache\") pod \"auth-test-model-383fde-kserve-57c55ccb8f-rqp6x\" (UID: \"36f441c7-2672-4581-9a0e-7142326e172b\") " pod="ai-tenant-e2e-shared-a-w5-383fde/auth-test-model-383fde-kserve-57c55ccb8f-rqp6x" Nov 01 16:20:12.772673 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:12.772568 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36f441c7-2672-4581-9a0e-7142326e172b-tmp-dir\") pod \"auth-test-model-383fde-kserve-57c55ccb8f-rqp6x\" (UID: \"36f441c7-2672-4581-9a0e-7142326e172b\") " pod="ai-tenant-e2e-shared-a-w5-383fde/auth-test-model-383fde-kserve-57c55ccb8f-rqp6x" Nov 01 16:20:12.876437 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:12.876398 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36f441c7-2672-4581-9a0e-7142326e172b-home\") pod \"auth-test-model-383fde-kserve-57c55ccb8f-rqp6x\" (UID: \"36f441c7-2672-4581-9a0e-7142326e172b\") " pod="ai-tenant-e2e-shared-a-w5-383fde/auth-test-model-383fde-kserve-57c55ccb8f-rqp6x" Nov 01 16:20:12.878445 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:12.878422 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36f441c7-2672-4581-9a0e-7142326e172b-tls-certs\") pod \"auth-test-model-383fde-kserve-57c55ccb8f-rqp6x\" (UID: \"36f441c7-2672-4581-9a0e-7142326e172b\") " pod="ai-tenant-e2e-shared-a-w5-383fde/auth-test-model-383fde-kserve-57c55ccb8f-rqp6x" Nov 01 16:20:12.878734 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:12.878719 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36f441c7-2672-4581-9a0e-7142326e172b-dshm\") pod \"auth-test-model-383fde-kserve-57c55ccb8f-rqp6x\" (UID: \"36f441c7-2672-4581-9a0e-7142326e172b\") " pod="ai-tenant-e2e-shared-a-w5-383fde/auth-test-model-383fde-kserve-57c55ccb8f-rqp6x" Nov 01 16:20:12.878925 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:12.878896 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36f441c7-2672-4581-9a0e-7142326e172b-model-cache\") pod \"auth-test-model-383fde-kserve-57c55ccb8f-rqp6x\" (UID: \"36f441c7-2672-4581-9a0e-7142326e172b\") " pod="ai-tenant-e2e-shared-a-w5-383fde/auth-test-model-383fde-kserve-57c55ccb8f-rqp6x" Nov 01 16:20:12.879166 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:12.879148 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36f441c7-2672-4581-9a0e-7142326e172b-tmp-dir\") pod \"auth-test-model-383fde-kserve-57c55ccb8f-rqp6x\" (UID: \"36f441c7-2672-4581-9a0e-7142326e172b\") " pod="ai-tenant-e2e-shared-a-w5-383fde/auth-test-model-383fde-kserve-57c55ccb8f-rqp6x" Nov 01 16:20:12.879848 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:12.879820 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spgll\" (UniqueName: \"kubernetes.io/projected/36f441c7-2672-4581-9a0e-7142326e172b-kube-api-access-spgll\") pod \"auth-test-model-383fde-kserve-57c55ccb8f-rqp6x\" (UID: \"36f441c7-2672-4581-9a0e-7142326e172b\") " pod="ai-tenant-e2e-shared-a-w5-383fde/auth-test-model-383fde-kserve-57c55ccb8f-rqp6x" Nov 01 16:20:12.889370 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:12.889334 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36f441c7-2672-4581-9a0e-7142326e172b-model-cache\") pod \"auth-test-model-383fde-kserve-57c55ccb8f-rqp6x\" (UID: \"36f441c7-2672-4581-9a0e-7142326e172b\") " pod="ai-tenant-e2e-shared-a-w5-383fde/auth-test-model-383fde-kserve-57c55ccb8f-rqp6x" Nov 01 16:20:12.890254 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:12.890219 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36f441c7-2672-4581-9a0e-7142326e172b-tmp-dir\") pod \"auth-test-model-383fde-kserve-57c55ccb8f-rqp6x\" (UID: \"36f441c7-2672-4581-9a0e-7142326e172b\") " pod="ai-tenant-e2e-shared-a-w5-383fde/auth-test-model-383fde-kserve-57c55ccb8f-rqp6x" Nov 01 16:20:12.892302 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:12.878289 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36f441c7-2672-4581-9a0e-7142326e172b-home\") pod \"auth-test-model-383fde-kserve-57c55ccb8f-rqp6x\" (UID: \"36f441c7-2672-4581-9a0e-7142326e172b\") " pod="ai-tenant-e2e-shared-a-w5-383fde/auth-test-model-383fde-kserve-57c55ccb8f-rqp6x" Nov 01 16:20:12.941186 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:12.937103 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36f441c7-2672-4581-9a0e-7142326e172b-dshm\") pod \"auth-test-model-383fde-kserve-57c55ccb8f-rqp6x\" (UID: \"36f441c7-2672-4581-9a0e-7142326e172b\") " pod="ai-tenant-e2e-shared-a-w5-383fde/auth-test-model-383fde-kserve-57c55ccb8f-rqp6x" Nov 01 16:20:12.941186 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:12.941092 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36f441c7-2672-4581-9a0e-7142326e172b-tls-certs\") pod \"auth-test-model-383fde-kserve-57c55ccb8f-rqp6x\" (UID: \"36f441c7-2672-4581-9a0e-7142326e172b\") " pod="ai-tenant-e2e-shared-a-w5-383fde/auth-test-model-383fde-kserve-57c55ccb8f-rqp6x" Nov 01 16:20:12.943801 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:12.943778 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spgll\" (UniqueName: \"kubernetes.io/projected/36f441c7-2672-4581-9a0e-7142326e172b-kube-api-access-spgll\") pod \"auth-test-model-383fde-kserve-57c55ccb8f-rqp6x\" (UID: \"36f441c7-2672-4581-9a0e-7142326e172b\") " pod="ai-tenant-e2e-shared-a-w5-383fde/auth-test-model-383fde-kserve-57c55ccb8f-rqp6x" Nov 01 16:20:12.971814 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:12.971554 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-383fde/auth-test-model-383fde-kserve-57c55ccb8f-rqp6x" Nov 01 16:20:13.448824 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:13.448719 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-383fde/auth-test-model-383fde-kserve-57c55ccb8f-rqp6x"] Nov 01 16:20:13.600038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:13.599992 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm"] Nov 01 16:20:13.600318 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:13.600285 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" podUID="ebf773ee-489a-418d-a031-327a467839c4" containerName="istio-proxy" containerID="cri-o://2fcdcc811dd8df484b2271e68fae197a6f7050a95ace4fb255c1c5ff0216ad64" gracePeriod=30 Nov 01 16:20:14.231463 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:14.231410 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-383fde/auth-test-model-383fde-kserve-57c55ccb8f-rqp6x" event={"ID":"36f441c7-2672-4581-9a0e-7142326e172b","Type":"ContainerStarted","Data":"b87713ac319cb123e6302ca79ca8ebd682c13e22482ead2dc659452186320cdc"} Nov 01 16:20:14.231956 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:14.231483 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-383fde/auth-test-model-383fde-kserve-57c55ccb8f-rqp6x" event={"ID":"36f441c7-2672-4581-9a0e-7142326e172b","Type":"ContainerStarted","Data":"5b8646d5d4def7609887b59fca56c3bb8c9a3a22803e6cf7194df280eb11cc16"} Nov 01 16:20:14.251843 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:14.251378 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w5-383fde/auth-test-model-383fde-kserve-57c55ccb8f-rqp6x" podStartSLOduration=2.251360061 podStartE2EDuration="2.251360061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:20:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:20:14.249060825 +0000 UTC m=+1947.580798475" watchObservedRunningTime="2026-11-01 16:20:14.251360061 +0000 UTC m=+1947.583097712" Nov 01 16:20:14.972573 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:14.972537 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-383fde/auth-test-model-383fde-kserve-57c55ccb8f-rqp6x" Nov 01 16:20:15.799824 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:15.799786 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99afca-6df5cd4d75-kqpvz" podUID="cdd9e45d-cd61-4a97-9d05-839b5cb4698b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:20:16.117094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:16.117065 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7225e5-646b876456-9wqwp" Nov 01 16:20:16.966170 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:16.966139 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" Nov 01 16:20:16.969697 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:16.969674 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" Nov 01 16:20:17.050711 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.050674 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-istio-data\") pod \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\" (UID: \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\") " Nov 01 16:20:17.050908 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.050851 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6716c2c-b539-4e3c-ac09-7f2d1426af74-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6716c2c-b539-4e3c-ac09-7f2d1426af74-istio-envoy\") pod \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\" (UID: \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\") " Nov 01 16:20:17.050908 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.050883 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6716c2c-b539-4e3c-ac09-7f2d1426af74-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6716c2c-b539-4e3c-ac09-7f2d1426af74-istio-data\") pod \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\" (UID: \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\") " Nov 01 16:20:17.051041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.050999 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6716c2c-b539-4e3c-ac09-7f2d1426af74-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6716c2c-b539-4e3c-ac09-7f2d1426af74-credential-socket\") pod \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\" (UID: \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\") " Nov 01 16:20:17.051104 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.051038 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-istio-data" pod "768d7927-13bd-4fe9-984e-c5ac56bc4fdb" (UID: "768d7927-13bd-4fe9-984e-c5ac56bc4fdb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:20:17.051104 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.051090 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-credential-socket\") pod \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\" (UID: \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\") " Nov 01 16:20:17.051220 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.051181 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-workload-socket\") pod \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\" (UID: \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\") " Nov 01 16:20:17.051278 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.051221 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-istio-podinfo\") pod \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\" (UID: \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\") " Nov 01 16:20:17.051330 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.051277 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-kube-api-access-cnk27\" (UniqueName: \"kubernetes.io/projected/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-kube-api-access-cnk27\") pod \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\" (UID: \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\") " Nov 01 16:20:17.051330 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.051308 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-istio-envoy\") pod \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\" (UID: \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\") " Nov 01 16:20:17.051446 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.051377 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6716c2c-b539-4e3c-ac09-7f2d1426af74-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6716c2c-b539-4e3c-ac09-7f2d1426af74-istio-podinfo\") pod \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\" (UID: \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\") " Nov 01 16:20:17.051446 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.051409 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6716c2c-b539-4e3c-ac09-7f2d1426af74-credential-socket" pod "b6716c2c-b539-4e3c-ac09-7f2d1426af74" (UID: "b6716c2c-b539-4e3c-ac09-7f2d1426af74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:20:17.051446 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.051428 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-istiod-ca-cert\") pod \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\" (UID: \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\") " Nov 01 16:20:17.051603 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.051472 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-istio-token\" (UniqueName: \"kubernetes.io/projected/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-istio-token\") pod \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\" (UID: \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\") " Nov 01 16:20:17.051603 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.051502 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6716c2c-b539-4e3c-ac09-7f2d1426af74-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6716c2c-b539-4e3c-ac09-7f2d1426af74-workload-socket\") pod \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\" (UID: \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\") " Nov 01 16:20:17.051603 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.051548 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6716c2c-b539-4e3c-ac09-7f2d1426af74-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6716c2c-b539-4e3c-ac09-7f2d1426af74-workload-certs\") pod \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\" (UID: \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\") " Nov 01 16:20:17.051761 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.051608 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6716c2c-b539-4e3c-ac09-7f2d1426af74-kube-api-access-sq5b6\" (UniqueName: \"kubernetes.io/projected/b6716c2c-b539-4e3c-ac09-7f2d1426af74-kube-api-access-sq5b6\") pod \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\" (UID: \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\") " Nov 01 16:20:17.051761 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.051641 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-workload-certs\") pod \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\" (UID: \"768d7927-13bd-4fe9-984e-c5ac56bc4fdb\") " Nov 01 16:20:17.051761 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.051677 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6716c2c-b539-4e3c-ac09-7f2d1426af74-istio-token\" (UniqueName: \"kubernetes.io/projected/b6716c2c-b539-4e3c-ac09-7f2d1426af74-istio-token\") pod \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\" (UID: \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\") " Nov 01 16:20:17.051761 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.051707 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-workload-socket" pod "768d7927-13bd-4fe9-984e-c5ac56bc4fdb" (UID: "768d7927-13bd-4fe9-984e-c5ac56bc4fdb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:20:17.051761 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.051719 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6716c2c-b539-4e3c-ac09-7f2d1426af74-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6716c2c-b539-4e3c-ac09-7f2d1426af74-istiod-ca-cert\") pod \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\" (UID: \"b6716c2c-b539-4e3c-ac09-7f2d1426af74\") " Nov 01 16:20:17.052052 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.051918 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-credential-socket" pod "768d7927-13bd-4fe9-984e-c5ac56bc4fdb" (UID: "768d7927-13bd-4fe9-984e-c5ac56bc4fdb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:20:17.052135 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.052112 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6716c2c-b539-4e3c-ac09-7f2d1426af74-istio-data" pod "b6716c2c-b539-4e3c-ac09-7f2d1426af74" (UID: "b6716c2c-b539-4e3c-ac09-7f2d1426af74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:20:17.052566 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.052204 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-istiod-ca-cert" pod "768d7927-13bd-4fe9-984e-c5ac56bc4fdb" (UID: "768d7927-13bd-4fe9-984e-c5ac56bc4fdb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:20:17.052566 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.052291 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-workload-certs" pod "768d7927-13bd-4fe9-984e-c5ac56bc4fdb" (UID: "768d7927-13bd-4fe9-984e-c5ac56bc4fdb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:20:17.053536 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.053037 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6716c2c-b539-4e3c-ac09-7f2d1426af74-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:17.053536 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.053067 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6716c2c-b539-4e3c-ac09-7f2d1426af74-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:17.053536 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.053083 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:17.053536 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.053097 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:17.053536 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.053114 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:17.053536 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.053128 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:17.053536 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.053142 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:17.053989 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.053550 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6716c2c-b539-4e3c-ac09-7f2d1426af74-workload-certs" pod "b6716c2c-b539-4e3c-ac09-7f2d1426af74" (UID: "b6716c2c-b539-4e3c-ac09-7f2d1426af74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:20:17.054098 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.054072 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6716c2c-b539-4e3c-ac09-7f2d1426af74-istiod-ca-cert" pod "b6716c2c-b539-4e3c-ac09-7f2d1426af74" (UID: "b6716c2c-b539-4e3c-ac09-7f2d1426af74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:20:17.054622 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.054593 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6716c2c-b539-4e3c-ac09-7f2d1426af74-workload-socket" pod "b6716c2c-b539-4e3c-ac09-7f2d1426af74" (UID: "b6716c2c-b539-4e3c-ac09-7f2d1426af74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:20:17.057157 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.057127 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6716c2c-b539-4e3c-ac09-7f2d1426af74-istio-envoy" pod "b6716c2c-b539-4e3c-ac09-7f2d1426af74" (UID: "b6716c2c-b539-4e3c-ac09-7f2d1426af74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:20:17.057889 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.057864 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-istio-podinfo" pod "768d7927-13bd-4fe9-984e-c5ac56bc4fdb" (UID: "768d7927-13bd-4fe9-984e-c5ac56bc4fdb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:20:17.057996 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.057897 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6716c2c-b539-4e3c-ac09-7f2d1426af74-istio-token" pod "b6716c2c-b539-4e3c-ac09-7f2d1426af74" (UID: "b6716c2c-b539-4e3c-ac09-7f2d1426af74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:20:17.057996 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.057953 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-kube-api-access-cnk27" pod "768d7927-13bd-4fe9-984e-c5ac56bc4fdb" (UID: "768d7927-13bd-4fe9-984e-c5ac56bc4fdb"). InnerVolumeSpecName "kube-api-access-cnk27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:20:17.058159 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.058046 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-istio-token" pod "768d7927-13bd-4fe9-984e-c5ac56bc4fdb" (UID: "768d7927-13bd-4fe9-984e-c5ac56bc4fdb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:20:17.058159 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.058095 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-istio-envoy" pod "768d7927-13bd-4fe9-984e-c5ac56bc4fdb" (UID: "768d7927-13bd-4fe9-984e-c5ac56bc4fdb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:20:17.059789 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.059764 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6716c2c-b539-4e3c-ac09-7f2d1426af74-kube-api-access-sq5b6" pod "b6716c2c-b539-4e3c-ac09-7f2d1426af74" (UID: "b6716c2c-b539-4e3c-ac09-7f2d1426af74"). InnerVolumeSpecName "kube-api-access-sq5b6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:20:17.059892 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.059844 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6716c2c-b539-4e3c-ac09-7f2d1426af74-istio-podinfo" pod "b6716c2c-b539-4e3c-ac09-7f2d1426af74" (UID: "b6716c2c-b539-4e3c-ac09-7f2d1426af74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:20:17.154863 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.154765 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6716c2c-b539-4e3c-ac09-7f2d1426af74-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:17.154863 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.154806 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:17.154863 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.154824 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnk27\" (UniqueName: \"kubernetes.io/projected/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-kube-api-access-cnk27\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:17.154863 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.154834 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:17.154863 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.154842 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6716c2c-b539-4e3c-ac09-7f2d1426af74-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:17.154863 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.154851 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/768d7927-13bd-4fe9-984e-c5ac56bc4fdb-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:17.154863 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.154859 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6716c2c-b539-4e3c-ac09-7f2d1426af74-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:17.154863 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.154867 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6716c2c-b539-4e3c-ac09-7f2d1426af74-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:17.155550 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.154876 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq5b6\" (UniqueName: \"kubernetes.io/projected/b6716c2c-b539-4e3c-ac09-7f2d1426af74-kube-api-access-sq5b6\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:17.155550 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.154884 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6716c2c-b539-4e3c-ac09-7f2d1426af74-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:17.155550 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.154892 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6716c2c-b539-4e3c-ac09-7f2d1426af74-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:17.315038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.310501 4363 generic.go:356] "Generic (PLEG): container finished" podID="768d7927-13bd-4fe9-984e-c5ac56bc4fdb" containerID="2e2c499dc24759b887be8c6339f77a616ee4e1ef1bc78d5eba9416ecbc58fe5d" exitCode=0 Nov 01 16:20:17.315038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.310609 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" event={"ID":"768d7927-13bd-4fe9-984e-c5ac56bc4fdb","Type":"ContainerDied","Data":"2e2c499dc24759b887be8c6339f77a616ee4e1ef1bc78d5eba9416ecbc58fe5d"} Nov 01 16:20:17.315038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.310660 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" event={"ID":"768d7927-13bd-4fe9-984e-c5ac56bc4fdb","Type":"ContainerDied","Data":"14f9d3c30258b49d3ca08dea9ba7e9292dd015dfe36fac4ec79307f2be674647"} Nov 01 16:20:17.315038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.310688 4363 scope.go:122] "RemoveContainer" containerID="2e2c499dc24759b887be8c6339f77a616ee4e1ef1bc78d5eba9416ecbc58fe5d" Nov 01 16:20:17.315038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.310846 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv" Nov 01 16:20:17.332081 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.325832 4363 generic.go:356] "Generic (PLEG): container finished" podID="b6716c2c-b539-4e3c-ac09-7f2d1426af74" containerID="992c76fb0b84fab448f1f2f44a73f0b15590dd6a7359a3a7dd6c038e52feacce" exitCode=0 Nov 01 16:20:17.332081 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.325967 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" event={"ID":"b6716c2c-b539-4e3c-ac09-7f2d1426af74","Type":"ContainerDied","Data":"992c76fb0b84fab448f1f2f44a73f0b15590dd6a7359a3a7dd6c038e52feacce"} Nov 01 16:20:17.332081 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.326035 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" event={"ID":"b6716c2c-b539-4e3c-ac09-7f2d1426af74","Type":"ContainerDied","Data":"d76521f60f9a38e58080b8f0e0a4ddc2d8998c98c9ebe736bd903631d87a4280"} Nov 01 16:20:17.332081 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.326120 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn" Nov 01 16:20:17.344247 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.343709 4363 scope.go:122] "RemoveContainer" containerID="2e2c499dc24759b887be8c6339f77a616ee4e1ef1bc78d5eba9416ecbc58fe5d" Nov 01 16:20:17.344247 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:20:17.344105 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e2c499dc24759b887be8c6339f77a616ee4e1ef1bc78d5eba9416ecbc58fe5d\": container with ID starting with 2e2c499dc24759b887be8c6339f77a616ee4e1ef1bc78d5eba9416ecbc58fe5d not found: ID does not exist" containerID="2e2c499dc24759b887be8c6339f77a616ee4e1ef1bc78d5eba9416ecbc58fe5d" Nov 01 16:20:17.344247 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.344135 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e2c499dc24759b887be8c6339f77a616ee4e1ef1bc78d5eba9416ecbc58fe5d"} err="failed to get container status \"2e2c499dc24759b887be8c6339f77a616ee4e1ef1bc78d5eba9416ecbc58fe5d\": rpc error: code = NotFound desc = could not find container \"2e2c499dc24759b887be8c6339f77a616ee4e1ef1bc78d5eba9416ecbc58fe5d\": container with ID starting with 2e2c499dc24759b887be8c6339f77a616ee4e1ef1bc78d5eba9416ecbc58fe5d not found: ID does not exist" Nov 01 16:20:17.344247 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.344156 4363 scope.go:122] "RemoveContainer" containerID="992c76fb0b84fab448f1f2f44a73f0b15590dd6a7359a3a7dd6c038e52feacce" Nov 01 16:20:17.349428 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.349400 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv"] Nov 01 16:20:17.357451 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.357427 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-mx6kv"] Nov 01 16:20:17.357975 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.357960 4363 scope.go:122] "RemoveContainer" containerID="95dd1549927c9b22958db180914ee1cfa32ed7449cd317694cbacfd7ff364525" Nov 01 16:20:17.368138 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.368115 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn"] Nov 01 16:20:17.371819 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.371795 4363 scope.go:122] "RemoveContainer" containerID="992c76fb0b84fab448f1f2f44a73f0b15590dd6a7359a3a7dd6c038e52feacce" Nov 01 16:20:17.372185 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:20:17.372161 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"992c76fb0b84fab448f1f2f44a73f0b15590dd6a7359a3a7dd6c038e52feacce\": container with ID starting with 992c76fb0b84fab448f1f2f44a73f0b15590dd6a7359a3a7dd6c038e52feacce not found: ID does not exist" containerID="992c76fb0b84fab448f1f2f44a73f0b15590dd6a7359a3a7dd6c038e52feacce" Nov 01 16:20:17.372300 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.372191 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"992c76fb0b84fab448f1f2f44a73f0b15590dd6a7359a3a7dd6c038e52feacce"} err="failed to get container status \"992c76fb0b84fab448f1f2f44a73f0b15590dd6a7359a3a7dd6c038e52feacce\": rpc error: code = NotFound desc = could not find container \"992c76fb0b84fab448f1f2f44a73f0b15590dd6a7359a3a7dd6c038e52feacce\": container with ID starting with 992c76fb0b84fab448f1f2f44a73f0b15590dd6a7359a3a7dd6c038e52feacce not found: ID does not exist" Nov 01 16:20:17.372300 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.372209 4363 scope.go:122] "RemoveContainer" containerID="95dd1549927c9b22958db180914ee1cfa32ed7449cd317694cbacfd7ff364525" Nov 01 16:20:17.372508 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:20:17.372491 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95dd1549927c9b22958db180914ee1cfa32ed7449cd317694cbacfd7ff364525\": container with ID starting with 95dd1549927c9b22958db180914ee1cfa32ed7449cd317694cbacfd7ff364525 not found: ID does not exist" containerID="95dd1549927c9b22958db180914ee1cfa32ed7449cd317694cbacfd7ff364525" Nov 01 16:20:17.372583 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.372511 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95dd1549927c9b22958db180914ee1cfa32ed7449cd317694cbacfd7ff364525"} err="failed to get container status \"95dd1549927c9b22958db180914ee1cfa32ed7449cd317694cbacfd7ff364525\": rpc error: code = NotFound desc = could not find container \"95dd1549927c9b22958db180914ee1cfa32ed7449cd317694cbacfd7ff364525\": container with ID starting with 95dd1549927c9b22958db180914ee1cfa32ed7449cd317694cbacfd7ff364525 not found: ID does not exist" Nov 01 16:20:17.373992 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:17.373972 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-kr4hn"] Nov 01 16:20:18.389093 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:20:18.389053 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:20:18.804202 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:18.802674 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="768d7927-13bd-4fe9-984e-c5ac56bc4fdb" path="/var/lib/kubelet/pods/768d7927-13bd-4fe9-984e-c5ac56bc4fdb/volumes" Nov 01 16:20:18.804202 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:18.803524 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6716c2c-b539-4e3c-ac09-7f2d1426af74" path="/var/lib/kubelet/pods/b6716c2c-b539-4e3c-ac09-7f2d1426af74/volumes" Nov 01 16:20:18.970903 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:18.970871 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" Nov 01 16:20:19.075449 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:19.075417 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebf773ee-489a-418d-a031-327a467839c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebf773ee-489a-418d-a031-327a467839c4-istio-envoy\") pod \"ebf773ee-489a-418d-a031-327a467839c4\" (UID: \"ebf773ee-489a-418d-a031-327a467839c4\") " Nov 01 16:20:19.075679 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:19.075522 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebf773ee-489a-418d-a031-327a467839c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebf773ee-489a-418d-a031-327a467839c4-istiod-ca-cert\") pod \"ebf773ee-489a-418d-a031-327a467839c4\" (UID: \"ebf773ee-489a-418d-a031-327a467839c4\") " Nov 01 16:20:19.075679 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:19.075550 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebf773ee-489a-418d-a031-327a467839c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebf773ee-489a-418d-a031-327a467839c4-workload-socket\") pod \"ebf773ee-489a-418d-a031-327a467839c4\" (UID: \"ebf773ee-489a-418d-a031-327a467839c4\") " Nov 01 16:20:19.075679 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:19.075572 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebf773ee-489a-418d-a031-327a467839c4-kube-api-access-4xgkk\" (UniqueName: \"kubernetes.io/projected/ebf773ee-489a-418d-a031-327a467839c4-kube-api-access-4xgkk\") pod \"ebf773ee-489a-418d-a031-327a467839c4\" (UID: \"ebf773ee-489a-418d-a031-327a467839c4\") " Nov 01 16:20:19.075679 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:19.075619 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ebf773ee-489a-418d-a031-327a467839c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebf773ee-489a-418d-a031-327a467839c4-istio-podinfo\") pod \"ebf773ee-489a-418d-a031-327a467839c4\" (UID: \"ebf773ee-489a-418d-a031-327a467839c4\") " Nov 01 16:20:19.075906 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:19.075853 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebf773ee-489a-418d-a031-327a467839c4-workload-socket" pod "ebf773ee-489a-418d-a031-327a467839c4" (UID: "ebf773ee-489a-418d-a031-327a467839c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:20:19.075971 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:19.075954 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebf773ee-489a-418d-a031-327a467839c4-istiod-ca-cert" pod "ebf773ee-489a-418d-a031-327a467839c4" (UID: "ebf773ee-489a-418d-a031-327a467839c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:20:19.076038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:19.075985 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebf773ee-489a-418d-a031-327a467839c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebf773ee-489a-418d-a031-327a467839c4-workload-certs\") pod \"ebf773ee-489a-418d-a031-327a467839c4\" (UID: \"ebf773ee-489a-418d-a031-327a467839c4\") " Nov 01 16:20:19.076103 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:19.076080 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebf773ee-489a-418d-a031-327a467839c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebf773ee-489a-418d-a031-327a467839c4-credential-socket\") pod \"ebf773ee-489a-418d-a031-327a467839c4\" (UID: \"ebf773ee-489a-418d-a031-327a467839c4\") " Nov 01 16:20:19.076164 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:19.076126 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebf773ee-489a-418d-a031-327a467839c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebf773ee-489a-418d-a031-327a467839c4-istio-data\") pod \"ebf773ee-489a-418d-a031-327a467839c4\" (UID: \"ebf773ee-489a-418d-a031-327a467839c4\") " Nov 01 16:20:19.076219 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:19.076181 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebf773ee-489a-418d-a031-327a467839c4-istio-token\" (UniqueName: \"kubernetes.io/projected/ebf773ee-489a-418d-a031-327a467839c4-istio-token\") pod \"ebf773ee-489a-418d-a031-327a467839c4\" (UID: \"ebf773ee-489a-418d-a031-327a467839c4\") " Nov 01 16:20:19.076275 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:19.076239 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebf773ee-489a-418d-a031-327a467839c4-workload-certs" pod "ebf773ee-489a-418d-a031-327a467839c4" (UID: "ebf773ee-489a-418d-a031-327a467839c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:20:19.076409 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:19.076381 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebf773ee-489a-418d-a031-327a467839c4-credential-socket" pod "ebf773ee-489a-418d-a031-327a467839c4" (UID: "ebf773ee-489a-418d-a031-327a467839c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:20:19.076409 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:19.076401 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebf773ee-489a-418d-a031-327a467839c4-istio-data" pod "ebf773ee-489a-418d-a031-327a467839c4" (UID: "ebf773ee-489a-418d-a031-327a467839c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:20:19.076914 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:19.076891 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebf773ee-489a-418d-a031-327a467839c4-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:19.077067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:19.076916 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebf773ee-489a-418d-a031-327a467839c4-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:19.077067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:19.076931 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebf773ee-489a-418d-a031-327a467839c4-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:19.077067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:19.076943 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebf773ee-489a-418d-a031-327a467839c4-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:19.077067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:19.076955 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebf773ee-489a-418d-a031-327a467839c4-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:19.078233 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:19.078206 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebf773ee-489a-418d-a031-327a467839c4-istio-envoy" pod "ebf773ee-489a-418d-a031-327a467839c4" (UID: "ebf773ee-489a-418d-a031-327a467839c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:20:19.078342 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:19.078306 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ebf773ee-489a-418d-a031-327a467839c4-istio-podinfo" pod "ebf773ee-489a-418d-a031-327a467839c4" (UID: "ebf773ee-489a-418d-a031-327a467839c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:20:19.078567 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:19.078543 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebf773ee-489a-418d-a031-327a467839c4-istio-token" pod "ebf773ee-489a-418d-a031-327a467839c4" (UID: "ebf773ee-489a-418d-a031-327a467839c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:20:19.079193 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:19.079165 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebf773ee-489a-418d-a031-327a467839c4-kube-api-access-4xgkk" pod "ebf773ee-489a-418d-a031-327a467839c4" (UID: "ebf773ee-489a-418d-a031-327a467839c4"). InnerVolumeSpecName "kube-api-access-4xgkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:20:19.178604 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:19.178524 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebf773ee-489a-418d-a031-327a467839c4-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:19.178604 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:19.178555 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebf773ee-489a-418d-a031-327a467839c4-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:19.178604 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:19.178566 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xgkk\" (UniqueName: \"kubernetes.io/projected/ebf773ee-489a-418d-a031-327a467839c4-kube-api-access-4xgkk\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:19.178604 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:19.178576 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebf773ee-489a-418d-a031-327a467839c4-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:19.384691 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:19.381049 4363 generic.go:356] "Generic (PLEG): container finished" podID="ebf773ee-489a-418d-a031-327a467839c4" containerID="2fcdcc811dd8df484b2271e68fae197a6f7050a95ace4fb255c1c5ff0216ad64" exitCode=0 Nov 01 16:20:19.384691 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:19.381231 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" event={"ID":"ebf773ee-489a-418d-a031-327a467839c4","Type":"ContainerDied","Data":"2fcdcc811dd8df484b2271e68fae197a6f7050a95ace4fb255c1c5ff0216ad64"} Nov 01 16:20:19.384691 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:19.381283 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" event={"ID":"ebf773ee-489a-418d-a031-327a467839c4","Type":"ContainerDied","Data":"2e4646bba90843876574a739b699881cc909d3d43547a9aa25992e92b9ff8dc0"} Nov 01 16:20:19.384691 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:19.381311 4363 scope.go:122] "RemoveContainer" containerID="2fcdcc811dd8df484b2271e68fae197a6f7050a95ace4fb255c1c5ff0216ad64" Nov 01 16:20:19.384691 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:19.381461 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm" Nov 01 16:20:19.406794 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:19.406769 4363 scope.go:122] "RemoveContainer" containerID="1fb51714c82041bd241aadebc9f1707c9825ad779c38ebfec5ffb437a84e719c" Nov 01 16:20:19.421234 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:19.421206 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm"] Nov 01 16:20:19.421656 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:19.421634 4363 scope.go:122] "RemoveContainer" containerID="2fcdcc811dd8df484b2271e68fae197a6f7050a95ace4fb255c1c5ff0216ad64" Nov 01 16:20:19.422024 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:20:19.421975 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fcdcc811dd8df484b2271e68fae197a6f7050a95ace4fb255c1c5ff0216ad64\": container with ID starting with 2fcdcc811dd8df484b2271e68fae197a6f7050a95ace4fb255c1c5ff0216ad64 not found: ID does not exist" containerID="2fcdcc811dd8df484b2271e68fae197a6f7050a95ace4fb255c1c5ff0216ad64" Nov 01 16:20:19.422118 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:19.422029 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fcdcc811dd8df484b2271e68fae197a6f7050a95ace4fb255c1c5ff0216ad64"} err="failed to get container status \"2fcdcc811dd8df484b2271e68fae197a6f7050a95ace4fb255c1c5ff0216ad64\": rpc error: code = NotFound desc = could not find container \"2fcdcc811dd8df484b2271e68fae197a6f7050a95ace4fb255c1c5ff0216ad64\": container with ID starting with 2fcdcc811dd8df484b2271e68fae197a6f7050a95ace4fb255c1c5ff0216ad64 not found: ID does not exist" Nov 01 16:20:19.422118 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:19.422055 4363 scope.go:122] "RemoveContainer" containerID="1fb51714c82041bd241aadebc9f1707c9825ad779c38ebfec5ffb437a84e719c" Nov 01 16:20:19.422376 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:20:19.422352 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fb51714c82041bd241aadebc9f1707c9825ad779c38ebfec5ffb437a84e719c\": container with ID starting with 1fb51714c82041bd241aadebc9f1707c9825ad779c38ebfec5ffb437a84e719c not found: ID does not exist" containerID="1fb51714c82041bd241aadebc9f1707c9825ad779c38ebfec5ffb437a84e719c" Nov 01 16:20:19.422468 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:19.422384 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fb51714c82041bd241aadebc9f1707c9825ad779c38ebfec5ffb437a84e719c"} err="failed to get container status \"1fb51714c82041bd241aadebc9f1707c9825ad779c38ebfec5ffb437a84e719c\": rpc error: code = NotFound desc = could not find container \"1fb51714c82041bd241aadebc9f1707c9825ad779c38ebfec5ffb437a84e719c\": container with ID starting with 1fb51714c82041bd241aadebc9f1707c9825ad779c38ebfec5ffb437a84e719c not found: ID does not exist" Nov 01 16:20:19.424496 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:19.424471 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-grqzm"] Nov 01 16:20:20.804280 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:20.804244 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebf773ee-489a-418d-a031-327a467839c4" path="/var/lib/kubelet/pods/ebf773ee-489a-418d-a031-327a467839c4/volumes" Nov 01 16:20:21.061443 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:21.061380 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99afca-6df5cd4d75-kqpvz" Nov 01 16:20:21.122316 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:21.122277 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdd9e45d-cd61-4a97-9d05-839b5cb4698b-kube-api-access-g5lcf\" (UniqueName: \"kubernetes.io/projected/cdd9e45d-cd61-4a97-9d05-839b5cb4698b-kube-api-access-g5lcf\") pod \"cdd9e45d-cd61-4a97-9d05-839b5cb4698b\" (UID: \"cdd9e45d-cd61-4a97-9d05-839b5cb4698b\") " Nov 01 16:20:21.122489 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:21.122435 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdd9e45d-cd61-4a97-9d05-839b5cb4698b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd9e45d-cd61-4a97-9d05-839b5cb4698b-plugins-config-volume\") pod \"cdd9e45d-cd61-4a97-9d05-839b5cb4698b\" (UID: \"cdd9e45d-cd61-4a97-9d05-839b5cb4698b\") " Nov 01 16:20:21.123574 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:21.123540 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdd9e45d-cd61-4a97-9d05-839b5cb4698b-plugins-config-volume" pod "cdd9e45d-cd61-4a97-9d05-839b5cb4698b" (UID: "cdd9e45d-cd61-4a97-9d05-839b5cb4698b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:20:21.127571 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:21.127541 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdd9e45d-cd61-4a97-9d05-839b5cb4698b-kube-api-access-g5lcf" pod "cdd9e45d-cd61-4a97-9d05-839b5cb4698b" (UID: "cdd9e45d-cd61-4a97-9d05-839b5cb4698b"). InnerVolumeSpecName "kube-api-access-g5lcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:20:21.235040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:21.227435 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5lcf\" (UniqueName: \"kubernetes.io/projected/cdd9e45d-cd61-4a97-9d05-839b5cb4698b-kube-api-access-g5lcf\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:21.235040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:21.227473 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd9e45d-cd61-4a97-9d05-839b5cb4698b-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:21.404771 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:21.404734 4363 generic.go:356] "Generic (PLEG): container finished" podID="cdd9e45d-cd61-4a97-9d05-839b5cb4698b" containerID="69f66ffc632eed9f5f72736c3230bb932dcc7cb07b8adacc8b51062186bb88b8" exitCode=0 Nov 01 16:20:21.404989 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:21.404806 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99afca-6df5cd4d75-kqpvz" Nov 01 16:20:21.404989 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:21.404829 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99afca-6df5cd4d75-kqpvz" event={"ID":"cdd9e45d-cd61-4a97-9d05-839b5cb4698b","Type":"ContainerDied","Data":"69f66ffc632eed9f5f72736c3230bb932dcc7cb07b8adacc8b51062186bb88b8"} Nov 01 16:20:21.404989 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:21.404893 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99afca-6df5cd4d75-kqpvz" event={"ID":"cdd9e45d-cd61-4a97-9d05-839b5cb4698b","Type":"ContainerDied","Data":"54e9dd2a38df97f8220cb3ec685cf4b919e0fe5d38ed18d9371155e2af3d308f"} Nov 01 16:20:21.404989 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:21.404927 4363 scope.go:122] "RemoveContainer" containerID="69f66ffc632eed9f5f72736c3230bb932dcc7cb07b8adacc8b51062186bb88b8" Nov 01 16:20:21.420417 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:21.420396 4363 scope.go:122] "RemoveContainer" containerID="69f66ffc632eed9f5f72736c3230bb932dcc7cb07b8adacc8b51062186bb88b8" Nov 01 16:20:21.420757 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:20:21.420737 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69f66ffc632eed9f5f72736c3230bb932dcc7cb07b8adacc8b51062186bb88b8\": container with ID starting with 69f66ffc632eed9f5f72736c3230bb932dcc7cb07b8adacc8b51062186bb88b8 not found: ID does not exist" containerID="69f66ffc632eed9f5f72736c3230bb932dcc7cb07b8adacc8b51062186bb88b8" Nov 01 16:20:21.420881 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:21.420767 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69f66ffc632eed9f5f72736c3230bb932dcc7cb07b8adacc8b51062186bb88b8"} err="failed to get container status \"69f66ffc632eed9f5f72736c3230bb932dcc7cb07b8adacc8b51062186bb88b8\": rpc error: code = NotFound desc = could not find container \"69f66ffc632eed9f5f72736c3230bb932dcc7cb07b8adacc8b51062186bb88b8\": container with ID starting with 69f66ffc632eed9f5f72736c3230bb932dcc7cb07b8adacc8b51062186bb88b8 not found: ID does not exist" Nov 01 16:20:21.430482 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:21.430447 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-99afca-6df5cd4d75-kqpvz"] Nov 01 16:20:21.434668 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:21.434647 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-99afca-6df5cd4d75-kqpvz"] Nov 01 16:20:22.801301 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:22.801263 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdd9e45d-cd61-4a97-9d05-839b5cb4698b" path="/var/lib/kubelet/pods/cdd9e45d-cd61-4a97-9d05-839b5cb4698b/volumes" Nov 01 16:20:22.973450 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:22.973417 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w5-383fde/auth-test-model-383fde-kserve-57c55ccb8f-rqp6x" Nov 01 16:20:23.021042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:23.020997 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w5-383fde/auth-test-model-383fde-kserve-57c55ccb8f-rqp6x" Nov 01 16:20:23.439590 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:23.439563 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-383fde/auth-test-model-383fde-kserve-57c55ccb8f-rqp6x" Nov 01 16:20:27.647479 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:27.647447 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-9a482b/auth-test-model-9a482b-kserve-6c554dbfd9-wqk65"] Nov 01 16:20:27.648290 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:27.648268 4363 cpu_manager.go:414] "containerMap: removing container" podUID="ebf773ee-489a-418d-a031-327a467839c4" containerName="istio-proxy" Nov 01 16:20:27.648290 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:27.648287 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebf773ee-489a-418d-a031-327a467839c4" containerName="istio-proxy" Nov 01 16:20:27.648503 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:27.648305 4363 cpu_manager.go:414] "containerMap: removing container" podUID="ebf773ee-489a-418d-a031-327a467839c4" containerName="istio-proxy" Nov 01 16:20:27.648503 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:27.648310 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebf773ee-489a-418d-a031-327a467839c4" containerName="istio-proxy" Nov 01 16:20:27.648503 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:27.648317 4363 cpu_manager.go:414] "containerMap: removing container" podUID="768d7927-13bd-4fe9-984e-c5ac56bc4fdb" containerName="istio-proxy" Nov 01 16:20:27.648503 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:27.648322 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="768d7927-13bd-4fe9-984e-c5ac56bc4fdb" containerName="istio-proxy" Nov 01 16:20:27.648503 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:27.648356 4363 cpu_manager.go:414] "containerMap: removing container" podUID="cdd9e45d-cd61-4a97-9d05-839b5cb4698b" containerName="payload-pre-processing" Nov 01 16:20:27.648503 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:27.648362 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdd9e45d-cd61-4a97-9d05-839b5cb4698b" containerName="payload-pre-processing" Nov 01 16:20:27.648503 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:27.648381 4363 cpu_manager.go:414] "containerMap: removing container" podUID="b6716c2c-b539-4e3c-ac09-7f2d1426af74" containerName="istio-proxy" Nov 01 16:20:27.648503 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:27.648386 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6716c2c-b539-4e3c-ac09-7f2d1426af74" containerName="istio-proxy" Nov 01 16:20:27.648503 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:27.648394 4363 cpu_manager.go:414] "containerMap: removing container" podUID="b6716c2c-b539-4e3c-ac09-7f2d1426af74" containerName="istio-proxy" Nov 01 16:20:27.648503 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:27.648399 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6716c2c-b539-4e3c-ac09-7f2d1426af74" containerName="istio-proxy" Nov 01 16:20:27.649047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:27.648513 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebf773ee-489a-418d-a031-327a467839c4" containerName="istio-proxy" Nov 01 16:20:27.649047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:27.648522 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="768d7927-13bd-4fe9-984e-c5ac56bc4fdb" containerName="istio-proxy" Nov 01 16:20:27.649047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:27.648533 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdd9e45d-cd61-4a97-9d05-839b5cb4698b" containerName="payload-pre-processing" Nov 01 16:20:27.649047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:27.648540 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6716c2c-b539-4e3c-ac09-7f2d1426af74" containerName="istio-proxy" Nov 01 16:20:27.649047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:27.648546 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6716c2c-b539-4e3c-ac09-7f2d1426af74" containerName="istio-proxy" Nov 01 16:20:27.654178 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:27.654149 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9a482b/auth-test-model-9a482b-kserve-6c554dbfd9-wqk65" Nov 01 16:20:27.661747 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:27.661721 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9a482b/auth-test-model-9a482b-kserve-6c554dbfd9-wqk65"] Nov 01 16:20:27.662307 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:27.662285 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-9a482b\"/\"openshift-service-ca.crt\"" Nov 01 16:20:27.662886 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:27.662863 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-9a482b\"/\"kube-root-ca.crt\"" Nov 01 16:20:27.663274 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:27.663255 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-9a482b\"/\"default-dockercfg-sl2lj\"" Nov 01 16:20:27.664229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:27.664208 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-9a482b\"/\"auth-test-model-9a482b-kserve-self-signed-certs\"" Nov 01 16:20:27.748780 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:27.748744 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ee17e76-6806-4747-9376-e9c8e851b70c-tmp-dir\") pod \"auth-test-model-9a482b-kserve-6c554dbfd9-wqk65\" (UID: \"1ee17e76-6806-4747-9376-e9c8e851b70c\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/auth-test-model-9a482b-kserve-6c554dbfd9-wqk65" Nov 01 16:20:27.748946 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:27.748822 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ee17e76-6806-4747-9376-e9c8e851b70c-dshm\") pod \"auth-test-model-9a482b-kserve-6c554dbfd9-wqk65\" (UID: \"1ee17e76-6806-4747-9376-e9c8e851b70c\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/auth-test-model-9a482b-kserve-6c554dbfd9-wqk65" Nov 01 16:20:27.748946 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:27.748872 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ee17e76-6806-4747-9376-e9c8e851b70c-model-cache\") pod \"auth-test-model-9a482b-kserve-6c554dbfd9-wqk65\" (UID: \"1ee17e76-6806-4747-9376-e9c8e851b70c\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/auth-test-model-9a482b-kserve-6c554dbfd9-wqk65" Nov 01 16:20:27.749042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:27.748983 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ee17e76-6806-4747-9376-e9c8e851b70c-tls-certs\") pod \"auth-test-model-9a482b-kserve-6c554dbfd9-wqk65\" (UID: \"1ee17e76-6806-4747-9376-e9c8e851b70c\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/auth-test-model-9a482b-kserve-6c554dbfd9-wqk65" Nov 01 16:20:27.749085 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:27.749053 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhjl7\" (UniqueName: \"kubernetes.io/projected/1ee17e76-6806-4747-9376-e9c8e851b70c-kube-api-access-jhjl7\") pod \"auth-test-model-9a482b-kserve-6c554dbfd9-wqk65\" (UID: \"1ee17e76-6806-4747-9376-e9c8e851b70c\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/auth-test-model-9a482b-kserve-6c554dbfd9-wqk65" Nov 01 16:20:27.749139 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:27.749121 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ee17e76-6806-4747-9376-e9c8e851b70c-home\") pod \"auth-test-model-9a482b-kserve-6c554dbfd9-wqk65\" (UID: \"1ee17e76-6806-4747-9376-e9c8e851b70c\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/auth-test-model-9a482b-kserve-6c554dbfd9-wqk65" Nov 01 16:20:27.851205 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:27.851158 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhjl7\" (UniqueName: \"kubernetes.io/projected/1ee17e76-6806-4747-9376-e9c8e851b70c-kube-api-access-jhjl7\") pod \"auth-test-model-9a482b-kserve-6c554dbfd9-wqk65\" (UID: \"1ee17e76-6806-4747-9376-e9c8e851b70c\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/auth-test-model-9a482b-kserve-6c554dbfd9-wqk65" Nov 01 16:20:27.851399 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:27.851230 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ee17e76-6806-4747-9376-e9c8e851b70c-home\") pod \"auth-test-model-9a482b-kserve-6c554dbfd9-wqk65\" (UID: \"1ee17e76-6806-4747-9376-e9c8e851b70c\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/auth-test-model-9a482b-kserve-6c554dbfd9-wqk65" Nov 01 16:20:27.851929 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:27.851501 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ee17e76-6806-4747-9376-e9c8e851b70c-tmp-dir\") pod \"auth-test-model-9a482b-kserve-6c554dbfd9-wqk65\" (UID: \"1ee17e76-6806-4747-9376-e9c8e851b70c\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/auth-test-model-9a482b-kserve-6c554dbfd9-wqk65" Nov 01 16:20:27.851929 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:27.851585 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ee17e76-6806-4747-9376-e9c8e851b70c-dshm\") pod \"auth-test-model-9a482b-kserve-6c554dbfd9-wqk65\" (UID: \"1ee17e76-6806-4747-9376-e9c8e851b70c\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/auth-test-model-9a482b-kserve-6c554dbfd9-wqk65" Nov 01 16:20:27.851929 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:27.851609 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ee17e76-6806-4747-9376-e9c8e851b70c-model-cache\") pod \"auth-test-model-9a482b-kserve-6c554dbfd9-wqk65\" (UID: \"1ee17e76-6806-4747-9376-e9c8e851b70c\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/auth-test-model-9a482b-kserve-6c554dbfd9-wqk65" Nov 01 16:20:27.851929 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:27.851671 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ee17e76-6806-4747-9376-e9c8e851b70c-tls-certs\") pod \"auth-test-model-9a482b-kserve-6c554dbfd9-wqk65\" (UID: \"1ee17e76-6806-4747-9376-e9c8e851b70c\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/auth-test-model-9a482b-kserve-6c554dbfd9-wqk65" Nov 01 16:20:27.851929 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:27.851874 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ee17e76-6806-4747-9376-e9c8e851b70c-home\") pod \"auth-test-model-9a482b-kserve-6c554dbfd9-wqk65\" (UID: \"1ee17e76-6806-4747-9376-e9c8e851b70c\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/auth-test-model-9a482b-kserve-6c554dbfd9-wqk65" Nov 01 16:20:27.852338 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:27.852157 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ee17e76-6806-4747-9376-e9c8e851b70c-model-cache\") pod \"auth-test-model-9a482b-kserve-6c554dbfd9-wqk65\" (UID: \"1ee17e76-6806-4747-9376-e9c8e851b70c\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/auth-test-model-9a482b-kserve-6c554dbfd9-wqk65" Nov 01 16:20:27.852338 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:27.852259 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ee17e76-6806-4747-9376-e9c8e851b70c-tmp-dir\") pod \"auth-test-model-9a482b-kserve-6c554dbfd9-wqk65\" (UID: \"1ee17e76-6806-4747-9376-e9c8e851b70c\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/auth-test-model-9a482b-kserve-6c554dbfd9-wqk65" Nov 01 16:20:27.855309 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:27.855282 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ee17e76-6806-4747-9376-e9c8e851b70c-dshm\") pod \"auth-test-model-9a482b-kserve-6c554dbfd9-wqk65\" (UID: \"1ee17e76-6806-4747-9376-e9c8e851b70c\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/auth-test-model-9a482b-kserve-6c554dbfd9-wqk65" Nov 01 16:20:27.855571 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:27.855551 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ee17e76-6806-4747-9376-e9c8e851b70c-tls-certs\") pod \"auth-test-model-9a482b-kserve-6c554dbfd9-wqk65\" (UID: \"1ee17e76-6806-4747-9376-e9c8e851b70c\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/auth-test-model-9a482b-kserve-6c554dbfd9-wqk65" Nov 01 16:20:27.871049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:27.859734 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhjl7\" (UniqueName: \"kubernetes.io/projected/1ee17e76-6806-4747-9376-e9c8e851b70c-kube-api-access-jhjl7\") pod \"auth-test-model-9a482b-kserve-6c554dbfd9-wqk65\" (UID: \"1ee17e76-6806-4747-9376-e9c8e851b70c\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/auth-test-model-9a482b-kserve-6c554dbfd9-wqk65" Nov 01 16:20:27.990092 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:27.985812 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9a482b/auth-test-model-9a482b-kserve-6c554dbfd9-wqk65" Nov 01 16:20:28.314602 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:28.314518 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-04f248/test-model-04f248-kserve-7cf9d4bf7f-kd4rz"] Nov 01 16:20:28.433676 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:28.433636 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9a482b/auth-test-model-9a482b-kserve-6c554dbfd9-wqk65"] Nov 01 16:20:28.482041 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:20:28.455857 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ee17e76_6806_4747_9376_e9c8e851b70c.slice/crio-a3872b20e93c080f0f0aaa1c08f609ae1a002b83e1a8ce39958c364517817e09 WatchSource:0}: Error finding container a3872b20e93c080f0f0aaa1c08f609ae1a002b83e1a8ce39958c364517817e09: Status 404 returned error can't find the container with id a3872b20e93c080f0f0aaa1c08f609ae1a002b83e1a8ce39958c364517817e09 Nov 01 16:20:28.539983 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:28.539904 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9a482b/auth-test-model-9a482b-kserve-6c554dbfd9-wqk65" event={"ID":"1ee17e76-6806-4747-9376-e9c8e851b70c","Type":"ContainerStarted","Data":"a3872b20e93c080f0f0aaa1c08f609ae1a002b83e1a8ce39958c364517817e09"} Nov 01 16:20:29.549475 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:29.549438 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9a482b/auth-test-model-9a482b-kserve-6c554dbfd9-wqk65" event={"ID":"1ee17e76-6806-4747-9376-e9c8e851b70c","Type":"ContainerStarted","Data":"91e9ec9369f7c75e4002867661adacb88f1bb12b5de86d5364d8969516d9dd8d"} Nov 01 16:20:29.566679 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:29.566628 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-9a482b/auth-test-model-9a482b-kserve-6c554dbfd9-wqk65" podStartSLOduration=2.566613315 podStartE2EDuration="2.566613315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:20:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:20:29.564853694 +0000 UTC m=+1962.896591344" watchObservedRunningTime="2026-11-01 16:20:29.566613315 +0000 UTC m=+1962.898350965" Nov 01 16:20:29.986462 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:29.986424 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-9a482b/auth-test-model-9a482b-kserve-6c554dbfd9-wqk65" Nov 01 16:20:32.350144 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:32.350107 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b"] Nov 01 16:20:32.351659 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:32.351631 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebf773ee-489a-418d-a031-327a467839c4" containerName="istio-proxy" Nov 01 16:20:32.357536 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:32.357511 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" Nov 01 16:20:32.364735 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:32.364704 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b"] Nov 01 16:20:32.432047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:32.432006 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-c78897-openshift-default-dockercfg-q2vpn\"" Nov 01 16:20:32.514290 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:32.506458 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b180ef5-70b3-4bf2-901b-db280a232bde-istio-podinfo\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b\" (UID: \"1b180ef5-70b3-4bf2-901b-db280a232bde\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" Nov 01 16:20:32.514290 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:32.506599 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkjlh\" (UniqueName: \"kubernetes.io/projected/1b180ef5-70b3-4bf2-901b-db280a232bde-kube-api-access-vkjlh\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b\" (UID: \"1b180ef5-70b3-4bf2-901b-db280a232bde\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" Nov 01 16:20:32.514290 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:32.506776 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b180ef5-70b3-4bf2-901b-db280a232bde-istio-envoy\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b\" (UID: \"1b180ef5-70b3-4bf2-901b-db280a232bde\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" Nov 01 16:20:32.514290 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:32.506942 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b180ef5-70b3-4bf2-901b-db280a232bde-credential-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b\" (UID: \"1b180ef5-70b3-4bf2-901b-db280a232bde\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" Nov 01 16:20:32.514290 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:32.507248 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b180ef5-70b3-4bf2-901b-db280a232bde-workload-certs\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b\" (UID: \"1b180ef5-70b3-4bf2-901b-db280a232bde\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" Nov 01 16:20:32.521372 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:32.520402 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b180ef5-70b3-4bf2-901b-db280a232bde-istio-data\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b\" (UID: \"1b180ef5-70b3-4bf2-901b-db280a232bde\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" Nov 01 16:20:32.521372 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:32.520548 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b180ef5-70b3-4bf2-901b-db280a232bde-istiod-ca-cert\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b\" (UID: \"1b180ef5-70b3-4bf2-901b-db280a232bde\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" Nov 01 16:20:32.521372 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:32.520590 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b180ef5-70b3-4bf2-901b-db280a232bde-istio-token\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b\" (UID: \"1b180ef5-70b3-4bf2-901b-db280a232bde\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" Nov 01 16:20:32.521372 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:32.520660 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b180ef5-70b3-4bf2-901b-db280a232bde-workload-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b\" (UID: \"1b180ef5-70b3-4bf2-901b-db280a232bde\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" Nov 01 16:20:32.634904 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:32.633292 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b180ef5-70b3-4bf2-901b-db280a232bde-istio-envoy\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b\" (UID: \"1b180ef5-70b3-4bf2-901b-db280a232bde\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" Nov 01 16:20:32.634904 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:32.633416 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b180ef5-70b3-4bf2-901b-db280a232bde-credential-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b\" (UID: \"1b180ef5-70b3-4bf2-901b-db280a232bde\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" Nov 01 16:20:32.634904 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:32.633598 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b180ef5-70b3-4bf2-901b-db280a232bde-workload-certs\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b\" (UID: \"1b180ef5-70b3-4bf2-901b-db280a232bde\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" Nov 01 16:20:32.634904 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:32.633668 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b180ef5-70b3-4bf2-901b-db280a232bde-istio-data\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b\" (UID: \"1b180ef5-70b3-4bf2-901b-db280a232bde\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" Nov 01 16:20:32.634904 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:32.633780 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b180ef5-70b3-4bf2-901b-db280a232bde-istiod-ca-cert\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b\" (UID: \"1b180ef5-70b3-4bf2-901b-db280a232bde\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" Nov 01 16:20:32.634904 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:32.633824 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b180ef5-70b3-4bf2-901b-db280a232bde-istio-token\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b\" (UID: \"1b180ef5-70b3-4bf2-901b-db280a232bde\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" Nov 01 16:20:32.634904 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:32.633887 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b180ef5-70b3-4bf2-901b-db280a232bde-workload-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b\" (UID: \"1b180ef5-70b3-4bf2-901b-db280a232bde\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" Nov 01 16:20:32.634904 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:32.633930 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b180ef5-70b3-4bf2-901b-db280a232bde-istio-podinfo\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b\" (UID: \"1b180ef5-70b3-4bf2-901b-db280a232bde\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" Nov 01 16:20:32.634904 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:32.633973 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkjlh\" (UniqueName: \"kubernetes.io/projected/1b180ef5-70b3-4bf2-901b-db280a232bde-kube-api-access-vkjlh\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b\" (UID: \"1b180ef5-70b3-4bf2-901b-db280a232bde\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" Nov 01 16:20:32.641037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:32.635929 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b180ef5-70b3-4bf2-901b-db280a232bde-workload-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b\" (UID: \"1b180ef5-70b3-4bf2-901b-db280a232bde\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" Nov 01 16:20:32.641037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:32.636251 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b180ef5-70b3-4bf2-901b-db280a232bde-workload-certs\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b\" (UID: \"1b180ef5-70b3-4bf2-901b-db280a232bde\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" Nov 01 16:20:32.641037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:32.636538 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b180ef5-70b3-4bf2-901b-db280a232bde-credential-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b\" (UID: \"1b180ef5-70b3-4bf2-901b-db280a232bde\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" Nov 01 16:20:32.641037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:32.636802 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b180ef5-70b3-4bf2-901b-db280a232bde-istio-data\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b\" (UID: \"1b180ef5-70b3-4bf2-901b-db280a232bde\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" Nov 01 16:20:32.641037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:32.637555 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b180ef5-70b3-4bf2-901b-db280a232bde-istiod-ca-cert\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b\" (UID: \"1b180ef5-70b3-4bf2-901b-db280a232bde\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" Nov 01 16:20:32.698047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:32.672115 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b180ef5-70b3-4bf2-901b-db280a232bde-istio-token\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b\" (UID: \"1b180ef5-70b3-4bf2-901b-db280a232bde\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" Nov 01 16:20:32.698047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:32.672678 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b180ef5-70b3-4bf2-901b-db280a232bde-istio-podinfo\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b\" (UID: \"1b180ef5-70b3-4bf2-901b-db280a232bde\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" Nov 01 16:20:32.698047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:32.676005 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkjlh\" (UniqueName: \"kubernetes.io/projected/1b180ef5-70b3-4bf2-901b-db280a232bde-kube-api-access-vkjlh\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b\" (UID: \"1b180ef5-70b3-4bf2-901b-db280a232bde\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" Nov 01 16:20:32.698047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:32.677430 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b180ef5-70b3-4bf2-901b-db280a232bde-istio-envoy\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b\" (UID: \"1b180ef5-70b3-4bf2-901b-db280a232bde\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" Nov 01 16:20:32.698047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:32.689596 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" Nov 01 16:20:33.126080 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:20:33.125905 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b180ef5_70b3_4bf2_901b_db280a232bde.slice/crio-c44c0544396c5b5a6d6619a9e8b5923d66ff56df09eebce10cd1c52903e97a79 WatchSource:0}: Error finding container c44c0544396c5b5a6d6619a9e8b5923d66ff56df09eebce10cd1c52903e97a79: Status 404 returned error can't find the container with id c44c0544396c5b5a6d6619a9e8b5923d66ff56df09eebce10cd1c52903e97a79 Nov 01 16:20:33.126337 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:33.126310 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b"] Nov 01 16:20:33.129449 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:33.129419 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:20:33.129535 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:33.129483 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:20:33.129535 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:33.129515 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:20:33.670061 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:33.667729 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" event={"ID":"1b180ef5-70b3-4bf2-901b-db280a232bde","Type":"ContainerStarted","Data":"63409c055dbbf5803b5aac62cd48411547732fa5158dbd6d67f1edcb277d5d0a"} Nov 01 16:20:33.670061 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:33.667817 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" event={"ID":"1b180ef5-70b3-4bf2-901b-db280a232bde","Type":"ContainerStarted","Data":"c44c0544396c5b5a6d6619a9e8b5923d66ff56df09eebce10cd1c52903e97a79"} Nov 01 16:20:33.693738 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:33.693711 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" Nov 01 16:20:34.697513 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:34.697479 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" Nov 01 16:20:34.719305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:34.719255 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" podStartSLOduration=2.7192414830000002 podStartE2EDuration="2.719241483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:20:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:20:33.693077787 +0000 UTC m=+1967.024815438" watchObservedRunningTime="2026-11-01 16:20:34.719241483 +0000 UTC m=+1968.050979203" Nov 01 16:20:35.684644 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:35.684610 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" Nov 01 16:20:35.685652 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:35.685632 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" Nov 01 16:20:37.986528 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:37.986483 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-9a482b/auth-test-model-9a482b-kserve-6c554dbfd9-wqk65" Nov 01 16:20:37.999456 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:37.999430 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-9a482b/auth-test-model-9a482b-kserve-6c554dbfd9-wqk65" Nov 01 16:20:38.723313 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:38.723282 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-9a482b/auth-test-model-9a482b-kserve-6c554dbfd9-wqk65" Nov 01 16:20:43.341804 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:43.341744 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-04f248/test-model-04f248-kserve-7cf9d4bf7f-kd4rz" podUID="3860672d-226a-47fd-a812-cca575bea518" containerName="main" containerID="cri-o://deae462eb6fc6d13fb764813e44bcf0465484eb57a238e2456ea84b8991599b2" gracePeriod=45 Nov 01 16:20:43.835744 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:43.835713 4363 generic.go:356] "Generic (PLEG): container finished" podID="3860672d-226a-47fd-a812-cca575bea518" containerID="deae462eb6fc6d13fb764813e44bcf0465484eb57a238e2456ea84b8991599b2" exitCode=0 Nov 01 16:20:43.835941 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:43.835795 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-04f248/test-model-04f248-kserve-7cf9d4bf7f-kd4rz" event={"ID":"3860672d-226a-47fd-a812-cca575bea518","Type":"ContainerDied","Data":"deae462eb6fc6d13fb764813e44bcf0465484eb57a238e2456ea84b8991599b2"} Nov 01 16:20:43.835941 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:43.835850 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-04f248/test-model-04f248-kserve-7cf9d4bf7f-kd4rz" event={"ID":"3860672d-226a-47fd-a812-cca575bea518","Type":"ContainerDied","Data":"59f87ca26ae0f29809650979c5187a46246473e166dec09de104bc253ab4c4c0"} Nov 01 16:20:43.835941 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:43.835866 4363 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59f87ca26ae0f29809650979c5187a46246473e166dec09de104bc253ab4c4c0" Nov 01 16:20:43.837448 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:43.837428 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-04f248/test-model-04f248-kserve-7cf9d4bf7f-kd4rz" Nov 01 16:20:43.877927 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:43.877886 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3860672d-226a-47fd-a812-cca575bea518-kube-api-access-t5x9z\" (UniqueName: \"kubernetes.io/projected/3860672d-226a-47fd-a812-cca575bea518-kube-api-access-t5x9z\") pod \"3860672d-226a-47fd-a812-cca575bea518\" (UID: \"3860672d-226a-47fd-a812-cca575bea518\") " Nov 01 16:20:43.878155 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:43.877990 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3860672d-226a-47fd-a812-cca575bea518-home\" (UniqueName: \"kubernetes.io/empty-dir/3860672d-226a-47fd-a812-cca575bea518-home\") pod \"3860672d-226a-47fd-a812-cca575bea518\" (UID: \"3860672d-226a-47fd-a812-cca575bea518\") " Nov 01 16:20:43.878155 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:43.878096 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3860672d-226a-47fd-a812-cca575bea518-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3860672d-226a-47fd-a812-cca575bea518-dshm\") pod \"3860672d-226a-47fd-a812-cca575bea518\" (UID: \"3860672d-226a-47fd-a812-cca575bea518\") " Nov 01 16:20:43.878155 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:43.878125 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3860672d-226a-47fd-a812-cca575bea518-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3860672d-226a-47fd-a812-cca575bea518-tmp-dir\") pod \"3860672d-226a-47fd-a812-cca575bea518\" (UID: \"3860672d-226a-47fd-a812-cca575bea518\") " Nov 01 16:20:43.878356 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:43.878211 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3860672d-226a-47fd-a812-cca575bea518-tls-certs\" (UniqueName: \"kubernetes.io/secret/3860672d-226a-47fd-a812-cca575bea518-tls-certs\") pod \"3860672d-226a-47fd-a812-cca575bea518\" (UID: \"3860672d-226a-47fd-a812-cca575bea518\") " Nov 01 16:20:43.878356 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:43.878242 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3860672d-226a-47fd-a812-cca575bea518-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3860672d-226a-47fd-a812-cca575bea518-model-cache\") pod \"3860672d-226a-47fd-a812-cca575bea518\" (UID: \"3860672d-226a-47fd-a812-cca575bea518\") " Nov 01 16:20:43.878356 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:43.878283 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3860672d-226a-47fd-a812-cca575bea518-home" pod "3860672d-226a-47fd-a812-cca575bea518" (UID: "3860672d-226a-47fd-a812-cca575bea518"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:20:43.878679 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:43.878652 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3860672d-226a-47fd-a812-cca575bea518-tmp-dir" pod "3860672d-226a-47fd-a812-cca575bea518" (UID: "3860672d-226a-47fd-a812-cca575bea518"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:20:43.878870 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:43.878851 4363 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3860672d-226a-47fd-a812-cca575bea518-home\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:43.878959 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:43.878875 4363 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3860672d-226a-47fd-a812-cca575bea518-tmp-dir\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:43.879003 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:43.878951 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3860672d-226a-47fd-a812-cca575bea518-model-cache" pod "3860672d-226a-47fd-a812-cca575bea518" (UID: "3860672d-226a-47fd-a812-cca575bea518"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:20:43.890560 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:43.885847 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3860672d-226a-47fd-a812-cca575bea518-dshm" pod "3860672d-226a-47fd-a812-cca575bea518" (UID: "3860672d-226a-47fd-a812-cca575bea518"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:20:43.890907 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:43.890887 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3860672d-226a-47fd-a812-cca575bea518-tls-certs" pod "3860672d-226a-47fd-a812-cca575bea518" (UID: "3860672d-226a-47fd-a812-cca575bea518"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:20:43.891670 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:43.891638 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3860672d-226a-47fd-a812-cca575bea518-kube-api-access-t5x9z" pod "3860672d-226a-47fd-a812-cca575bea518" (UID: "3860672d-226a-47fd-a812-cca575bea518"). InnerVolumeSpecName "kube-api-access-t5x9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:20:43.987572 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:43.983009 4363 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3860672d-226a-47fd-a812-cca575bea518-dshm\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:43.987572 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:43.983064 4363 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3860672d-226a-47fd-a812-cca575bea518-tls-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:43.987572 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:43.983078 4363 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3860672d-226a-47fd-a812-cca575bea518-model-cache\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:43.987572 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:43.983094 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5x9z\" (UniqueName: \"kubernetes.io/projected/3860672d-226a-47fd-a812-cca575bea518-kube-api-access-t5x9z\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:44.797638 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:44.797597 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-04f248/test-model-04f248-kserve-7cf9d4bf7f-kd4rz" podUID="3860672d-226a-47fd-a812-cca575bea518" containerName="main" probeResult="failure" output="Get \"https://10.134.0.186:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:20:44.870258 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:44.870229 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-04f248/test-model-04f248-kserve-7cf9d4bf7f-kd4rz" Nov 01 16:20:44.895565 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:44.893378 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-04f248/test-model-04f248-kserve-7cf9d4bf7f-kd4rz"] Nov 01 16:20:44.899057 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:44.898473 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-04f248/test-model-04f248-kserve-7cf9d4bf7f-kd4rz"] Nov 01 16:20:46.804669 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:46.804636 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3860672d-226a-47fd-a812-cca575bea518" path="/var/lib/kubelet/pods/3860672d-226a-47fd-a812-cca575bea518/volumes" Nov 01 16:20:48.311760 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:20:48.311722 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:20:49.705703 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:49.705665 4363 scope.go:122] "RemoveContainer" containerID="deae462eb6fc6d13fb764813e44bcf0465484eb57a238e2456ea84b8991599b2" Nov 01 16:20:53.903781 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:53.903745 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-04f248-6f57f7dc7-snmtt"] Nov 01 16:20:53.925063 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:53.913694 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-04f248-65b9669fcf-7fzhs"] Nov 01 16:20:56.614483 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:56.614446 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-f01de5/auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt"] Nov 01 16:20:56.615812 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:56.615781 4363 cpu_manager.go:414] "containerMap: removing container" podUID="3860672d-226a-47fd-a812-cca575bea518" containerName="main" Nov 01 16:20:56.615812 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:56.615810 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3860672d-226a-47fd-a812-cca575bea518" containerName="main" Nov 01 16:20:56.616291 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:56.616059 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="3860672d-226a-47fd-a812-cca575bea518" containerName="main" Nov 01 16:20:56.627661 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:56.627638 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-f01de5/auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt" Nov 01 16:20:56.631659 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:56.631624 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-f01de5/auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt"] Nov 01 16:20:56.631804 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:56.631692 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-f01de5\"/\"default-dockercfg-mjz9s\"" Nov 01 16:20:56.633754 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:56.633735 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-f01de5\"/\"auth-test-model-f01de5-kserve-self-signed-certs\"" Nov 01 16:20:56.635359 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:56.635343 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-f01de5\"/\"openshift-service-ca.crt\"" Nov 01 16:20:56.635506 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:56.635426 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-f01de5\"/\"kube-root-ca.crt\"" Nov 01 16:20:56.672584 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:56.672552 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfkdh\" (UniqueName: \"kubernetes.io/projected/03bcbe9c-db2f-4915-920f-3252117ac51a-kube-api-access-rfkdh\") pod \"auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt\" (UID: \"03bcbe9c-db2f-4915-920f-3252117ac51a\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt" Nov 01 16:20:56.672755 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:56.672677 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03bcbe9c-db2f-4915-920f-3252117ac51a-tmp-dir\") pod \"auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt\" (UID: \"03bcbe9c-db2f-4915-920f-3252117ac51a\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt" Nov 01 16:20:56.672931 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:56.672912 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03bcbe9c-db2f-4915-920f-3252117ac51a-model-cache\") pod \"auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt\" (UID: \"03bcbe9c-db2f-4915-920f-3252117ac51a\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt" Nov 01 16:20:56.673011 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:56.672957 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03bcbe9c-db2f-4915-920f-3252117ac51a-dshm\") pod \"auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt\" (UID: \"03bcbe9c-db2f-4915-920f-3252117ac51a\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt" Nov 01 16:20:56.673086 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:56.673069 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03bcbe9c-db2f-4915-920f-3252117ac51a-tls-certs\") pod \"auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt\" (UID: \"03bcbe9c-db2f-4915-920f-3252117ac51a\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt" Nov 01 16:20:56.673309 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:56.673164 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03bcbe9c-db2f-4915-920f-3252117ac51a-home\") pod \"auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt\" (UID: \"03bcbe9c-db2f-4915-920f-3252117ac51a\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt" Nov 01 16:20:56.774621 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:56.774589 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03bcbe9c-db2f-4915-920f-3252117ac51a-tmp-dir\") pod \"auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt\" (UID: \"03bcbe9c-db2f-4915-920f-3252117ac51a\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt" Nov 01 16:20:56.774806 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:56.774748 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03bcbe9c-db2f-4915-920f-3252117ac51a-model-cache\") pod \"auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt\" (UID: \"03bcbe9c-db2f-4915-920f-3252117ac51a\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt" Nov 01 16:20:56.774806 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:56.774788 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03bcbe9c-db2f-4915-920f-3252117ac51a-dshm\") pod \"auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt\" (UID: \"03bcbe9c-db2f-4915-920f-3252117ac51a\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt" Nov 01 16:20:56.774948 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:56.774855 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03bcbe9c-db2f-4915-920f-3252117ac51a-tls-certs\") pod \"auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt\" (UID: \"03bcbe9c-db2f-4915-920f-3252117ac51a\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt" Nov 01 16:20:56.775011 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:56.774944 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03bcbe9c-db2f-4915-920f-3252117ac51a-home\") pod \"auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt\" (UID: \"03bcbe9c-db2f-4915-920f-3252117ac51a\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt" Nov 01 16:20:56.775011 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:56.775061 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03bcbe9c-db2f-4915-920f-3252117ac51a-tmp-dir\") pod \"auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt\" (UID: \"03bcbe9c-db2f-4915-920f-3252117ac51a\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt" Nov 01 16:20:56.775011 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:56.775090 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfkdh\" (UniqueName: \"kubernetes.io/projected/03bcbe9c-db2f-4915-920f-3252117ac51a-kube-api-access-rfkdh\") pod \"auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt\" (UID: \"03bcbe9c-db2f-4915-920f-3252117ac51a\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt" Nov 01 16:20:56.775011 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:56.775221 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03bcbe9c-db2f-4915-920f-3252117ac51a-model-cache\") pod \"auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt\" (UID: \"03bcbe9c-db2f-4915-920f-3252117ac51a\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt" Nov 01 16:20:56.775670 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:56.775396 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03bcbe9c-db2f-4915-920f-3252117ac51a-home\") pod \"auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt\" (UID: \"03bcbe9c-db2f-4915-920f-3252117ac51a\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt" Nov 01 16:20:56.815038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:56.812752 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03bcbe9c-db2f-4915-920f-3252117ac51a-tls-certs\") pod \"auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt\" (UID: \"03bcbe9c-db2f-4915-920f-3252117ac51a\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt" Nov 01 16:20:56.839041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:56.823542 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03bcbe9c-db2f-4915-920f-3252117ac51a-dshm\") pod \"auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt\" (UID: \"03bcbe9c-db2f-4915-920f-3252117ac51a\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt" Nov 01 16:20:56.926608 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:56.926484 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfkdh\" (UniqueName: \"kubernetes.io/projected/03bcbe9c-db2f-4915-920f-3252117ac51a-kube-api-access-rfkdh\") pod \"auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt\" (UID: \"03bcbe9c-db2f-4915-920f-3252117ac51a\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt" Nov 01 16:20:56.952591 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:56.952533 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-f01de5/auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt" Nov 01 16:20:57.291806 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:57.291777 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-f01de5/auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt"] Nov 01 16:20:57.293195 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:20:57.293170 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03bcbe9c_db2f_4915_920f_3252117ac51a.slice/crio-b9b27e8445edada8a6f3fbb92e0e0d84e7ac35183181ab9d7a5031d56ea9f70c WatchSource:0}: Error finding container b9b27e8445edada8a6f3fbb92e0e0d84e7ac35183181ab9d7a5031d56ea9f70c: Status 404 returned error can't find the container with id b9b27e8445edada8a6f3fbb92e0e0d84e7ac35183181ab9d7a5031d56ea9f70c Nov 01 16:20:58.024090 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:58.024047 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-f01de5/auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt" event={"ID":"03bcbe9c-db2f-4915-920f-3252117ac51a","Type":"ContainerStarted","Data":"06e7bc5050b66c8deb38ea733de1acaff0af52c8e41f8aa6ba096d9970ee19fa"} Nov 01 16:20:58.024601 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:58.024099 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-f01de5/auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt" event={"ID":"03bcbe9c-db2f-4915-920f-3252117ac51a","Type":"ContainerStarted","Data":"b9b27e8445edada8a6f3fbb92e0e0d84e7ac35183181ab9d7a5031d56ea9f70c"} Nov 01 16:20:58.041998 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:58.041949 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-f01de5/auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt" podStartSLOduration=2.041935004 podStartE2EDuration="2.041935004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:20:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:20:58.038847327 +0000 UTC m=+1991.370585011" watchObservedRunningTime="2026-11-01 16:20:58.041935004 +0000 UTC m=+1991.373672701" Nov 01 16:20:58.904036 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:58.903983 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-04f248-6f57f7dc7-snmtt" podUID="6c62027d-8353-4200-82f7-365988964d51" containerName="payload-processing" containerID="cri-o://8b28ce784160139d75f45aa92a1e35273a7aa3706ffecb8bd813a31e2e7a52a9" gracePeriod=25 Nov 01 16:20:58.914086 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:58.914055 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04f248-65b9669fcf-7fzhs" podUID="e923b2d1-5a22-4eca-973d-6a94650574b4" containerName="payload-pre-processing" containerID="cri-o://f6d385b745181d874ca57a877ccc4eeccf0d85a85c48699a2b64864c3a0c2fe0" gracePeriod=25 Nov 01 16:20:58.953300 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:20:58.953277 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-f01de5/auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt" Nov 01 16:21:01.488393 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:01.488355 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04f248-65b9669fcf-7fzhs" podUID="e923b2d1-5a22-4eca-973d-6a94650574b4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:21:01.488815 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:01.488353 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-04f248-6f57f7dc7-snmtt" podUID="6c62027d-8353-4200-82f7-365988964d51" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:21:05.192050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:05.191985 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-04f248-668fc47648-hptb8"] Nov 01 16:21:05.194852 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:05.192244 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04f248-668fc47648-hptb8" podUID="67731def-6473-4c83-a367-6960146d4907" containerName="maas-api" containerID="cri-o://af61302b33c10503b5cddee91256dfa122c474e2923d7ccf92ffce91ba14d3da" gracePeriod=30 Nov 01 16:21:05.909361 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:05.909334 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04f248-668fc47648-hptb8" Nov 01 16:21:05.966235 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:05.966199 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67731def-6473-4c83-a367-6960146d4907-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67731def-6473-4c83-a367-6960146d4907-maas-api-tls\") pod \"67731def-6473-4c83-a367-6960146d4907\" (UID: \"67731def-6473-4c83-a367-6960146d4907\") " Nov 01 16:21:05.966387 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:05.966298 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67731def-6473-4c83-a367-6960146d4907-kube-api-access-kbbv2\" (UniqueName: \"kubernetes.io/projected/67731def-6473-4c83-a367-6960146d4907-kube-api-access-kbbv2\") pod \"67731def-6473-4c83-a367-6960146d4907\" (UID: \"67731def-6473-4c83-a367-6960146d4907\") " Nov 01 16:21:05.966387 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:05.966349 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67731def-6473-4c83-a367-6960146d4907-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67731def-6473-4c83-a367-6960146d4907-openshift-service-ca\") pod \"67731def-6473-4c83-a367-6960146d4907\" (UID: \"67731def-6473-4c83-a367-6960146d4907\") " Nov 01 16:21:05.966686 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:05.966665 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67731def-6473-4c83-a367-6960146d4907-metrics-certs\" (UniqueName: \"kubernetes.io/secret/67731def-6473-4c83-a367-6960146d4907-metrics-certs\") pod \"67731def-6473-4c83-a367-6960146d4907\" (UID: \"67731def-6473-4c83-a367-6960146d4907\") " Nov 01 16:21:05.966797 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:05.966762 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67731def-6473-4c83-a367-6960146d4907-openshift-service-ca" pod "67731def-6473-4c83-a367-6960146d4907" (UID: "67731def-6473-4c83-a367-6960146d4907"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:21:05.967712 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:05.967682 4363 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67731def-6473-4c83-a367-6960146d4907-openshift-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:05.968863 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:05.968840 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67731def-6473-4c83-a367-6960146d4907-metrics-certs" pod "67731def-6473-4c83-a367-6960146d4907" (UID: "67731def-6473-4c83-a367-6960146d4907"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:21:05.969082 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:05.969054 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67731def-6473-4c83-a367-6960146d4907-maas-api-tls" pod "67731def-6473-4c83-a367-6960146d4907" (UID: "67731def-6473-4c83-a367-6960146d4907"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:21:05.969181 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:05.969105 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67731def-6473-4c83-a367-6960146d4907-kube-api-access-kbbv2" pod "67731def-6473-4c83-a367-6960146d4907" (UID: "67731def-6473-4c83-a367-6960146d4907"). InnerVolumeSpecName "kube-api-access-kbbv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:21:06.069145 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:06.069109 4363 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67731def-6473-4c83-a367-6960146d4907-metrics-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:06.069145 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:06.069139 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67731def-6473-4c83-a367-6960146d4907-maas-api-tls\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:06.069145 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:06.069151 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbbv2\" (UniqueName: \"kubernetes.io/projected/67731def-6473-4c83-a367-6960146d4907-kube-api-access-kbbv2\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:06.087985 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:06.087955 4363 generic.go:356] "Generic (PLEG): container finished" podID="67731def-6473-4c83-a367-6960146d4907" containerID="af61302b33c10503b5cddee91256dfa122c474e2923d7ccf92ffce91ba14d3da" exitCode=0 Nov 01 16:21:06.088213 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:06.088005 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04f248-668fc47648-hptb8" Nov 01 16:21:06.088213 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:06.088046 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04f248-668fc47648-hptb8" event={"ID":"67731def-6473-4c83-a367-6960146d4907","Type":"ContainerDied","Data":"af61302b33c10503b5cddee91256dfa122c474e2923d7ccf92ffce91ba14d3da"} Nov 01 16:21:06.088213 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:06.088106 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04f248-668fc47648-hptb8" event={"ID":"67731def-6473-4c83-a367-6960146d4907","Type":"ContainerDied","Data":"bea1e34ea5c7247d57a679217b6c1c05baa943d6b49df03c15881f11ce632453"} Nov 01 16:21:06.088213 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:06.088139 4363 scope.go:122] "RemoveContainer" containerID="af61302b33c10503b5cddee91256dfa122c474e2923d7ccf92ffce91ba14d3da" Nov 01 16:21:06.100796 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:06.100776 4363 scope.go:122] "RemoveContainer" containerID="af61302b33c10503b5cddee91256dfa122c474e2923d7ccf92ffce91ba14d3da" Nov 01 16:21:06.101096 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:21:06.101069 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af61302b33c10503b5cddee91256dfa122c474e2923d7ccf92ffce91ba14d3da\": container with ID starting with af61302b33c10503b5cddee91256dfa122c474e2923d7ccf92ffce91ba14d3da not found: ID does not exist" containerID="af61302b33c10503b5cddee91256dfa122c474e2923d7ccf92ffce91ba14d3da" Nov 01 16:21:06.101206 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:06.101107 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af61302b33c10503b5cddee91256dfa122c474e2923d7ccf92ffce91ba14d3da"} err="failed to get container status \"af61302b33c10503b5cddee91256dfa122c474e2923d7ccf92ffce91ba14d3da\": rpc error: code = NotFound desc = could not find container \"af61302b33c10503b5cddee91256dfa122c474e2923d7ccf92ffce91ba14d3da\": container with ID starting with af61302b33c10503b5cddee91256dfa122c474e2923d7ccf92ffce91ba14d3da not found: ID does not exist" Nov 01 16:21:06.117052 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:06.113936 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-04f248-668fc47648-hptb8"] Nov 01 16:21:06.118300 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:06.118280 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-04f248-668fc47648-hptb8"] Nov 01 16:21:06.804599 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:06.804560 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67731def-6473-4c83-a367-6960146d4907" path="/var/lib/kubelet/pods/67731def-6473-4c83-a367-6960146d4907/volumes" Nov 01 16:21:06.953013 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:06.952980 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-f01de5/auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt" Nov 01 16:21:06.966172 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:06.966147 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-f01de5/auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt" Nov 01 16:21:07.110285 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:07.110251 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-f01de5/auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt" Nov 01 16:21:07.584679 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:07.584599 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl"] Nov 01 16:21:07.585445 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:07.585428 4363 cpu_manager.go:414] "containerMap: removing container" podUID="67731def-6473-4c83-a367-6960146d4907" containerName="maas-api" Nov 01 16:21:07.585495 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:07.585446 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67731def-6473-4c83-a367-6960146d4907" containerName="maas-api" Nov 01 16:21:07.585629 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:07.585619 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="67731def-6473-4c83-a367-6960146d4907" containerName="maas-api" Nov 01 16:21:07.590629 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:07.590602 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" Nov 01 16:21:07.599766 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:07.599571 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-c8d121-openshift-default-dockercfg-rn95k\"" Nov 01 16:21:07.601690 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:07.601660 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl"] Nov 01 16:21:07.701864 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:07.701824 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq94m\" (UniqueName: \"kubernetes.io/projected/6120a22c-a6cc-4cf9-a478-7566b8bc4694-kube-api-access-lq94m\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl\" (UID: \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" Nov 01 16:21:07.702048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:07.701894 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6120a22c-a6cc-4cf9-a478-7566b8bc4694-istio-token\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl\" (UID: \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" Nov 01 16:21:07.702127 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:07.702049 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6120a22c-a6cc-4cf9-a478-7566b8bc4694-credential-socket\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl\" (UID: \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" Nov 01 16:21:07.702183 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:07.702165 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6120a22c-a6cc-4cf9-a478-7566b8bc4694-workload-certs\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl\" (UID: \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" Nov 01 16:21:07.702284 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:07.702269 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6120a22c-a6cc-4cf9-a478-7566b8bc4694-istiod-ca-cert\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl\" (UID: \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" Nov 01 16:21:07.702398 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:07.702382 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6120a22c-a6cc-4cf9-a478-7566b8bc4694-istio-envoy\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl\" (UID: \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" Nov 01 16:21:07.702527 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:07.702512 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6120a22c-a6cc-4cf9-a478-7566b8bc4694-workload-socket\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl\" (UID: \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" Nov 01 16:21:07.702564 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:07.702549 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6120a22c-a6cc-4cf9-a478-7566b8bc4694-istio-podinfo\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl\" (UID: \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" Nov 01 16:21:07.702601 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:07.702579 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6120a22c-a6cc-4cf9-a478-7566b8bc4694-istio-data\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl\" (UID: \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" Nov 01 16:21:07.803406 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:07.803372 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6120a22c-a6cc-4cf9-a478-7566b8bc4694-istio-envoy\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl\" (UID: \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" Nov 01 16:21:07.803567 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:07.803472 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6120a22c-a6cc-4cf9-a478-7566b8bc4694-workload-socket\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl\" (UID: \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" Nov 01 16:21:07.803567 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:07.803495 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6120a22c-a6cc-4cf9-a478-7566b8bc4694-istio-podinfo\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl\" (UID: \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" Nov 01 16:21:07.803567 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:07.803513 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6120a22c-a6cc-4cf9-a478-7566b8bc4694-istio-data\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl\" (UID: \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" Nov 01 16:21:07.803567 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:07.803547 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq94m\" (UniqueName: \"kubernetes.io/projected/6120a22c-a6cc-4cf9-a478-7566b8bc4694-kube-api-access-lq94m\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl\" (UID: \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" Nov 01 16:21:07.803824 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:07.803585 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6120a22c-a6cc-4cf9-a478-7566b8bc4694-istio-token\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl\" (UID: \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" Nov 01 16:21:07.803824 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:07.803632 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6120a22c-a6cc-4cf9-a478-7566b8bc4694-credential-socket\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl\" (UID: \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" Nov 01 16:21:07.803824 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:07.803719 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6120a22c-a6cc-4cf9-a478-7566b8bc4694-workload-certs\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl\" (UID: \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" Nov 01 16:21:07.803824 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:07.803813 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6120a22c-a6cc-4cf9-a478-7566b8bc4694-istiod-ca-cert\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl\" (UID: \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" Nov 01 16:21:07.804148 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:07.803918 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6120a22c-a6cc-4cf9-a478-7566b8bc4694-workload-socket\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl\" (UID: \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" Nov 01 16:21:07.804148 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:07.804129 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6120a22c-a6cc-4cf9-a478-7566b8bc4694-credential-socket\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl\" (UID: \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" Nov 01 16:21:07.804274 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:07.804253 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6120a22c-a6cc-4cf9-a478-7566b8bc4694-istio-data\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl\" (UID: \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" Nov 01 16:21:07.804332 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:07.804283 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6120a22c-a6cc-4cf9-a478-7566b8bc4694-workload-certs\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl\" (UID: \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" Nov 01 16:21:07.804560 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:07.804536 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6120a22c-a6cc-4cf9-a478-7566b8bc4694-istiod-ca-cert\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl\" (UID: \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" Nov 01 16:21:07.806423 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:07.806397 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6120a22c-a6cc-4cf9-a478-7566b8bc4694-istio-envoy\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl\" (UID: \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" Nov 01 16:21:07.806780 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:07.806605 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6120a22c-a6cc-4cf9-a478-7566b8bc4694-istio-podinfo\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl\" (UID: \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" Nov 01 16:21:07.810983 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:07.810963 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq94m\" (UniqueName: \"kubernetes.io/projected/6120a22c-a6cc-4cf9-a478-7566b8bc4694-kube-api-access-lq94m\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl\" (UID: \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" Nov 01 16:21:07.811119 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:07.810991 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6120a22c-a6cc-4cf9-a478-7566b8bc4694-istio-token\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl\" (UID: \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" Nov 01 16:21:07.909158 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:07.909129 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" Nov 01 16:21:08.090150 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:08.090116 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw"] Nov 01 16:21:08.108898 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:08.108478 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw"] Nov 01 16:21:08.108898 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:08.108596 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" Nov 01 16:21:08.113382 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:08.113252 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-383fde-openshift-default-dockercfg-lxnqs\"" Nov 01 16:21:08.233480 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:08.233401 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f975360-c532-4eb2-aae5-5233d03d027f-istio-token\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw\" (UID: \"3f975360-c532-4eb2-aae5-5233d03d027f\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" Nov 01 16:21:08.233645 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:08.233482 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f975360-c532-4eb2-aae5-5233d03d027f-istio-data\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw\" (UID: \"3f975360-c532-4eb2-aae5-5233d03d027f\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" Nov 01 16:21:08.233645 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:08.233565 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f975360-c532-4eb2-aae5-5233d03d027f-istio-podinfo\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw\" (UID: \"3f975360-c532-4eb2-aae5-5233d03d027f\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" Nov 01 16:21:08.233794 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:08.233667 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f975360-c532-4eb2-aae5-5233d03d027f-istiod-ca-cert\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw\" (UID: \"3f975360-c532-4eb2-aae5-5233d03d027f\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" Nov 01 16:21:08.233794 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:08.233734 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f975360-c532-4eb2-aae5-5233d03d027f-credential-socket\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw\" (UID: \"3f975360-c532-4eb2-aae5-5233d03d027f\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" Nov 01 16:21:08.233926 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:08.233805 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f975360-c532-4eb2-aae5-5233d03d027f-workload-certs\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw\" (UID: \"3f975360-c532-4eb2-aae5-5233d03d027f\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" Nov 01 16:21:08.233926 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:08.233837 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f975360-c532-4eb2-aae5-5233d03d027f-workload-socket\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw\" (UID: \"3f975360-c532-4eb2-aae5-5233d03d027f\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" Nov 01 16:21:08.233926 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:08.233860 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8jwg\" (UniqueName: \"kubernetes.io/projected/3f975360-c532-4eb2-aae5-5233d03d027f-kube-api-access-z8jwg\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw\" (UID: \"3f975360-c532-4eb2-aae5-5233d03d027f\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" Nov 01 16:21:08.233926 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:08.233900 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f975360-c532-4eb2-aae5-5233d03d027f-istio-envoy\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw\" (UID: \"3f975360-c532-4eb2-aae5-5233d03d027f\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" Nov 01 16:21:08.257254 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:08.257224 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl"] Nov 01 16:21:08.257930 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:21:08.257902 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6120a22c_a6cc_4cf9_a478_7566b8bc4694.slice/crio-f8a65071ca4a23c7208256dfaace2eaf850f5815d6d068a5c285e6e01f2ce072 WatchSource:0}: Error finding container f8a65071ca4a23c7208256dfaace2eaf850f5815d6d068a5c285e6e01f2ce072: Status 404 returned error can't find the container with id f8a65071ca4a23c7208256dfaace2eaf850f5815d6d068a5c285e6e01f2ce072 Nov 01 16:21:08.261311 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:08.261265 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:21:08.261423 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:08.261347 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:21:08.261423 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:08.261392 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:21:08.335241 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:08.335213 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f975360-c532-4eb2-aae5-5233d03d027f-istio-token\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw\" (UID: \"3f975360-c532-4eb2-aae5-5233d03d027f\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" Nov 01 16:21:08.335357 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:08.335254 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f975360-c532-4eb2-aae5-5233d03d027f-istio-data\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw\" (UID: \"3f975360-c532-4eb2-aae5-5233d03d027f\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" Nov 01 16:21:08.335357 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:08.335315 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f975360-c532-4eb2-aae5-5233d03d027f-istio-podinfo\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw\" (UID: \"3f975360-c532-4eb2-aae5-5233d03d027f\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" Nov 01 16:21:08.335458 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:08.335375 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f975360-c532-4eb2-aae5-5233d03d027f-istiod-ca-cert\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw\" (UID: \"3f975360-c532-4eb2-aae5-5233d03d027f\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" Nov 01 16:21:08.335516 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:08.335461 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f975360-c532-4eb2-aae5-5233d03d027f-credential-socket\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw\" (UID: \"3f975360-c532-4eb2-aae5-5233d03d027f\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" Nov 01 16:21:08.335576 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:08.335520 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f975360-c532-4eb2-aae5-5233d03d027f-workload-certs\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw\" (UID: \"3f975360-c532-4eb2-aae5-5233d03d027f\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" Nov 01 16:21:08.335672 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:08.335651 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f975360-c532-4eb2-aae5-5233d03d027f-workload-socket\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw\" (UID: \"3f975360-c532-4eb2-aae5-5233d03d027f\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" Nov 01 16:21:08.335741 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:08.335697 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8jwg\" (UniqueName: \"kubernetes.io/projected/3f975360-c532-4eb2-aae5-5233d03d027f-kube-api-access-z8jwg\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw\" (UID: \"3f975360-c532-4eb2-aae5-5233d03d027f\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" Nov 01 16:21:08.335741 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:08.335730 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f975360-c532-4eb2-aae5-5233d03d027f-istio-envoy\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw\" (UID: \"3f975360-c532-4eb2-aae5-5233d03d027f\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" Nov 01 16:21:08.335871 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:08.335764 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f975360-c532-4eb2-aae5-5233d03d027f-istio-data\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw\" (UID: \"3f975360-c532-4eb2-aae5-5233d03d027f\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" Nov 01 16:21:08.335871 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:08.335830 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f975360-c532-4eb2-aae5-5233d03d027f-workload-certs\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw\" (UID: \"3f975360-c532-4eb2-aae5-5233d03d027f\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" Nov 01 16:21:08.336119 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:08.336097 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f975360-c532-4eb2-aae5-5233d03d027f-workload-socket\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw\" (UID: \"3f975360-c532-4eb2-aae5-5233d03d027f\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" Nov 01 16:21:08.336260 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:08.336240 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f975360-c532-4eb2-aae5-5233d03d027f-credential-socket\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw\" (UID: \"3f975360-c532-4eb2-aae5-5233d03d027f\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" Nov 01 16:21:08.336327 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:08.336314 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f975360-c532-4eb2-aae5-5233d03d027f-istiod-ca-cert\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw\" (UID: \"3f975360-c532-4eb2-aae5-5233d03d027f\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" Nov 01 16:21:08.337784 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:08.337764 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f975360-c532-4eb2-aae5-5233d03d027f-istio-podinfo\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw\" (UID: \"3f975360-c532-4eb2-aae5-5233d03d027f\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" Nov 01 16:21:08.338072 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:08.338055 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f975360-c532-4eb2-aae5-5233d03d027f-istio-envoy\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw\" (UID: \"3f975360-c532-4eb2-aae5-5233d03d027f\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" Nov 01 16:21:08.343582 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:08.343561 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f975360-c532-4eb2-aae5-5233d03d027f-istio-token\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw\" (UID: \"3f975360-c532-4eb2-aae5-5233d03d027f\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" Nov 01 16:21:08.343830 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:08.343811 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8jwg\" (UniqueName: \"kubernetes.io/projected/3f975360-c532-4eb2-aae5-5233d03d027f-kube-api-access-z8jwg\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw\" (UID: \"3f975360-c532-4eb2-aae5-5233d03d027f\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" Nov 01 16:21:08.437311 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:08.437275 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" Nov 01 16:21:08.738638 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:08.737446 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw"] Nov 01 16:21:08.743375 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:21:08.743306 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f975360_c532_4eb2_aae5_5233d03d027f.slice/crio-42f6204571f554fb52cb54bf96f98512d146bef69c3961304f54dd6088800b08 WatchSource:0}: Error finding container 42f6204571f554fb52cb54bf96f98512d146bef69c3961304f54dd6088800b08: Status 404 returned error can't find the container with id 42f6204571f554fb52cb54bf96f98512d146bef69c3961304f54dd6088800b08 Nov 01 16:21:08.750402 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:08.750361 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:21:08.750521 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:08.750451 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:21:08.750521 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:08.750495 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:21:09.131862 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:09.131822 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" event={"ID":"3f975360-c532-4eb2-aae5-5233d03d027f","Type":"ContainerStarted","Data":"acce3348d5e54a50e9dda71501d6beee9c94c42c9a8b964d75ed057c8beef1a6"} Nov 01 16:21:09.132501 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:09.132466 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" event={"ID":"3f975360-c532-4eb2-aae5-5233d03d027f","Type":"ContainerStarted","Data":"42f6204571f554fb52cb54bf96f98512d146bef69c3961304f54dd6088800b08"} Nov 01 16:21:09.133827 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:09.133801 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" event={"ID":"6120a22c-a6cc-4cf9-a478-7566b8bc4694","Type":"ContainerStarted","Data":"e04e57a9a871051c678b60133c6707b3c87ab05bda4280e70f540283e547b80f"} Nov 01 16:21:09.133929 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:09.133850 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" event={"ID":"6120a22c-a6cc-4cf9-a478-7566b8bc4694","Type":"ContainerStarted","Data":"f8a65071ca4a23c7208256dfaace2eaf850f5815d6d068a5c285e6e01f2ce072"} Nov 01 16:21:09.158606 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:09.158565 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" podStartSLOduration=1.158551796 podStartE2EDuration="1.158551796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:21:09.152802917 +0000 UTC m=+2002.484540568" watchObservedRunningTime="2026-11-01 16:21:09.158551796 +0000 UTC m=+2002.490289446" Nov 01 16:21:09.439815 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:09.439449 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" Nov 01 16:21:09.442501 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:09.442129 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Nov 01 16:21:09.442501 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:09.442189 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" podUID="3f975360-c532-4eb2-aae5-5233d03d027f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Nov 01 16:21:09.912033 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:09.911982 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" Nov 01 16:21:09.914127 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:09.914096 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Nov 01 16:21:09.914350 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:09.914322 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" podUID="6120a22c-a6cc-4cf9-a478-7566b8bc4694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Nov 01 16:21:10.441932 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:10.441423 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Nov 01 16:21:10.441932 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:10.441482 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" podUID="3f975360-c532-4eb2-aae5-5233d03d027f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Nov 01 16:21:10.979042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:10.951551 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Nov 01 16:21:10.979042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:10.951624 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" podUID="6120a22c-a6cc-4cf9-a478-7566b8bc4694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Nov 01 16:21:11.444043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:11.443061 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Nov 01 16:21:11.444043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:11.443146 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" podUID="3f975360-c532-4eb2-aae5-5233d03d027f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Nov 01 16:21:11.527071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:11.519126 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-04f248-6f57f7dc7-snmtt" podUID="6c62027d-8353-4200-82f7-365988964d51" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:21:11.527266 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:11.527135 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04f248-65b9669fcf-7fzhs" podUID="e923b2d1-5a22-4eca-973d-6a94650574b4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:21:11.917042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:11.915205 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Nov 01 16:21:11.917042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:11.915264 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" podUID="6120a22c-a6cc-4cf9-a478-7566b8bc4694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Nov 01 16:21:12.438398 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:12.438340 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Nov 01 16:21:12.440524 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:12.440483 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" podUID="3f975360-c532-4eb2-aae5-5233d03d027f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Nov 01 16:21:12.851042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:12.850670 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" podStartSLOduration=5.850648112 podStartE2EDuration="5.850648112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:21:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:21:09.176344132 +0000 UTC m=+2002.508081782" watchObservedRunningTime="2026-11-01 16:21:12.850648112 +0000 UTC m=+2006.182385767" Nov 01 16:21:12.937595 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:12.936398 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Nov 01 16:21:12.937595 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:12.936461 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" podUID="6120a22c-a6cc-4cf9-a478-7566b8bc4694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Nov 01 16:21:12.948512 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:12.947789 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj"] Nov 01 16:21:12.948512 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:12.948067 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" podUID="388de32a-ab53-48d9-983d-ce302916e7a8" containerName="istio-proxy" containerID="cri-o://d46c601161a716fbd254be500daa4c9ce6687747c8f52b2941278d1d8c6dae07" gracePeriod=30 Nov 01 16:21:13.511511 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:13.478890 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9a482b/auth-test-model-9a482b-kserve-6c554dbfd9-wqk65"] Nov 01 16:21:13.511915 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:13.479041 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Nov 01 16:21:13.512647 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:13.512578 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" podUID="3f975360-c532-4eb2-aae5-5233d03d027f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Nov 01 16:21:13.911042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:13.910836 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Nov 01 16:21:13.911042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:13.910908 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" podUID="6120a22c-a6cc-4cf9-a478-7566b8bc4694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Nov 01 16:21:14.441261 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:14.441225 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Nov 01 16:21:14.441655 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:14.441626 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" podUID="3f975360-c532-4eb2-aae5-5233d03d027f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Nov 01 16:21:14.912051 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:14.910986 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Nov 01 16:21:14.930043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:14.928127 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" podUID="6120a22c-a6cc-4cf9-a478-7566b8bc4694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Nov 01 16:21:15.438030 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:15.437967 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Nov 01 16:21:15.438250 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:15.438043 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" podUID="3f975360-c532-4eb2-aae5-5233d03d027f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Nov 01 16:21:15.917276 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:15.917179 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" Nov 01 16:21:15.918578 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:15.918556 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" Nov 01 16:21:15.919174 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:15.919156 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" Nov 01 16:21:16.443915 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:16.443886 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" Nov 01 16:21:16.444314 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:16.444279 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" Nov 01 16:21:16.445096 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:16.445075 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" Nov 01 16:21:18.259973 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:18.259947 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" Nov 01 16:21:18.323307 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:21:18.323161 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:21:18.494916 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:18.494828 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/388de32a-ab53-48d9-983d-ce302916e7a8-istio-token\" (UniqueName: \"kubernetes.io/projected/388de32a-ab53-48d9-983d-ce302916e7a8-istio-token\") pod \"388de32a-ab53-48d9-983d-ce302916e7a8\" (UID: \"388de32a-ab53-48d9-983d-ce302916e7a8\") " Nov 01 16:21:18.495132 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:18.494949 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/388de32a-ab53-48d9-983d-ce302916e7a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/388de32a-ab53-48d9-983d-ce302916e7a8-istio-data\") pod \"388de32a-ab53-48d9-983d-ce302916e7a8\" (UID: \"388de32a-ab53-48d9-983d-ce302916e7a8\") " Nov 01 16:21:18.495132 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:18.494995 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/388de32a-ab53-48d9-983d-ce302916e7a8-kube-api-access-t7kr4\" (UniqueName: \"kubernetes.io/projected/388de32a-ab53-48d9-983d-ce302916e7a8-kube-api-access-t7kr4\") pod \"388de32a-ab53-48d9-983d-ce302916e7a8\" (UID: \"388de32a-ab53-48d9-983d-ce302916e7a8\") " Nov 01 16:21:18.495132 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:18.495063 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/388de32a-ab53-48d9-983d-ce302916e7a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/388de32a-ab53-48d9-983d-ce302916e7a8-istio-envoy\") pod \"388de32a-ab53-48d9-983d-ce302916e7a8\" (UID: \"388de32a-ab53-48d9-983d-ce302916e7a8\") " Nov 01 16:21:18.495298 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:18.495152 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/388de32a-ab53-48d9-983d-ce302916e7a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/388de32a-ab53-48d9-983d-ce302916e7a8-istio-podinfo\") pod \"388de32a-ab53-48d9-983d-ce302916e7a8\" (UID: \"388de32a-ab53-48d9-983d-ce302916e7a8\") " Nov 01 16:21:18.495298 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:18.495181 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/388de32a-ab53-48d9-983d-ce302916e7a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/388de32a-ab53-48d9-983d-ce302916e7a8-workload-socket\") pod \"388de32a-ab53-48d9-983d-ce302916e7a8\" (UID: \"388de32a-ab53-48d9-983d-ce302916e7a8\") " Nov 01 16:21:18.495298 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:18.495206 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/388de32a-ab53-48d9-983d-ce302916e7a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/388de32a-ab53-48d9-983d-ce302916e7a8-credential-socket\") pod \"388de32a-ab53-48d9-983d-ce302916e7a8\" (UID: \"388de32a-ab53-48d9-983d-ce302916e7a8\") " Nov 01 16:21:18.495298 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:18.495281 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388de32a-ab53-48d9-983d-ce302916e7a8-istio-data" pod "388de32a-ab53-48d9-983d-ce302916e7a8" (UID: "388de32a-ab53-48d9-983d-ce302916e7a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:21:18.495431 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:18.495332 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/388de32a-ab53-48d9-983d-ce302916e7a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/388de32a-ab53-48d9-983d-ce302916e7a8-istiod-ca-cert\") pod \"388de32a-ab53-48d9-983d-ce302916e7a8\" (UID: \"388de32a-ab53-48d9-983d-ce302916e7a8\") " Nov 01 16:21:18.495431 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:18.495397 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/388de32a-ab53-48d9-983d-ce302916e7a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/388de32a-ab53-48d9-983d-ce302916e7a8-workload-certs\") pod \"388de32a-ab53-48d9-983d-ce302916e7a8\" (UID: \"388de32a-ab53-48d9-983d-ce302916e7a8\") " Nov 01 16:21:18.495918 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:18.495887 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/388de32a-ab53-48d9-983d-ce302916e7a8-istiod-ca-cert" pod "388de32a-ab53-48d9-983d-ce302916e7a8" (UID: "388de32a-ab53-48d9-983d-ce302916e7a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:21:18.496162 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:18.496136 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388de32a-ab53-48d9-983d-ce302916e7a8-workload-certs" pod "388de32a-ab53-48d9-983d-ce302916e7a8" (UID: "388de32a-ab53-48d9-983d-ce302916e7a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:21:18.496341 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:18.496187 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388de32a-ab53-48d9-983d-ce302916e7a8-workload-socket" pod "388de32a-ab53-48d9-983d-ce302916e7a8" (UID: "388de32a-ab53-48d9-983d-ce302916e7a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:21:18.496561 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:18.496540 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388de32a-ab53-48d9-983d-ce302916e7a8-credential-socket" pod "388de32a-ab53-48d9-983d-ce302916e7a8" (UID: "388de32a-ab53-48d9-983d-ce302916e7a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:21:18.496660 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:18.496555 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/388de32a-ab53-48d9-983d-ce302916e7a8-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:18.496660 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:18.496578 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/388de32a-ab53-48d9-983d-ce302916e7a8-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:18.496660 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:18.496595 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/388de32a-ab53-48d9-983d-ce302916e7a8-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:18.496660 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:18.496617 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/388de32a-ab53-48d9-983d-ce302916e7a8-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:18.497648 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:18.497625 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388de32a-ab53-48d9-983d-ce302916e7a8-istio-envoy" pod "388de32a-ab53-48d9-983d-ce302916e7a8" (UID: "388de32a-ab53-48d9-983d-ce302916e7a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:21:18.497737 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:18.497642 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/388de32a-ab53-48d9-983d-ce302916e7a8-istio-token" pod "388de32a-ab53-48d9-983d-ce302916e7a8" (UID: "388de32a-ab53-48d9-983d-ce302916e7a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:21:18.497737 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:18.497688 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/388de32a-ab53-48d9-983d-ce302916e7a8-kube-api-access-t7kr4" pod "388de32a-ab53-48d9-983d-ce302916e7a8" (UID: "388de32a-ab53-48d9-983d-ce302916e7a8"). InnerVolumeSpecName "kube-api-access-t7kr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:21:18.498774 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:18.498746 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/388de32a-ab53-48d9-983d-ce302916e7a8-istio-podinfo" pod "388de32a-ab53-48d9-983d-ce302916e7a8" (UID: "388de32a-ab53-48d9-983d-ce302916e7a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:21:18.554699 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:18.554672 4363 generic.go:356] "Generic (PLEG): container finished" podID="388de32a-ab53-48d9-983d-ce302916e7a8" containerID="d46c601161a716fbd254be500daa4c9ce6687747c8f52b2941278d1d8c6dae07" exitCode=0 Nov 01 16:21:18.554840 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:18.554721 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" Nov 01 16:21:18.554840 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:18.554763 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" event={"ID":"388de32a-ab53-48d9-983d-ce302916e7a8","Type":"ContainerDied","Data":"d46c601161a716fbd254be500daa4c9ce6687747c8f52b2941278d1d8c6dae07"} Nov 01 16:21:18.554840 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:18.554821 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj" event={"ID":"388de32a-ab53-48d9-983d-ce302916e7a8","Type":"ContainerDied","Data":"8488b5b2ba1e4ed9c4f6a693ae0909259643c10abaab03025d1798f9f4d6b976"} Nov 01 16:21:18.554970 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:18.554844 4363 scope.go:122] "RemoveContainer" containerID="d46c601161a716fbd254be500daa4c9ce6687747c8f52b2941278d1d8c6dae07" Nov 01 16:21:18.567250 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:18.567235 4363 scope.go:122] "RemoveContainer" containerID="4781f143604d4c1c79729186e3dd8913f8dd19537443e0fa4c9bba5a1c4a0081" Nov 01 16:21:18.577814 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:18.577790 4363 scope.go:122] "RemoveContainer" containerID="d46c601161a716fbd254be500daa4c9ce6687747c8f52b2941278d1d8c6dae07" Nov 01 16:21:18.578126 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:21:18.578101 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d46c601161a716fbd254be500daa4c9ce6687747c8f52b2941278d1d8c6dae07\": container with ID starting with d46c601161a716fbd254be500daa4c9ce6687747c8f52b2941278d1d8c6dae07 not found: ID does not exist" containerID="d46c601161a716fbd254be500daa4c9ce6687747c8f52b2941278d1d8c6dae07" Nov 01 16:21:18.578216 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:18.578140 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d46c601161a716fbd254be500daa4c9ce6687747c8f52b2941278d1d8c6dae07"} err="failed to get container status \"d46c601161a716fbd254be500daa4c9ce6687747c8f52b2941278d1d8c6dae07\": rpc error: code = NotFound desc = could not find container \"d46c601161a716fbd254be500daa4c9ce6687747c8f52b2941278d1d8c6dae07\": container with ID starting with d46c601161a716fbd254be500daa4c9ce6687747c8f52b2941278d1d8c6dae07 not found: ID does not exist" Nov 01 16:21:18.578216 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:18.578165 4363 scope.go:122] "RemoveContainer" containerID="4781f143604d4c1c79729186e3dd8913f8dd19537443e0fa4c9bba5a1c4a0081" Nov 01 16:21:18.578477 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:21:18.578456 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4781f143604d4c1c79729186e3dd8913f8dd19537443e0fa4c9bba5a1c4a0081\": container with ID starting with 4781f143604d4c1c79729186e3dd8913f8dd19537443e0fa4c9bba5a1c4a0081 not found: ID does not exist" containerID="4781f143604d4c1c79729186e3dd8913f8dd19537443e0fa4c9bba5a1c4a0081" Nov 01 16:21:18.578543 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:18.578484 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4781f143604d4c1c79729186e3dd8913f8dd19537443e0fa4c9bba5a1c4a0081"} err="failed to get container status \"4781f143604d4c1c79729186e3dd8913f8dd19537443e0fa4c9bba5a1c4a0081\": rpc error: code = NotFound desc = could not find container \"4781f143604d4c1c79729186e3dd8913f8dd19537443e0fa4c9bba5a1c4a0081\": container with ID starting with 4781f143604d4c1c79729186e3dd8913f8dd19537443e0fa4c9bba5a1c4a0081 not found: ID does not exist" Nov 01 16:21:18.579366 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:18.579329 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj"] Nov 01 16:21:18.586629 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:18.586608 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-04f248-openshift-default-7d6557f5c5-vm6dj"] Nov 01 16:21:18.598400 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:18.598374 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/388de32a-ab53-48d9-983d-ce302916e7a8-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:18.598490 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:18.598405 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7kr4\" (UniqueName: \"kubernetes.io/projected/388de32a-ab53-48d9-983d-ce302916e7a8-kube-api-access-t7kr4\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:18.598490 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:18.598420 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/388de32a-ab53-48d9-983d-ce302916e7a8-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:18.598490 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:18.598434 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/388de32a-ab53-48d9-983d-ce302916e7a8-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:18.598490 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:18.598450 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/388de32a-ab53-48d9-983d-ce302916e7a8-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:18.805373 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:18.805290 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="388de32a-ab53-48d9-983d-ce302916e7a8" path="/var/lib/kubelet/pods/388de32a-ab53-48d9-983d-ce302916e7a8/volumes" Nov 01 16:21:19.310801 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:19.310770 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-04f248-6f57f7dc7-snmtt" Nov 01 16:21:19.368717 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:19.368686 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04f248-65b9669fcf-7fzhs" Nov 01 16:21:19.415186 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:19.415147 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e923b2d1-5a22-4eca-973d-6a94650574b4-kube-api-access-8vdng\" (UniqueName: \"kubernetes.io/projected/e923b2d1-5a22-4eca-973d-6a94650574b4-kube-api-access-8vdng\") pod \"e923b2d1-5a22-4eca-973d-6a94650574b4\" (UID: \"e923b2d1-5a22-4eca-973d-6a94650574b4\") " Nov 01 16:21:19.415373 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:19.415235 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c62027d-8353-4200-82f7-365988964d51-kube-api-access-mvr2t\" (UniqueName: \"kubernetes.io/projected/6c62027d-8353-4200-82f7-365988964d51-kube-api-access-mvr2t\") pod \"6c62027d-8353-4200-82f7-365988964d51\" (UID: \"6c62027d-8353-4200-82f7-365988964d51\") " Nov 01 16:21:19.415469 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:19.415451 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c62027d-8353-4200-82f7-365988964d51-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c62027d-8353-4200-82f7-365988964d51-plugins-config-volume\") pod \"6c62027d-8353-4200-82f7-365988964d51\" (UID: \"6c62027d-8353-4200-82f7-365988964d51\") " Nov 01 16:21:19.415610 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:19.415593 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e923b2d1-5a22-4eca-973d-6a94650574b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e923b2d1-5a22-4eca-973d-6a94650574b4-plugins-config-volume\") pod \"e923b2d1-5a22-4eca-973d-6a94650574b4\" (UID: \"e923b2d1-5a22-4eca-973d-6a94650574b4\") " Nov 01 16:21:19.416897 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:19.416645 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e923b2d1-5a22-4eca-973d-6a94650574b4-plugins-config-volume" pod "e923b2d1-5a22-4eca-973d-6a94650574b4" (UID: "e923b2d1-5a22-4eca-973d-6a94650574b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:21:19.418519 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:19.417444 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c62027d-8353-4200-82f7-365988964d51-plugins-config-volume" pod "6c62027d-8353-4200-82f7-365988964d51" (UID: "6c62027d-8353-4200-82f7-365988964d51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:21:19.419438 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:19.419412 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e923b2d1-5a22-4eca-973d-6a94650574b4-kube-api-access-8vdng" pod "e923b2d1-5a22-4eca-973d-6a94650574b4" (UID: "e923b2d1-5a22-4eca-973d-6a94650574b4"). InnerVolumeSpecName "kube-api-access-8vdng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:21:19.423776 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:19.423748 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c62027d-8353-4200-82f7-365988964d51-kube-api-access-mvr2t" pod "6c62027d-8353-4200-82f7-365988964d51" (UID: "6c62027d-8353-4200-82f7-365988964d51"). InnerVolumeSpecName "kube-api-access-mvr2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:21:19.518293 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:19.518257 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vdng\" (UniqueName: \"kubernetes.io/projected/e923b2d1-5a22-4eca-973d-6a94650574b4-kube-api-access-8vdng\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:19.518293 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:19.518293 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvr2t\" (UniqueName: \"kubernetes.io/projected/6c62027d-8353-4200-82f7-365988964d51-kube-api-access-mvr2t\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:19.518493 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:19.518308 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c62027d-8353-4200-82f7-365988964d51-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:19.518493 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:19.518324 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e923b2d1-5a22-4eca-973d-6a94650574b4-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:19.563889 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:19.563856 4363 generic.go:356] "Generic (PLEG): container finished" podID="6c62027d-8353-4200-82f7-365988964d51" containerID="8b28ce784160139d75f45aa92a1e35273a7aa3706ffecb8bd813a31e2e7a52a9" exitCode=0 Nov 01 16:21:19.564109 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:19.563905 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-04f248-6f57f7dc7-snmtt" Nov 01 16:21:19.564109 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:19.563947 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-04f248-6f57f7dc7-snmtt" event={"ID":"6c62027d-8353-4200-82f7-365988964d51","Type":"ContainerDied","Data":"8b28ce784160139d75f45aa92a1e35273a7aa3706ffecb8bd813a31e2e7a52a9"} Nov 01 16:21:19.564109 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:19.564046 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-04f248-6f57f7dc7-snmtt" event={"ID":"6c62027d-8353-4200-82f7-365988964d51","Type":"ContainerDied","Data":"06621aa34d1f53ad83e85c7f56c1338c21986c5d4eec9f4eee2a846458bd9f91"} Nov 01 16:21:19.564109 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:19.564080 4363 scope.go:122] "RemoveContainer" containerID="8b28ce784160139d75f45aa92a1e35273a7aa3706ffecb8bd813a31e2e7a52a9" Nov 01 16:21:19.566864 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:19.566839 4363 generic.go:356] "Generic (PLEG): container finished" podID="e923b2d1-5a22-4eca-973d-6a94650574b4" containerID="f6d385b745181d874ca57a877ccc4eeccf0d85a85c48699a2b64864c3a0c2fe0" exitCode=0 Nov 01 16:21:19.566985 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:19.566878 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04f248-65b9669fcf-7fzhs" Nov 01 16:21:19.566985 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:19.566916 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04f248-65b9669fcf-7fzhs" event={"ID":"e923b2d1-5a22-4eca-973d-6a94650574b4","Type":"ContainerDied","Data":"f6d385b745181d874ca57a877ccc4eeccf0d85a85c48699a2b64864c3a0c2fe0"} Nov 01 16:21:19.566985 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:19.566960 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04f248-65b9669fcf-7fzhs" event={"ID":"e923b2d1-5a22-4eca-973d-6a94650574b4","Type":"ContainerDied","Data":"8d1490118ad470482bf7b471a41cf01b6d7db97de602578b4a5b142ba0f6a1f7"} Nov 01 16:21:19.576619 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:19.576590 4363 scope.go:122] "RemoveContainer" containerID="8b28ce784160139d75f45aa92a1e35273a7aa3706ffecb8bd813a31e2e7a52a9" Nov 01 16:21:19.578432 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:21:19.578410 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b28ce784160139d75f45aa92a1e35273a7aa3706ffecb8bd813a31e2e7a52a9\": container with ID starting with 8b28ce784160139d75f45aa92a1e35273a7aa3706ffecb8bd813a31e2e7a52a9 not found: ID does not exist" containerID="8b28ce784160139d75f45aa92a1e35273a7aa3706ffecb8bd813a31e2e7a52a9" Nov 01 16:21:19.578539 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:19.578443 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b28ce784160139d75f45aa92a1e35273a7aa3706ffecb8bd813a31e2e7a52a9"} err="failed to get container status \"8b28ce784160139d75f45aa92a1e35273a7aa3706ffecb8bd813a31e2e7a52a9\": rpc error: code = NotFound desc = could not find container \"8b28ce784160139d75f45aa92a1e35273a7aa3706ffecb8bd813a31e2e7a52a9\": container with ID starting with 8b28ce784160139d75f45aa92a1e35273a7aa3706ffecb8bd813a31e2e7a52a9 not found: ID does not exist" Nov 01 16:21:19.578539 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:19.578465 4363 scope.go:122] "RemoveContainer" containerID="f6d385b745181d874ca57a877ccc4eeccf0d85a85c48699a2b64864c3a0c2fe0" Nov 01 16:21:19.597247 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:19.597185 4363 scope.go:122] "RemoveContainer" containerID="f6d385b745181d874ca57a877ccc4eeccf0d85a85c48699a2b64864c3a0c2fe0" Nov 01 16:21:19.597616 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:21:19.597583 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6d385b745181d874ca57a877ccc4eeccf0d85a85c48699a2b64864c3a0c2fe0\": container with ID starting with f6d385b745181d874ca57a877ccc4eeccf0d85a85c48699a2b64864c3a0c2fe0 not found: ID does not exist" containerID="f6d385b745181d874ca57a877ccc4eeccf0d85a85c48699a2b64864c3a0c2fe0" Nov 01 16:21:19.597724 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:19.597625 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6d385b745181d874ca57a877ccc4eeccf0d85a85c48699a2b64864c3a0c2fe0"} err="failed to get container status \"f6d385b745181d874ca57a877ccc4eeccf0d85a85c48699a2b64864c3a0c2fe0\": rpc error: code = NotFound desc = could not find container \"f6d385b745181d874ca57a877ccc4eeccf0d85a85c48699a2b64864c3a0c2fe0\": container with ID starting with f6d385b745181d874ca57a877ccc4eeccf0d85a85c48699a2b64864c3a0c2fe0 not found: ID does not exist" Nov 01 16:21:19.601914 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:19.601889 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-04f248-6f57f7dc7-snmtt"] Nov 01 16:21:19.606044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:19.606006 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-04f248-6f57f7dc7-snmtt"] Nov 01 16:21:19.613429 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:19.613399 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-04f248-65b9669fcf-7fzhs"] Nov 01 16:21:19.617416 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:19.617393 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-04f248-65b9669fcf-7fzhs"] Nov 01 16:21:20.819160 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:20.816864 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c62027d-8353-4200-82f7-365988964d51" path="/var/lib/kubelet/pods/6c62027d-8353-4200-82f7-365988964d51/volumes" Nov 01 16:21:20.827327 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:20.827294 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e923b2d1-5a22-4eca-973d-6a94650574b4" path="/var/lib/kubelet/pods/e923b2d1-5a22-4eca-973d-6a94650574b4/volumes" Nov 01 16:21:20.843703 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:20.843670 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-f01de5/auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt"] Nov 01 16:21:23.468282 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:23.468237 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-10b00a-659db85879-m4pxv"] Nov 01 16:21:23.480562 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:23.480534 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-10b00a-5cdcfd8b5f-fwz5p"] Nov 01 16:21:24.931121 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:24.930737 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-7225e5/auth-test-model-7225e5-kserve-84f65d66df-jd8qc"] Nov 01 16:21:24.932083 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:24.932055 4363 cpu_manager.go:414] "containerMap: removing container" podUID="388de32a-ab53-48d9-983d-ce302916e7a8" containerName="istio-proxy" Nov 01 16:21:24.932083 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:24.932081 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="388de32a-ab53-48d9-983d-ce302916e7a8" containerName="istio-proxy" Nov 01 16:21:24.932303 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:24.932116 4363 cpu_manager.go:414] "containerMap: removing container" podUID="6c62027d-8353-4200-82f7-365988964d51" containerName="payload-processing" Nov 01 16:21:24.932303 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:24.932125 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c62027d-8353-4200-82f7-365988964d51" containerName="payload-processing" Nov 01 16:21:24.932303 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:24.932208 4363 cpu_manager.go:414] "containerMap: removing container" podUID="e923b2d1-5a22-4eca-973d-6a94650574b4" containerName="payload-pre-processing" Nov 01 16:21:24.932303 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:24.932216 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e923b2d1-5a22-4eca-973d-6a94650574b4" containerName="payload-pre-processing" Nov 01 16:21:24.933115 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:24.933094 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="e923b2d1-5a22-4eca-973d-6a94650574b4" containerName="payload-pre-processing" Nov 01 16:21:24.933261 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:24.933243 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="388de32a-ab53-48d9-983d-ce302916e7a8" containerName="istio-proxy" Nov 01 16:21:24.933261 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:24.933262 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="388de32a-ab53-48d9-983d-ce302916e7a8" containerName="istio-proxy" Nov 01 16:21:24.933430 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:24.933276 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c62027d-8353-4200-82f7-365988964d51" containerName="payload-processing" Nov 01 16:21:24.940306 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:24.940280 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-7225e5/auth-test-model-7225e5-kserve-84f65d66df-jd8qc" Nov 01 16:21:24.946041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:24.945582 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-7225e5/auth-test-model-7225e5-kserve-84f65d66df-jd8qc"] Nov 01 16:21:24.948415 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:24.948395 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-7225e5\"/\"default-dockercfg-45xs6\"" Nov 01 16:21:24.949989 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:24.949314 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-7225e5\"/\"kube-root-ca.crt\"" Nov 01 16:21:24.949989 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:24.949409 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-7225e5\"/\"openshift-service-ca.crt\"" Nov 01 16:21:24.949989 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:24.949475 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-7225e5\"/\"auth-test-model-7225e5-kserve-self-signed-certs\"" Nov 01 16:21:24.954067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:24.954000 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/901ec925-24b0-4f15-a356-b7a5da163158-model-cache\") pod \"auth-test-model-7225e5-kserve-84f65d66df-jd8qc\" (UID: \"901ec925-24b0-4f15-a356-b7a5da163158\") " pod="ai-tenant-e2e-shared-a-w2-7225e5/auth-test-model-7225e5-kserve-84f65d66df-jd8qc" Nov 01 16:21:24.954185 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:24.954071 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/901ec925-24b0-4f15-a356-b7a5da163158-tls-certs\") pod \"auth-test-model-7225e5-kserve-84f65d66df-jd8qc\" (UID: \"901ec925-24b0-4f15-a356-b7a5da163158\") " pod="ai-tenant-e2e-shared-a-w2-7225e5/auth-test-model-7225e5-kserve-84f65d66df-jd8qc" Nov 01 16:21:24.954253 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:24.954191 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/901ec925-24b0-4f15-a356-b7a5da163158-home\") pod \"auth-test-model-7225e5-kserve-84f65d66df-jd8qc\" (UID: \"901ec925-24b0-4f15-a356-b7a5da163158\") " pod="ai-tenant-e2e-shared-a-w2-7225e5/auth-test-model-7225e5-kserve-84f65d66df-jd8qc" Nov 01 16:21:24.954311 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:24.954298 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89jxw\" (UniqueName: \"kubernetes.io/projected/901ec925-24b0-4f15-a356-b7a5da163158-kube-api-access-89jxw\") pod \"auth-test-model-7225e5-kserve-84f65d66df-jd8qc\" (UID: \"901ec925-24b0-4f15-a356-b7a5da163158\") " pod="ai-tenant-e2e-shared-a-w2-7225e5/auth-test-model-7225e5-kserve-84f65d66df-jd8qc" Nov 01 16:21:24.954380 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:24.954367 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/901ec925-24b0-4f15-a356-b7a5da163158-dshm\") pod \"auth-test-model-7225e5-kserve-84f65d66df-jd8qc\" (UID: \"901ec925-24b0-4f15-a356-b7a5da163158\") " pod="ai-tenant-e2e-shared-a-w2-7225e5/auth-test-model-7225e5-kserve-84f65d66df-jd8qc" Nov 01 16:21:24.954437 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:24.954390 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/901ec925-24b0-4f15-a356-b7a5da163158-tmp-dir\") pod \"auth-test-model-7225e5-kserve-84f65d66df-jd8qc\" (UID: \"901ec925-24b0-4f15-a356-b7a5da163158\") " pod="ai-tenant-e2e-shared-a-w2-7225e5/auth-test-model-7225e5-kserve-84f65d66df-jd8qc" Nov 01 16:21:25.056248 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:25.056215 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/901ec925-24b0-4f15-a356-b7a5da163158-model-cache\") pod \"auth-test-model-7225e5-kserve-84f65d66df-jd8qc\" (UID: \"901ec925-24b0-4f15-a356-b7a5da163158\") " pod="ai-tenant-e2e-shared-a-w2-7225e5/auth-test-model-7225e5-kserve-84f65d66df-jd8qc" Nov 01 16:21:25.056442 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:25.056266 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/901ec925-24b0-4f15-a356-b7a5da163158-tls-certs\") pod \"auth-test-model-7225e5-kserve-84f65d66df-jd8qc\" (UID: \"901ec925-24b0-4f15-a356-b7a5da163158\") " pod="ai-tenant-e2e-shared-a-w2-7225e5/auth-test-model-7225e5-kserve-84f65d66df-jd8qc" Nov 01 16:21:25.056442 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:25.056331 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/901ec925-24b0-4f15-a356-b7a5da163158-home\") pod \"auth-test-model-7225e5-kserve-84f65d66df-jd8qc\" (UID: \"901ec925-24b0-4f15-a356-b7a5da163158\") " pod="ai-tenant-e2e-shared-a-w2-7225e5/auth-test-model-7225e5-kserve-84f65d66df-jd8qc" Nov 01 16:21:25.056442 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:25.056416 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89jxw\" (UniqueName: \"kubernetes.io/projected/901ec925-24b0-4f15-a356-b7a5da163158-kube-api-access-89jxw\") pod \"auth-test-model-7225e5-kserve-84f65d66df-jd8qc\" (UID: \"901ec925-24b0-4f15-a356-b7a5da163158\") " pod="ai-tenant-e2e-shared-a-w2-7225e5/auth-test-model-7225e5-kserve-84f65d66df-jd8qc" Nov 01 16:21:25.056631 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:25.056503 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/901ec925-24b0-4f15-a356-b7a5da163158-dshm\") pod \"auth-test-model-7225e5-kserve-84f65d66df-jd8qc\" (UID: \"901ec925-24b0-4f15-a356-b7a5da163158\") " pod="ai-tenant-e2e-shared-a-w2-7225e5/auth-test-model-7225e5-kserve-84f65d66df-jd8qc" Nov 01 16:21:25.056631 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:25.056532 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/901ec925-24b0-4f15-a356-b7a5da163158-tmp-dir\") pod \"auth-test-model-7225e5-kserve-84f65d66df-jd8qc\" (UID: \"901ec925-24b0-4f15-a356-b7a5da163158\") " pod="ai-tenant-e2e-shared-a-w2-7225e5/auth-test-model-7225e5-kserve-84f65d66df-jd8qc" Nov 01 16:21:25.056731 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:25.056704 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/901ec925-24b0-4f15-a356-b7a5da163158-model-cache\") pod \"auth-test-model-7225e5-kserve-84f65d66df-jd8qc\" (UID: \"901ec925-24b0-4f15-a356-b7a5da163158\") " pod="ai-tenant-e2e-shared-a-w2-7225e5/auth-test-model-7225e5-kserve-84f65d66df-jd8qc" Nov 01 16:21:25.056817 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:25.056768 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/901ec925-24b0-4f15-a356-b7a5da163158-home\") pod \"auth-test-model-7225e5-kserve-84f65d66df-jd8qc\" (UID: \"901ec925-24b0-4f15-a356-b7a5da163158\") " pod="ai-tenant-e2e-shared-a-w2-7225e5/auth-test-model-7225e5-kserve-84f65d66df-jd8qc" Nov 01 16:21:25.056935 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:25.056909 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/901ec925-24b0-4f15-a356-b7a5da163158-tmp-dir\") pod \"auth-test-model-7225e5-kserve-84f65d66df-jd8qc\" (UID: \"901ec925-24b0-4f15-a356-b7a5da163158\") " pod="ai-tenant-e2e-shared-a-w2-7225e5/auth-test-model-7225e5-kserve-84f65d66df-jd8qc" Nov 01 16:21:25.058864 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:25.058839 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/901ec925-24b0-4f15-a356-b7a5da163158-dshm\") pod \"auth-test-model-7225e5-kserve-84f65d66df-jd8qc\" (UID: \"901ec925-24b0-4f15-a356-b7a5da163158\") " pod="ai-tenant-e2e-shared-a-w2-7225e5/auth-test-model-7225e5-kserve-84f65d66df-jd8qc" Nov 01 16:21:25.058983 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:25.058877 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/901ec925-24b0-4f15-a356-b7a5da163158-tls-certs\") pod \"auth-test-model-7225e5-kserve-84f65d66df-jd8qc\" (UID: \"901ec925-24b0-4f15-a356-b7a5da163158\") " pod="ai-tenant-e2e-shared-a-w2-7225e5/auth-test-model-7225e5-kserve-84f65d66df-jd8qc" Nov 01 16:21:25.063516 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:25.063490 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89jxw\" (UniqueName: \"kubernetes.io/projected/901ec925-24b0-4f15-a356-b7a5da163158-kube-api-access-89jxw\") pod \"auth-test-model-7225e5-kserve-84f65d66df-jd8qc\" (UID: \"901ec925-24b0-4f15-a356-b7a5da163158\") " pod="ai-tenant-e2e-shared-a-w2-7225e5/auth-test-model-7225e5-kserve-84f65d66df-jd8qc" Nov 01 16:21:25.268401 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:25.268307 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-7225e5/auth-test-model-7225e5-kserve-84f65d66df-jd8qc" Nov 01 16:21:25.567942 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:25.567857 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-7225e5/auth-test-model-7225e5-kserve-84f65d66df-jd8qc"] Nov 01 16:21:25.591466 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:21:25.574773 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod901ec925_24b0_4f15_a356_b7a5da163158.slice/crio-a8940ffb361429ed76c4791e0636a22971cb9362c843e0d1a33d501d0022edae WatchSource:0}: Error finding container a8940ffb361429ed76c4791e0636a22971cb9362c843e0d1a33d501d0022edae: Status 404 returned error can't find the container with id a8940ffb361429ed76c4791e0636a22971cb9362c843e0d1a33d501d0022edae Nov 01 16:21:25.681988 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:25.680988 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-7225e5/auth-test-model-7225e5-kserve-84f65d66df-jd8qc" event={"ID":"901ec925-24b0-4f15-a356-b7a5da163158","Type":"ContainerStarted","Data":"db4bec0bf45dc84723c13ba4883f1bb50308d69649e5ea75aaf0c355084fd540"} Nov 01 16:21:25.681988 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:25.681055 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-7225e5/auth-test-model-7225e5-kserve-84f65d66df-jd8qc" event={"ID":"901ec925-24b0-4f15-a356-b7a5da163158","Type":"ContainerStarted","Data":"a8940ffb361429ed76c4791e0636a22971cb9362c843e0d1a33d501d0022edae"} Nov 01 16:21:26.268709 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:26.268678 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-7225e5/auth-test-model-7225e5-kserve-84f65d66df-jd8qc" Nov 01 16:21:28.468762 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:28.468722 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-10b00a-659db85879-m4pxv" podUID="b39da192-a1b1-43f9-b6c4-b2186df9063f" containerName="payload-processing" containerID="cri-o://0a76156d8f3310f0e93ae9fe38ea98cfd8af56c01af6b4467c011d68eae2f380" gracePeriod=25 Nov 01 16:21:28.481747 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:28.481669 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-10b00a-5cdcfd8b5f-fwz5p" podUID="d3007207-2dac-46d0-970c-8138373dc953" containerName="payload-pre-processing" containerID="cri-o://58f84855ecde8d855316da17f8bb0bf0bac8979a691786fd023e1680f930dbda" gracePeriod=25 Nov 01 16:21:28.847889 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:28.847854 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-9a482b/auth-test-model-9a482b-kserve-6c554dbfd9-wqk65" podUID="1ee17e76-6806-4747-9376-e9c8e851b70c" containerName="main" containerID="cri-o://91e9ec9369f7c75e4002867661adacb88f1bb12b5de86d5364d8969516d9dd8d" gracePeriod=45 Nov 01 16:21:29.191193 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:29.191160 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9a482b/auth-test-model-9a482b-kserve-6c554dbfd9-wqk65" Nov 01 16:21:29.210086 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:29.210012 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-7225e5/auth-test-model-7225e5-kserve-84f65d66df-jd8qc" podStartSLOduration=5.209994937 podStartE2EDuration="5.209994937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:21:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:21:25.728373404 +0000 UTC m=+2019.060111055" watchObservedRunningTime="2026-11-01 16:21:29.209994937 +0000 UTC m=+2022.541732588" Nov 01 16:21:29.247259 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:29.247226 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ee17e76-6806-4747-9376-e9c8e851b70c-kube-api-access-jhjl7\" (UniqueName: \"kubernetes.io/projected/1ee17e76-6806-4747-9376-e9c8e851b70c-kube-api-access-jhjl7\") pod \"1ee17e76-6806-4747-9376-e9c8e851b70c\" (UID: \"1ee17e76-6806-4747-9376-e9c8e851b70c\") " Nov 01 16:21:29.247438 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:29.247310 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ee17e76-6806-4747-9376-e9c8e851b70c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ee17e76-6806-4747-9376-e9c8e851b70c-dshm\") pod \"1ee17e76-6806-4747-9376-e9c8e851b70c\" (UID: \"1ee17e76-6806-4747-9376-e9c8e851b70c\") " Nov 01 16:21:29.247438 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:29.247433 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ee17e76-6806-4747-9376-e9c8e851b70c-tls-certs\" (UniqueName: \"kubernetes.io/secret/1ee17e76-6806-4747-9376-e9c8e851b70c-tls-certs\") pod \"1ee17e76-6806-4747-9376-e9c8e851b70c\" (UID: \"1ee17e76-6806-4747-9376-e9c8e851b70c\") " Nov 01 16:21:29.247596 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:29.247540 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ee17e76-6806-4747-9376-e9c8e851b70c-home\" (UniqueName: \"kubernetes.io/empty-dir/1ee17e76-6806-4747-9376-e9c8e851b70c-home\") pod \"1ee17e76-6806-4747-9376-e9c8e851b70c\" (UID: \"1ee17e76-6806-4747-9376-e9c8e851b70c\") " Nov 01 16:21:29.247596 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:29.247586 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ee17e76-6806-4747-9376-e9c8e851b70c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ee17e76-6806-4747-9376-e9c8e851b70c-tmp-dir\") pod \"1ee17e76-6806-4747-9376-e9c8e851b70c\" (UID: \"1ee17e76-6806-4747-9376-e9c8e851b70c\") " Nov 01 16:21:29.247913 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:29.247616 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ee17e76-6806-4747-9376-e9c8e851b70c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ee17e76-6806-4747-9376-e9c8e851b70c-model-cache\") pod \"1ee17e76-6806-4747-9376-e9c8e851b70c\" (UID: \"1ee17e76-6806-4747-9376-e9c8e851b70c\") " Nov 01 16:21:29.247983 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:29.247961 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ee17e76-6806-4747-9376-e9c8e851b70c-tmp-dir" pod "1ee17e76-6806-4747-9376-e9c8e851b70c" (UID: "1ee17e76-6806-4747-9376-e9c8e851b70c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:21:29.248145 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:29.248115 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ee17e76-6806-4747-9376-e9c8e851b70c-model-cache" pod "1ee17e76-6806-4747-9376-e9c8e851b70c" (UID: "1ee17e76-6806-4747-9376-e9c8e851b70c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:21:29.248350 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:29.248327 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ee17e76-6806-4747-9376-e9c8e851b70c-home" pod "1ee17e76-6806-4747-9376-e9c8e851b70c" (UID: "1ee17e76-6806-4747-9376-e9c8e851b70c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:21:29.248649 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:29.248625 4363 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ee17e76-6806-4747-9376-e9c8e851b70c-home\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:29.248743 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:29.248655 4363 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ee17e76-6806-4747-9376-e9c8e851b70c-tmp-dir\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:29.248743 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:29.248670 4363 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ee17e76-6806-4747-9376-e9c8e851b70c-model-cache\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:29.250143 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:29.250116 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ee17e76-6806-4747-9376-e9c8e851b70c-dshm" pod "1ee17e76-6806-4747-9376-e9c8e851b70c" (UID: "1ee17e76-6806-4747-9376-e9c8e851b70c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:21:29.250250 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:29.250219 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ee17e76-6806-4747-9376-e9c8e851b70c-tls-certs" pod "1ee17e76-6806-4747-9376-e9c8e851b70c" (UID: "1ee17e76-6806-4747-9376-e9c8e851b70c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:21:29.250771 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:29.250738 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ee17e76-6806-4747-9376-e9c8e851b70c-kube-api-access-jhjl7" pod "1ee17e76-6806-4747-9376-e9c8e851b70c" (UID: "1ee17e76-6806-4747-9376-e9c8e851b70c"). InnerVolumeSpecName "kube-api-access-jhjl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:21:29.349780 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:29.349748 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhjl7\" (UniqueName: \"kubernetes.io/projected/1ee17e76-6806-4747-9376-e9c8e851b70c-kube-api-access-jhjl7\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:29.349780 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:29.349777 4363 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ee17e76-6806-4747-9376-e9c8e851b70c-dshm\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:29.349780 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:29.349787 4363 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ee17e76-6806-4747-9376-e9c8e851b70c-tls-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:29.717932 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:29.717900 4363 generic.go:356] "Generic (PLEG): container finished" podID="1ee17e76-6806-4747-9376-e9c8e851b70c" containerID="91e9ec9369f7c75e4002867661adacb88f1bb12b5de86d5364d8969516d9dd8d" exitCode=0 Nov 01 16:21:29.718470 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:29.717972 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9a482b/auth-test-model-9a482b-kserve-6c554dbfd9-wqk65" Nov 01 16:21:29.718470 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:29.717979 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9a482b/auth-test-model-9a482b-kserve-6c554dbfd9-wqk65" event={"ID":"1ee17e76-6806-4747-9376-e9c8e851b70c","Type":"ContainerDied","Data":"91e9ec9369f7c75e4002867661adacb88f1bb12b5de86d5364d8969516d9dd8d"} Nov 01 16:21:29.718470 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:29.718046 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9a482b/auth-test-model-9a482b-kserve-6c554dbfd9-wqk65" event={"ID":"1ee17e76-6806-4747-9376-e9c8e851b70c","Type":"ContainerDied","Data":"a3872b20e93c080f0f0aaa1c08f609ae1a002b83e1a8ce39958c364517817e09"} Nov 01 16:21:29.718470 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:29.718072 4363 scope.go:122] "RemoveContainer" containerID="91e9ec9369f7c75e4002867661adacb88f1bb12b5de86d5364d8969516d9dd8d" Nov 01 16:21:29.730511 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:29.730488 4363 scope.go:122] "RemoveContainer" containerID="91e9ec9369f7c75e4002867661adacb88f1bb12b5de86d5364d8969516d9dd8d" Nov 01 16:21:29.730811 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:21:29.730790 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91e9ec9369f7c75e4002867661adacb88f1bb12b5de86d5364d8969516d9dd8d\": container with ID starting with 91e9ec9369f7c75e4002867661adacb88f1bb12b5de86d5364d8969516d9dd8d not found: ID does not exist" containerID="91e9ec9369f7c75e4002867661adacb88f1bb12b5de86d5364d8969516d9dd8d" Nov 01 16:21:29.730869 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:29.730820 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91e9ec9369f7c75e4002867661adacb88f1bb12b5de86d5364d8969516d9dd8d"} err="failed to get container status \"91e9ec9369f7c75e4002867661adacb88f1bb12b5de86d5364d8969516d9dd8d\": rpc error: code = NotFound desc = could not find container \"91e9ec9369f7c75e4002867661adacb88f1bb12b5de86d5364d8969516d9dd8d\": container with ID starting with 91e9ec9369f7c75e4002867661adacb88f1bb12b5de86d5364d8969516d9dd8d not found: ID does not exist" Nov 01 16:21:29.760067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:29.757081 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9a482b/auth-test-model-9a482b-kserve-6c554dbfd9-wqk65"] Nov 01 16:21:29.827061 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:29.822323 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9a482b/auth-test-model-9a482b-kserve-6c554dbfd9-wqk65"] Nov 01 16:21:30.224678 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:30.224640 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-9a482b/sub-test-model-9a482b-kserve-d54b499b5-4svj8"] Nov 01 16:21:30.225920 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:30.225897 4363 cpu_manager.go:414] "containerMap: removing container" podUID="1ee17e76-6806-4747-9376-e9c8e851b70c" containerName="main" Nov 01 16:21:30.225985 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:30.225923 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ee17e76-6806-4747-9376-e9c8e851b70c" containerName="main" Nov 01 16:21:30.226035 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:30.225990 4363 cpu_manager.go:414] "containerMap: removing container" podUID="388de32a-ab53-48d9-983d-ce302916e7a8" containerName="istio-proxy" Nov 01 16:21:30.226035 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:30.225998 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="388de32a-ab53-48d9-983d-ce302916e7a8" containerName="istio-proxy" Nov 01 16:21:30.226232 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:30.226217 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ee17e76-6806-4747-9376-e9c8e851b70c" containerName="main" Nov 01 16:21:30.231758 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:30.231737 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9a482b/sub-test-model-9a482b-kserve-d54b499b5-4svj8" Nov 01 16:21:30.235239 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:30.235216 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-9a482b\"/\"kube-root-ca.crt\"" Nov 01 16:21:30.235403 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:30.235289 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-9a482b\"/\"openshift-service-ca.crt\"" Nov 01 16:21:30.235896 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:30.235880 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-9a482b\"/\"default-dockercfg-sl2lj\"" Nov 01 16:21:30.236001 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:30.235921 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-9a482b\"/\"sub-test-model-9a482b-kserve-self-signed-certs\"" Nov 01 16:21:30.238755 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:30.238730 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9a482b/sub-test-model-9a482b-kserve-d54b499b5-4svj8"] Nov 01 16:21:30.316069 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:30.301447 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b70b20e2-9bf8-49b5-b68b-6b45ef65f737-tls-certs\") pod \"sub-test-model-9a482b-kserve-d54b499b5-4svj8\" (UID: \"b70b20e2-9bf8-49b5-b68b-6b45ef65f737\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/sub-test-model-9a482b-kserve-d54b499b5-4svj8" Nov 01 16:21:30.330135 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:30.326951 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4krmk\" (UniqueName: \"kubernetes.io/projected/b70b20e2-9bf8-49b5-b68b-6b45ef65f737-kube-api-access-4krmk\") pod \"sub-test-model-9a482b-kserve-d54b499b5-4svj8\" (UID: \"b70b20e2-9bf8-49b5-b68b-6b45ef65f737\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/sub-test-model-9a482b-kserve-d54b499b5-4svj8" Nov 01 16:21:30.330546 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:30.330511 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b70b20e2-9bf8-49b5-b68b-6b45ef65f737-model-cache\") pod \"sub-test-model-9a482b-kserve-d54b499b5-4svj8\" (UID: \"b70b20e2-9bf8-49b5-b68b-6b45ef65f737\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/sub-test-model-9a482b-kserve-d54b499b5-4svj8" Nov 01 16:21:30.330792 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:30.330773 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b70b20e2-9bf8-49b5-b68b-6b45ef65f737-dshm\") pod \"sub-test-model-9a482b-kserve-d54b499b5-4svj8\" (UID: \"b70b20e2-9bf8-49b5-b68b-6b45ef65f737\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/sub-test-model-9a482b-kserve-d54b499b5-4svj8" Nov 01 16:21:30.331009 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:30.330939 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b70b20e2-9bf8-49b5-b68b-6b45ef65f737-tmp-dir\") pod \"sub-test-model-9a482b-kserve-d54b499b5-4svj8\" (UID: \"b70b20e2-9bf8-49b5-b68b-6b45ef65f737\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/sub-test-model-9a482b-kserve-d54b499b5-4svj8" Nov 01 16:21:30.332443 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:30.332418 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b70b20e2-9bf8-49b5-b68b-6b45ef65f737-home\") pod \"sub-test-model-9a482b-kserve-d54b499b5-4svj8\" (UID: \"b70b20e2-9bf8-49b5-b68b-6b45ef65f737\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/sub-test-model-9a482b-kserve-d54b499b5-4svj8" Nov 01 16:21:30.434071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:30.434034 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b70b20e2-9bf8-49b5-b68b-6b45ef65f737-dshm\") pod \"sub-test-model-9a482b-kserve-d54b499b5-4svj8\" (UID: \"b70b20e2-9bf8-49b5-b68b-6b45ef65f737\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/sub-test-model-9a482b-kserve-d54b499b5-4svj8" Nov 01 16:21:30.434223 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:30.434107 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b70b20e2-9bf8-49b5-b68b-6b45ef65f737-tmp-dir\") pod \"sub-test-model-9a482b-kserve-d54b499b5-4svj8\" (UID: \"b70b20e2-9bf8-49b5-b68b-6b45ef65f737\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/sub-test-model-9a482b-kserve-d54b499b5-4svj8" Nov 01 16:21:30.434288 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:30.434239 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b70b20e2-9bf8-49b5-b68b-6b45ef65f737-home\") pod \"sub-test-model-9a482b-kserve-d54b499b5-4svj8\" (UID: \"b70b20e2-9bf8-49b5-b68b-6b45ef65f737\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/sub-test-model-9a482b-kserve-d54b499b5-4svj8" Nov 01 16:21:30.434375 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:30.434357 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b70b20e2-9bf8-49b5-b68b-6b45ef65f737-tls-certs\") pod \"sub-test-model-9a482b-kserve-d54b499b5-4svj8\" (UID: \"b70b20e2-9bf8-49b5-b68b-6b45ef65f737\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/sub-test-model-9a482b-kserve-d54b499b5-4svj8" Nov 01 16:21:30.434450 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:30.434429 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4krmk\" (UniqueName: \"kubernetes.io/projected/b70b20e2-9bf8-49b5-b68b-6b45ef65f737-kube-api-access-4krmk\") pod \"sub-test-model-9a482b-kserve-d54b499b5-4svj8\" (UID: \"b70b20e2-9bf8-49b5-b68b-6b45ef65f737\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/sub-test-model-9a482b-kserve-d54b499b5-4svj8" Nov 01 16:21:30.434862 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:30.434551 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b70b20e2-9bf8-49b5-b68b-6b45ef65f737-model-cache\") pod \"sub-test-model-9a482b-kserve-d54b499b5-4svj8\" (UID: \"b70b20e2-9bf8-49b5-b68b-6b45ef65f737\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/sub-test-model-9a482b-kserve-d54b499b5-4svj8" Nov 01 16:21:30.434862 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:30.434642 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b70b20e2-9bf8-49b5-b68b-6b45ef65f737-tmp-dir\") pod \"sub-test-model-9a482b-kserve-d54b499b5-4svj8\" (UID: \"b70b20e2-9bf8-49b5-b68b-6b45ef65f737\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/sub-test-model-9a482b-kserve-d54b499b5-4svj8" Nov 01 16:21:30.434862 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:30.434677 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b70b20e2-9bf8-49b5-b68b-6b45ef65f737-home\") pod \"sub-test-model-9a482b-kserve-d54b499b5-4svj8\" (UID: \"b70b20e2-9bf8-49b5-b68b-6b45ef65f737\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/sub-test-model-9a482b-kserve-d54b499b5-4svj8" Nov 01 16:21:30.435177 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:30.434926 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b70b20e2-9bf8-49b5-b68b-6b45ef65f737-model-cache\") pod \"sub-test-model-9a482b-kserve-d54b499b5-4svj8\" (UID: \"b70b20e2-9bf8-49b5-b68b-6b45ef65f737\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/sub-test-model-9a482b-kserve-d54b499b5-4svj8" Nov 01 16:21:30.436433 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:30.436408 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b70b20e2-9bf8-49b5-b68b-6b45ef65f737-dshm\") pod \"sub-test-model-9a482b-kserve-d54b499b5-4svj8\" (UID: \"b70b20e2-9bf8-49b5-b68b-6b45ef65f737\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/sub-test-model-9a482b-kserve-d54b499b5-4svj8" Nov 01 16:21:30.437027 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:30.436999 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b70b20e2-9bf8-49b5-b68b-6b45ef65f737-tls-certs\") pod \"sub-test-model-9a482b-kserve-d54b499b5-4svj8\" (UID: \"b70b20e2-9bf8-49b5-b68b-6b45ef65f737\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/sub-test-model-9a482b-kserve-d54b499b5-4svj8" Nov 01 16:21:30.442628 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:30.442603 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4krmk\" (UniqueName: \"kubernetes.io/projected/b70b20e2-9bf8-49b5-b68b-6b45ef65f737-kube-api-access-4krmk\") pod \"sub-test-model-9a482b-kserve-d54b499b5-4svj8\" (UID: \"b70b20e2-9bf8-49b5-b68b-6b45ef65f737\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/sub-test-model-9a482b-kserve-d54b499b5-4svj8" Nov 01 16:21:30.556086 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:30.553474 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9a482b/sub-test-model-9a482b-kserve-d54b499b5-4svj8" Nov 01 16:21:30.805131 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:30.804796 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ee17e76-6806-4747-9376-e9c8e851b70c" path="/var/lib/kubelet/pods/1ee17e76-6806-4747-9376-e9c8e851b70c/volumes" Nov 01 16:21:30.938154 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:30.938124 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9a482b/sub-test-model-9a482b-kserve-d54b499b5-4svj8"] Nov 01 16:21:30.938808 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:21:30.938704 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb70b20e2_9bf8_49b5_b68b_6b45ef65f737.slice/crio-11444d767cd08157d3e978bd50128a2e32335cb92260e137addf51d03fae300e WatchSource:0}: Error finding container 11444d767cd08157d3e978bd50128a2e32335cb92260e137addf51d03fae300e: Status 404 returned error can't find the container with id 11444d767cd08157d3e978bd50128a2e32335cb92260e137addf51d03fae300e Nov 01 16:21:31.744525 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:31.744487 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9a482b/sub-test-model-9a482b-kserve-d54b499b5-4svj8" event={"ID":"b70b20e2-9bf8-49b5-b68b-6b45ef65f737","Type":"ContainerStarted","Data":"2019bd5e04a1dee1227ac5eca1ca2fa5e54083ae36e220eb275f05af100ad450"} Nov 01 16:21:31.744525 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:31.744539 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9a482b/sub-test-model-9a482b-kserve-d54b499b5-4svj8" event={"ID":"b70b20e2-9bf8-49b5-b68b-6b45ef65f737","Type":"ContainerStarted","Data":"11444d767cd08157d3e978bd50128a2e32335cb92260e137addf51d03fae300e"} Nov 01 16:21:31.762789 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:31.762741 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-9a482b/sub-test-model-9a482b-kserve-d54b499b5-4svj8" podStartSLOduration=1.7627269939999999 podStartE2EDuration="1.762726994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:21:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:21:31.760959137 +0000 UTC m=+2025.092696788" watchObservedRunningTime="2026-11-01 16:21:31.762726994 +0000 UTC m=+2025.094464643" Nov 01 16:21:32.554400 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:32.554364 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-9a482b/sub-test-model-9a482b-kserve-d54b499b5-4svj8" Nov 01 16:21:35.275079 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:35.273674 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-7225e5/auth-test-model-7225e5-kserve-84f65d66df-jd8qc" Nov 01 16:21:35.379149 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:35.379066 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-7225e5/auth-test-model-7225e5-kserve-84f65d66df-jd8qc" Nov 01 16:21:35.831521 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:35.831487 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-7225e5/auth-test-model-7225e5-kserve-84f65d66df-jd8qc" Nov 01 16:21:35.869590 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:35.869552 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-f01de5/auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt" podUID="03bcbe9c-db2f-4915-920f-3252117ac51a" containerName="main" containerID="cri-o://06e7bc5050b66c8deb38ea733de1acaff0af52c8e41f8aa6ba096d9970ee19fa" gracePeriod=45 Nov 01 16:21:36.188705 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:36.188677 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-f01de5/auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt" Nov 01 16:21:36.303041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:36.302861 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03bcbe9c-db2f-4915-920f-3252117ac51a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03bcbe9c-db2f-4915-920f-3252117ac51a-tmp-dir\") pod \"03bcbe9c-db2f-4915-920f-3252117ac51a\" (UID: \"03bcbe9c-db2f-4915-920f-3252117ac51a\") " Nov 01 16:21:36.303041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:36.302928 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03bcbe9c-db2f-4915-920f-3252117ac51a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03bcbe9c-db2f-4915-920f-3252117ac51a-model-cache\") pod \"03bcbe9c-db2f-4915-920f-3252117ac51a\" (UID: \"03bcbe9c-db2f-4915-920f-3252117ac51a\") " Nov 01 16:21:36.303596 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:36.303068 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03bcbe9c-db2f-4915-920f-3252117ac51a-home\" (UniqueName: \"kubernetes.io/empty-dir/03bcbe9c-db2f-4915-920f-3252117ac51a-home\") pod \"03bcbe9c-db2f-4915-920f-3252117ac51a\" (UID: \"03bcbe9c-db2f-4915-920f-3252117ac51a\") " Nov 01 16:21:36.303596 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:36.303153 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03bcbe9c-db2f-4915-920f-3252117ac51a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/03bcbe9c-db2f-4915-920f-3252117ac51a-dshm\") pod \"03bcbe9c-db2f-4915-920f-3252117ac51a\" (UID: \"03bcbe9c-db2f-4915-920f-3252117ac51a\") " Nov 01 16:21:36.303596 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:36.303206 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03bcbe9c-db2f-4915-920f-3252117ac51a-kube-api-access-rfkdh\" (UniqueName: \"kubernetes.io/projected/03bcbe9c-db2f-4915-920f-3252117ac51a-kube-api-access-rfkdh\") pod \"03bcbe9c-db2f-4915-920f-3252117ac51a\" (UID: \"03bcbe9c-db2f-4915-920f-3252117ac51a\") " Nov 01 16:21:36.303596 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:36.303407 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03bcbe9c-db2f-4915-920f-3252117ac51a-tls-certs\" (UniqueName: \"kubernetes.io/secret/03bcbe9c-db2f-4915-920f-3252117ac51a-tls-certs\") pod \"03bcbe9c-db2f-4915-920f-3252117ac51a\" (UID: \"03bcbe9c-db2f-4915-920f-3252117ac51a\") " Nov 01 16:21:36.360055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:36.304691 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03bcbe9c-db2f-4915-920f-3252117ac51a-model-cache" pod "03bcbe9c-db2f-4915-920f-3252117ac51a" (UID: "03bcbe9c-db2f-4915-920f-3252117ac51a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:21:36.360055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:36.306240 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03bcbe9c-db2f-4915-920f-3252117ac51a-home" pod "03bcbe9c-db2f-4915-920f-3252117ac51a" (UID: "03bcbe9c-db2f-4915-920f-3252117ac51a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:21:36.360055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:36.306506 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03bcbe9c-db2f-4915-920f-3252117ac51a-tmp-dir" pod "03bcbe9c-db2f-4915-920f-3252117ac51a" (UID: "03bcbe9c-db2f-4915-920f-3252117ac51a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:21:36.389550 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:36.389491 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03bcbe9c-db2f-4915-920f-3252117ac51a-dshm" pod "03bcbe9c-db2f-4915-920f-3252117ac51a" (UID: "03bcbe9c-db2f-4915-920f-3252117ac51a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:21:36.390127 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:36.390085 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03bcbe9c-db2f-4915-920f-3252117ac51a-kube-api-access-rfkdh" pod "03bcbe9c-db2f-4915-920f-3252117ac51a" (UID: "03bcbe9c-db2f-4915-920f-3252117ac51a"). InnerVolumeSpecName "kube-api-access-rfkdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:21:36.390398 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:36.390377 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03bcbe9c-db2f-4915-920f-3252117ac51a-tls-certs" pod "03bcbe9c-db2f-4915-920f-3252117ac51a" (UID: "03bcbe9c-db2f-4915-920f-3252117ac51a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:21:36.433169 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:36.430144 4363 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03bcbe9c-db2f-4915-920f-3252117ac51a-dshm\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:36.433169 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:36.430184 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfkdh\" (UniqueName: \"kubernetes.io/projected/03bcbe9c-db2f-4915-920f-3252117ac51a-kube-api-access-rfkdh\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:36.433169 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:36.430199 4363 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03bcbe9c-db2f-4915-920f-3252117ac51a-tls-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:36.433169 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:36.430236 4363 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03bcbe9c-db2f-4915-920f-3252117ac51a-tmp-dir\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:36.433169 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:36.430249 4363 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03bcbe9c-db2f-4915-920f-3252117ac51a-model-cache\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:36.433169 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:36.430263 4363 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03bcbe9c-db2f-4915-920f-3252117ac51a-home\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:36.957134 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:36.951306 4363 generic.go:356] "Generic (PLEG): container finished" podID="03bcbe9c-db2f-4915-920f-3252117ac51a" containerID="06e7bc5050b66c8deb38ea733de1acaff0af52c8e41f8aa6ba096d9970ee19fa" exitCode=0 Nov 01 16:21:36.957134 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:36.952633 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-f01de5/auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt" Nov 01 16:21:36.960609 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:36.959594 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-f01de5/auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt" event={"ID":"03bcbe9c-db2f-4915-920f-3252117ac51a","Type":"ContainerDied","Data":"06e7bc5050b66c8deb38ea733de1acaff0af52c8e41f8aa6ba096d9970ee19fa"} Nov 01 16:21:36.968354 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:36.963829 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-f01de5/auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt" event={"ID":"03bcbe9c-db2f-4915-920f-3252117ac51a","Type":"ContainerDied","Data":"b9b27e8445edada8a6f3fbb92e0e0d84e7ac35183181ab9d7a5031d56ea9f70c"} Nov 01 16:21:36.968354 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:36.963880 4363 scope.go:122] "RemoveContainer" containerID="06e7bc5050b66c8deb38ea733de1acaff0af52c8e41f8aa6ba096d9970ee19fa" Nov 01 16:21:37.020651 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:37.020624 4363 scope.go:122] "RemoveContainer" containerID="06e7bc5050b66c8deb38ea733de1acaff0af52c8e41f8aa6ba096d9970ee19fa" Nov 01 16:21:37.026447 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:21:37.026409 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06e7bc5050b66c8deb38ea733de1acaff0af52c8e41f8aa6ba096d9970ee19fa\": container with ID starting with 06e7bc5050b66c8deb38ea733de1acaff0af52c8e41f8aa6ba096d9970ee19fa not found: ID does not exist" containerID="06e7bc5050b66c8deb38ea733de1acaff0af52c8e41f8aa6ba096d9970ee19fa" Nov 01 16:21:37.026667 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:37.026638 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06e7bc5050b66c8deb38ea733de1acaff0af52c8e41f8aa6ba096d9970ee19fa"} err="failed to get container status \"06e7bc5050b66c8deb38ea733de1acaff0af52c8e41f8aa6ba096d9970ee19fa\": rpc error: code = NotFound desc = could not find container \"06e7bc5050b66c8deb38ea733de1acaff0af52c8e41f8aa6ba096d9970ee19fa\": container with ID starting with 06e7bc5050b66c8deb38ea733de1acaff0af52c8e41f8aa6ba096d9970ee19fa not found: ID does not exist" Nov 01 16:21:37.035906 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:37.035871 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-f01de5/auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt"] Nov 01 16:21:37.043755 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:37.043709 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-f01de5/auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt"] Nov 01 16:21:37.100754 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:37.099630 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-f01de5/auth-test-model-f01de5-kserve-6fcbc5846f-q9wnt" podUID="03bcbe9c-db2f-4915-920f-3252117ac51a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.212:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:21:37.163044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:37.162933 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-10b00a-659db85879-m4pxv" podUID="b39da192-a1b1-43f9-b6c4-b2186df9063f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:21:38.174382 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:38.174298 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-10b00a-5cdcfd8b5f-fwz5p" podUID="d3007207-2dac-46d0-970c-8138373dc953" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:21:38.814373 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:38.814338 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03bcbe9c-db2f-4915-920f-3252117ac51a" path="/var/lib/kubelet/pods/03bcbe9c-db2f-4915-920f-3252117ac51a/volumes" Nov 01 16:21:39.423040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:39.420395 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4"] Nov 01 16:21:39.423040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:39.421723 4363 cpu_manager.go:414] "containerMap: removing container" podUID="03bcbe9c-db2f-4915-920f-3252117ac51a" containerName="main" Nov 01 16:21:39.423040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:39.421745 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03bcbe9c-db2f-4915-920f-3252117ac51a" containerName="main" Nov 01 16:21:39.423040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:39.421975 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="03bcbe9c-db2f-4915-920f-3252117ac51a" containerName="main" Nov 01 16:21:39.453262 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:39.453224 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4"] Nov 01 16:21:39.453434 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:39.453362 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" Nov 01 16:21:39.469488 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:39.468595 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-5ccd7b-openshift-default-dockercfg-f92xw\"" Nov 01 16:21:39.574050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:39.538403 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f69b8bb-4aa6-4713-8b2f-138c5d358858-istio-envoy\") pod \"e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4\" (UID: \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\") " pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" Nov 01 16:21:39.574050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:39.538458 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f69b8bb-4aa6-4713-8b2f-138c5d358858-istio-token\") pod \"e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4\" (UID: \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\") " pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" Nov 01 16:21:39.574050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:39.538581 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f69b8bb-4aa6-4713-8b2f-138c5d358858-credential-socket\") pod \"e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4\" (UID: \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\") " pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" Nov 01 16:21:39.574050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:39.538623 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t26mt\" (UniqueName: \"kubernetes.io/projected/5f69b8bb-4aa6-4713-8b2f-138c5d358858-kube-api-access-t26mt\") pod \"e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4\" (UID: \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\") " pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" Nov 01 16:21:39.574050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:39.538680 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f69b8bb-4aa6-4713-8b2f-138c5d358858-workload-certs\") pod \"e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4\" (UID: \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\") " pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" Nov 01 16:21:39.574050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:39.538716 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f69b8bb-4aa6-4713-8b2f-138c5d358858-istiod-ca-cert\") pod \"e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4\" (UID: \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\") " pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" Nov 01 16:21:39.574050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:39.538792 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f69b8bb-4aa6-4713-8b2f-138c5d358858-istio-data\") pod \"e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4\" (UID: \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\") " pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" Nov 01 16:21:39.574050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:39.538891 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f69b8bb-4aa6-4713-8b2f-138c5d358858-istio-podinfo\") pod \"e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4\" (UID: \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\") " pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" Nov 01 16:21:39.574050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:39.538957 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f69b8bb-4aa6-4713-8b2f-138c5d358858-workload-socket\") pod \"e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4\" (UID: \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\") " pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" Nov 01 16:21:39.649039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:39.642188 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f69b8bb-4aa6-4713-8b2f-138c5d358858-credential-socket\") pod \"e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4\" (UID: \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\") " pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" Nov 01 16:21:39.649039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:39.642250 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t26mt\" (UniqueName: \"kubernetes.io/projected/5f69b8bb-4aa6-4713-8b2f-138c5d358858-kube-api-access-t26mt\") pod \"e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4\" (UID: \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\") " pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" Nov 01 16:21:39.649039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:39.642304 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f69b8bb-4aa6-4713-8b2f-138c5d358858-workload-certs\") pod \"e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4\" (UID: \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\") " pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" Nov 01 16:21:39.649039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:39.642334 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f69b8bb-4aa6-4713-8b2f-138c5d358858-istiod-ca-cert\") pod \"e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4\" (UID: \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\") " pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" Nov 01 16:21:39.649039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:39.642400 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f69b8bb-4aa6-4713-8b2f-138c5d358858-istio-data\") pod \"e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4\" (UID: \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\") " pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" Nov 01 16:21:39.649039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:39.642490 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f69b8bb-4aa6-4713-8b2f-138c5d358858-istio-podinfo\") pod \"e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4\" (UID: \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\") " pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" Nov 01 16:21:39.649039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:39.642548 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f69b8bb-4aa6-4713-8b2f-138c5d358858-workload-socket\") pod \"e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4\" (UID: \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\") " pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" Nov 01 16:21:39.649039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:39.642670 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f69b8bb-4aa6-4713-8b2f-138c5d358858-istio-envoy\") pod \"e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4\" (UID: \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\") " pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" Nov 01 16:21:39.649039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:39.642692 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f69b8bb-4aa6-4713-8b2f-138c5d358858-istio-token\") pod \"e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4\" (UID: \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\") " pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" Nov 01 16:21:39.649039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:39.643481 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f69b8bb-4aa6-4713-8b2f-138c5d358858-credential-socket\") pod \"e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4\" (UID: \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\") " pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" Nov 01 16:21:39.649039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:39.643928 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f69b8bb-4aa6-4713-8b2f-138c5d358858-workload-certs\") pod \"e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4\" (UID: \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\") " pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" Nov 01 16:21:39.649039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:39.644497 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f69b8bb-4aa6-4713-8b2f-138c5d358858-istiod-ca-cert\") pod \"e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4\" (UID: \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\") " pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" Nov 01 16:21:39.649039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:39.644741 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f69b8bb-4aa6-4713-8b2f-138c5d358858-istio-data\") pod \"e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4\" (UID: \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\") " pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" Nov 01 16:21:39.649039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:39.645273 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f69b8bb-4aa6-4713-8b2f-138c5d358858-workload-socket\") pod \"e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4\" (UID: \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\") " pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" Nov 01 16:21:39.675099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:39.668286 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f69b8bb-4aa6-4713-8b2f-138c5d358858-istio-token\") pod \"e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4\" (UID: \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\") " pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" Nov 01 16:21:39.677512 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:39.677480 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f69b8bb-4aa6-4713-8b2f-138c5d358858-istio-envoy\") pod \"e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4\" (UID: \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\") " pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" Nov 01 16:21:39.678367 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:39.678341 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f69b8bb-4aa6-4713-8b2f-138c5d358858-istio-podinfo\") pod \"e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4\" (UID: \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\") " pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" Nov 01 16:21:39.683680 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:39.683631 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t26mt\" (UniqueName: \"kubernetes.io/projected/5f69b8bb-4aa6-4713-8b2f-138c5d358858-kube-api-access-t26mt\") pod \"e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4\" (UID: \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\") " pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" Nov 01 16:21:39.830570 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:39.830530 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" Nov 01 16:21:40.251086 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:40.248887 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4"] Nov 01 16:21:40.317042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:40.301981 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:21:40.317042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:40.302101 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:21:40.317042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:40.302150 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:21:40.555313 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:40.555234 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-9a482b/sub-test-model-9a482b-kserve-d54b499b5-4svj8" Nov 01 16:21:40.568346 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:40.568322 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-9a482b/sub-test-model-9a482b-kserve-d54b499b5-4svj8" Nov 01 16:21:41.059666 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:41.059625 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" event={"ID":"5f69b8bb-4aa6-4713-8b2f-138c5d358858","Type":"ContainerStarted","Data":"68171b1bd2001cb00c9191f8272dfe0efca9dd69364dfbdaad7ff19e1bd3b8a4"} Nov 01 16:21:41.059876 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:41.059690 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" event={"ID":"5f69b8bb-4aa6-4713-8b2f-138c5d358858","Type":"ContainerStarted","Data":"879fa1d7d9ec9c2481483ddb5d6901e822d827b4b9256a615b30276a1cfd9845"} Nov 01 16:21:41.071562 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:41.071532 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-9a482b/sub-test-model-9a482b-kserve-d54b499b5-4svj8" Nov 01 16:21:41.120531 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:41.120467 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" podStartSLOduration=2.120449995 podStartE2EDuration="2.120449995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:21:41.101497726 +0000 UTC m=+2034.433235370" watchObservedRunningTime="2026-11-01 16:21:41.120449995 +0000 UTC m=+2034.452187646" Nov 01 16:21:41.831445 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:41.831409 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" Nov 01 16:21:41.866609 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:41.866079 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" Nov 01 16:21:42.070819 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:42.070660 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" Nov 01 16:21:42.072437 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:42.072411 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" Nov 01 16:21:47.162484 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:47.162443 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-10b00a-659db85879-m4pxv" podUID="b39da192-a1b1-43f9-b6c4-b2186df9063f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:21:47.465739 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:47.465638 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc"] Nov 01 16:21:47.472364 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:47.472338 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" Nov 01 16:21:47.555955 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:47.555920 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc"] Nov 01 16:21:47.558654 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:47.557547 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77a6ec04-31e5-4d7e-a829-a4a0704de9db-istio-token\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc\" (UID: \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" Nov 01 16:21:47.558654 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:47.557605 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77a6ec04-31e5-4d7e-a829-a4a0704de9db-istiod-ca-cert\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc\" (UID: \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" Nov 01 16:21:47.558654 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:47.557688 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77a6ec04-31e5-4d7e-a829-a4a0704de9db-credential-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc\" (UID: \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" Nov 01 16:21:47.558654 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:47.557718 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77a6ec04-31e5-4d7e-a829-a4a0704de9db-istio-podinfo\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc\" (UID: \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" Nov 01 16:21:47.558654 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:47.557890 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77a6ec04-31e5-4d7e-a829-a4a0704de9db-istio-envoy\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc\" (UID: \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" Nov 01 16:21:47.558654 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:47.557920 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twtkw\" (UniqueName: \"kubernetes.io/projected/77a6ec04-31e5-4d7e-a829-a4a0704de9db-kube-api-access-twtkw\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc\" (UID: \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" Nov 01 16:21:47.558654 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:47.558033 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77a6ec04-31e5-4d7e-a829-a4a0704de9db-workload-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc\" (UID: \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" Nov 01 16:21:47.558654 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:47.558143 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77a6ec04-31e5-4d7e-a829-a4a0704de9db-istio-data\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc\" (UID: \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" Nov 01 16:21:47.558654 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:47.558228 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77a6ec04-31e5-4d7e-a829-a4a0704de9db-workload-certs\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc\" (UID: \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" Nov 01 16:21:47.659645 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:47.659610 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77a6ec04-31e5-4d7e-a829-a4a0704de9db-workload-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc\" (UID: \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" Nov 01 16:21:47.659836 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:47.659681 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77a6ec04-31e5-4d7e-a829-a4a0704de9db-istio-data\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc\" (UID: \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" Nov 01 16:21:47.659836 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:47.659726 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77a6ec04-31e5-4d7e-a829-a4a0704de9db-workload-certs\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc\" (UID: \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" Nov 01 16:21:47.659836 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:47.659782 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77a6ec04-31e5-4d7e-a829-a4a0704de9db-istio-token\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc\" (UID: \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" Nov 01 16:21:47.659836 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:47.659803 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77a6ec04-31e5-4d7e-a829-a4a0704de9db-istiod-ca-cert\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc\" (UID: \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" Nov 01 16:21:47.660117 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:47.659851 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77a6ec04-31e5-4d7e-a829-a4a0704de9db-credential-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc\" (UID: \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" Nov 01 16:21:47.660117 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:47.659881 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77a6ec04-31e5-4d7e-a829-a4a0704de9db-istio-podinfo\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc\" (UID: \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" Nov 01 16:21:47.660117 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:47.660007 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77a6ec04-31e5-4d7e-a829-a4a0704de9db-istio-envoy\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc\" (UID: \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" Nov 01 16:21:47.660117 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:47.660093 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twtkw\" (UniqueName: \"kubernetes.io/projected/77a6ec04-31e5-4d7e-a829-a4a0704de9db-kube-api-access-twtkw\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc\" (UID: \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" Nov 01 16:21:47.660117 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:47.660096 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77a6ec04-31e5-4d7e-a829-a4a0704de9db-istio-data\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc\" (UID: \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" Nov 01 16:21:47.660389 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:47.660349 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77a6ec04-31e5-4d7e-a829-a4a0704de9db-workload-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc\" (UID: \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" Nov 01 16:21:47.660448 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:47.660391 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77a6ec04-31e5-4d7e-a829-a4a0704de9db-workload-certs\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc\" (UID: \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" Nov 01 16:21:47.660490 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:47.660473 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77a6ec04-31e5-4d7e-a829-a4a0704de9db-credential-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc\" (UID: \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" Nov 01 16:21:47.660909 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:47.660883 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77a6ec04-31e5-4d7e-a829-a4a0704de9db-istiod-ca-cert\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc\" (UID: \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" Nov 01 16:21:47.662394 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:47.662374 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77a6ec04-31e5-4d7e-a829-a4a0704de9db-istio-envoy\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc\" (UID: \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" Nov 01 16:21:47.662532 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:47.662512 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77a6ec04-31e5-4d7e-a829-a4a0704de9db-istio-podinfo\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc\" (UID: \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" Nov 01 16:21:47.668804 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:47.668781 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77a6ec04-31e5-4d7e-a829-a4a0704de9db-istio-token\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc\" (UID: \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" Nov 01 16:21:47.669359 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:47.669340 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twtkw\" (UniqueName: \"kubernetes.io/projected/77a6ec04-31e5-4d7e-a829-a4a0704de9db-kube-api-access-twtkw\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc\" (UID: \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" Nov 01 16:21:47.877453 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:47.877139 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" Nov 01 16:21:48.173890 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:48.173857 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-10b00a-5cdcfd8b5f-fwz5p" podUID="d3007207-2dac-46d0-970c-8138373dc953" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:21:48.204994 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:48.204955 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc"] Nov 01 16:21:48.217947 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:48.217894 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:21:48.218143 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:48.217992 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:21:48.218143 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:48.218056 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:21:48.341909 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:21:48.341877 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:21:48.851282 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:48.851255 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-10b00a-5cdcfd8b5f-fwz5p" Nov 01 16:21:48.906257 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:48.906225 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-10b00a-659db85879-m4pxv" Nov 01 16:21:49.011146 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:49.011117 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3007207-2dac-46d0-970c-8138373dc953-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3007207-2dac-46d0-970c-8138373dc953-plugins-config-volume\") pod \"d3007207-2dac-46d0-970c-8138373dc953\" (UID: \"d3007207-2dac-46d0-970c-8138373dc953\") " Nov 01 16:21:49.011318 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:49.011204 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b39da192-a1b1-43f9-b6c4-b2186df9063f-kube-api-access-kkthr\" (UniqueName: \"kubernetes.io/projected/b39da192-a1b1-43f9-b6c4-b2186df9063f-kube-api-access-kkthr\") pod \"b39da192-a1b1-43f9-b6c4-b2186df9063f\" (UID: \"b39da192-a1b1-43f9-b6c4-b2186df9063f\") " Nov 01 16:21:49.011318 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:49.011311 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3007207-2dac-46d0-970c-8138373dc953-kube-api-access-qc8gr\" (UniqueName: \"kubernetes.io/projected/d3007207-2dac-46d0-970c-8138373dc953-kube-api-access-qc8gr\") pod \"d3007207-2dac-46d0-970c-8138373dc953\" (UID: \"d3007207-2dac-46d0-970c-8138373dc953\") " Nov 01 16:21:49.011598 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:49.011568 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b39da192-a1b1-43f9-b6c4-b2186df9063f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b39da192-a1b1-43f9-b6c4-b2186df9063f-plugins-config-volume\") pod \"b39da192-a1b1-43f9-b6c4-b2186df9063f\" (UID: \"b39da192-a1b1-43f9-b6c4-b2186df9063f\") " Nov 01 16:21:49.011703 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:49.011603 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3007207-2dac-46d0-970c-8138373dc953-plugins-config-volume" pod "d3007207-2dac-46d0-970c-8138373dc953" (UID: "d3007207-2dac-46d0-970c-8138373dc953"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:21:49.012102 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:49.012080 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b39da192-a1b1-43f9-b6c4-b2186df9063f-plugins-config-volume" pod "b39da192-a1b1-43f9-b6c4-b2186df9063f" (UID: "b39da192-a1b1-43f9-b6c4-b2186df9063f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:21:49.012412 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:49.012391 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3007207-2dac-46d0-970c-8138373dc953-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:49.012493 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:49.012416 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b39da192-a1b1-43f9-b6c4-b2186df9063f-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:49.014137 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:49.014108 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b39da192-a1b1-43f9-b6c4-b2186df9063f-kube-api-access-kkthr" pod "b39da192-a1b1-43f9-b6c4-b2186df9063f" (UID: "b39da192-a1b1-43f9-b6c4-b2186df9063f"). InnerVolumeSpecName "kube-api-access-kkthr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:21:49.014628 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:49.014601 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3007207-2dac-46d0-970c-8138373dc953-kube-api-access-qc8gr" pod "d3007207-2dac-46d0-970c-8138373dc953" (UID: "d3007207-2dac-46d0-970c-8138373dc953"). InnerVolumeSpecName "kube-api-access-qc8gr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:21:49.134042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:49.126158 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc8gr\" (UniqueName: \"kubernetes.io/projected/d3007207-2dac-46d0-970c-8138373dc953-kube-api-access-qc8gr\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:49.134042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:49.126198 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkthr\" (UniqueName: \"kubernetes.io/projected/b39da192-a1b1-43f9-b6c4-b2186df9063f-kube-api-access-kkthr\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:49.228058 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:49.227977 4363 generic.go:356] "Generic (PLEG): container finished" podID="1b180ef5-70b3-4bf2-901b-db280a232bde" containerID="63409c055dbbf5803b5aac62cd48411547732fa5158dbd6d67f1edcb277d5d0a" exitCode=0 Nov 01 16:21:49.228565 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:49.228061 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" event={"ID":"1b180ef5-70b3-4bf2-901b-db280a232bde","Type":"ContainerDied","Data":"63409c055dbbf5803b5aac62cd48411547732fa5158dbd6d67f1edcb277d5d0a"} Nov 01 16:21:49.228565 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:49.228440 4363 scope.go:122] "RemoveContainer" containerID="63409c055dbbf5803b5aac62cd48411547732fa5158dbd6d67f1edcb277d5d0a" Nov 01 16:21:49.230398 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:49.230363 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:21:49.230529 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:49.230456 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:21:49.230529 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:49.230501 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:21:49.230797 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:49.230775 4363 generic.go:356] "Generic (PLEG): container finished" podID="b39da192-a1b1-43f9-b6c4-b2186df9063f" containerID="0a76156d8f3310f0e93ae9fe38ea98cfd8af56c01af6b4467c011d68eae2f380" exitCode=0 Nov 01 16:21:49.230885 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:49.230868 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-10b00a-659db85879-m4pxv" event={"ID":"b39da192-a1b1-43f9-b6c4-b2186df9063f","Type":"ContainerDied","Data":"0a76156d8f3310f0e93ae9fe38ea98cfd8af56c01af6b4467c011d68eae2f380"} Nov 01 16:21:49.230945 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:49.230915 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-10b00a-659db85879-m4pxv" event={"ID":"b39da192-a1b1-43f9-b6c4-b2186df9063f","Type":"ContainerDied","Data":"0c35554e3be4974c58abcc91938a608e7a23c88f6a3593e4fbfb94a070b4421d"} Nov 01 16:21:49.230945 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:49.230938 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-10b00a-659db85879-m4pxv" Nov 01 16:21:49.231064 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:49.230946 4363 scope.go:122] "RemoveContainer" containerID="0a76156d8f3310f0e93ae9fe38ea98cfd8af56c01af6b4467c011d68eae2f380" Nov 01 16:21:49.239983 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:49.239959 4363 generic.go:356] "Generic (PLEG): container finished" podID="d3007207-2dac-46d0-970c-8138373dc953" containerID="58f84855ecde8d855316da17f8bb0bf0bac8979a691786fd023e1680f930dbda" exitCode=0 Nov 01 16:21:49.240115 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:49.240001 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-10b00a-5cdcfd8b5f-fwz5p" Nov 01 16:21:49.240115 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:49.240101 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-10b00a-5cdcfd8b5f-fwz5p" event={"ID":"d3007207-2dac-46d0-970c-8138373dc953","Type":"ContainerDied","Data":"58f84855ecde8d855316da17f8bb0bf0bac8979a691786fd023e1680f930dbda"} Nov 01 16:21:49.240295 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:49.240137 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-10b00a-5cdcfd8b5f-fwz5p" event={"ID":"d3007207-2dac-46d0-970c-8138373dc953","Type":"ContainerDied","Data":"c1d9a525b387441a6b0d7a7b05822c827960fc85cd122af6a2bf57a98e065922"} Nov 01 16:21:49.242426 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:49.242387 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" event={"ID":"77a6ec04-31e5-4d7e-a829-a4a0704de9db","Type":"ContainerStarted","Data":"fa3edfd3e8a5051dd7849547b19960f645038e9203313e3cc18a81bc2ce8f196"} Nov 01 16:21:49.242540 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:49.242440 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" event={"ID":"77a6ec04-31e5-4d7e-a829-a4a0704de9db","Type":"ContainerStarted","Data":"e25152270718b002bd773a8a47ee01071e3e61ace84809ad611edd0032db2dd5"} Nov 01 16:21:49.263254 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:49.262595 4363 scope.go:122] "RemoveContainer" containerID="0a76156d8f3310f0e93ae9fe38ea98cfd8af56c01af6b4467c011d68eae2f380" Nov 01 16:21:49.263254 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:21:49.262927 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a76156d8f3310f0e93ae9fe38ea98cfd8af56c01af6b4467c011d68eae2f380\": container with ID starting with 0a76156d8f3310f0e93ae9fe38ea98cfd8af56c01af6b4467c011d68eae2f380 not found: ID does not exist" containerID="0a76156d8f3310f0e93ae9fe38ea98cfd8af56c01af6b4467c011d68eae2f380" Nov 01 16:21:49.263254 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:49.262956 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a76156d8f3310f0e93ae9fe38ea98cfd8af56c01af6b4467c011d68eae2f380"} err="failed to get container status \"0a76156d8f3310f0e93ae9fe38ea98cfd8af56c01af6b4467c011d68eae2f380\": rpc error: code = NotFound desc = could not find container \"0a76156d8f3310f0e93ae9fe38ea98cfd8af56c01af6b4467c011d68eae2f380\": container with ID starting with 0a76156d8f3310f0e93ae9fe38ea98cfd8af56c01af6b4467c011d68eae2f380 not found: ID does not exist" Nov 01 16:21:49.263254 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:49.262978 4363 scope.go:122] "RemoveContainer" containerID="58f84855ecde8d855316da17f8bb0bf0bac8979a691786fd023e1680f930dbda" Nov 01 16:21:49.285550 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:49.285509 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" podStartSLOduration=2.285496307 podStartE2EDuration="2.285496307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:21:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:21:49.284457682 +0000 UTC m=+2042.616195333" watchObservedRunningTime="2026-11-01 16:21:49.285496307 +0000 UTC m=+2042.617233956" Nov 01 16:21:49.317129 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:49.317103 4363 scope.go:122] "RemoveContainer" containerID="58f84855ecde8d855316da17f8bb0bf0bac8979a691786fd023e1680f930dbda" Nov 01 16:21:49.322838 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:21:49.322759 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58f84855ecde8d855316da17f8bb0bf0bac8979a691786fd023e1680f930dbda\": container with ID starting with 58f84855ecde8d855316da17f8bb0bf0bac8979a691786fd023e1680f930dbda not found: ID does not exist" containerID="58f84855ecde8d855316da17f8bb0bf0bac8979a691786fd023e1680f930dbda" Nov 01 16:21:49.322838 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:49.322802 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58f84855ecde8d855316da17f8bb0bf0bac8979a691786fd023e1680f930dbda"} err="failed to get container status \"58f84855ecde8d855316da17f8bb0bf0bac8979a691786fd023e1680f930dbda\": rpc error: code = NotFound desc = could not find container \"58f84855ecde8d855316da17f8bb0bf0bac8979a691786fd023e1680f930dbda\": container with ID starting with 58f84855ecde8d855316da17f8bb0bf0bac8979a691786fd023e1680f930dbda not found: ID does not exist" Nov 01 16:21:49.327572 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:49.327554 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-10b00a-659db85879-m4pxv"] Nov 01 16:21:49.339376 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:49.339306 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-10b00a-659db85879-m4pxv"] Nov 01 16:21:49.346409 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:49.346357 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-10b00a-5cdcfd8b5f-fwz5p"] Nov 01 16:21:49.353893 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:49.353864 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-10b00a-5cdcfd8b5f-fwz5p"] Nov 01 16:21:49.878528 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:49.878166 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" Nov 01 16:21:49.896503 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:49.895378 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body= Nov 01 16:21:49.896503 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:49.895448 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" podUID="77a6ec04-31e5-4d7e-a829-a4a0704de9db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" Nov 01 16:21:50.252598 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:50.252501 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" event={"ID":"1b180ef5-70b3-4bf2-901b-db280a232bde","Type":"ContainerStarted","Data":"82aaead6ae9f4be0f0297f718b4fa0f607322da5fe981cdcb7c21aac92d6e198"} Nov 01 16:21:50.697040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:50.686134 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" Nov 01 16:21:50.718041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:50.698635 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" Nov 01 16:21:50.718041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:50.698790 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Nov 01 16:21:50.718041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:50.698838 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" podUID="1b180ef5-70b3-4bf2-901b-db280a232bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Nov 01 16:21:50.853940 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:50.853900 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b39da192-a1b1-43f9-b6c4-b2186df9063f" path="/var/lib/kubelet/pods/b39da192-a1b1-43f9-b6c4-b2186df9063f/volumes" Nov 01 16:21:50.854589 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:50.854570 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3007207-2dac-46d0-970c-8138373dc953" path="/var/lib/kubelet/pods/d3007207-2dac-46d0-970c-8138373dc953/volumes" Nov 01 16:21:50.885040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:50.882359 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body= Nov 01 16:21:50.885040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:50.882413 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" podUID="77a6ec04-31e5-4d7e-a829-a4a0704de9db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" Nov 01 16:21:51.692048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:51.691415 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Nov 01 16:21:51.692048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:51.691484 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" podUID="1b180ef5-70b3-4bf2-901b-db280a232bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Nov 01 16:21:51.881393 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:51.881356 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body= Nov 01 16:21:51.881671 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:51.881639 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" podUID="77a6ec04-31e5-4d7e-a829-a4a0704de9db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" Nov 01 16:21:52.692150 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:52.692116 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Nov 01 16:21:52.692745 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:52.692709 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" podUID="1b180ef5-70b3-4bf2-901b-db280a232bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Nov 01 16:21:52.900042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:52.891536 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body= Nov 01 16:21:52.900042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:52.891608 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" podUID="77a6ec04-31e5-4d7e-a829-a4a0704de9db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" Nov 01 16:21:53.045808 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:53.043838 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a4240a-745976f465-st69v"] Nov 01 16:21:53.051812 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:53.051780 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a4240a-86f876f9cf-9bqvq"] Nov 01 16:21:53.697041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:53.693949 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Nov 01 16:21:53.697041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:53.694048 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" podUID="1b180ef5-70b3-4bf2-901b-db280a232bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Nov 01 16:21:53.878347 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:53.878309 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body= Nov 01 16:21:53.878649 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:53.878615 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" podUID="77a6ec04-31e5-4d7e-a829-a4a0704de9db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" Nov 01 16:21:54.694043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:54.691419 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Nov 01 16:21:54.694043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:54.691479 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" podUID="1b180ef5-70b3-4bf2-901b-db280a232bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Nov 01 16:21:54.885039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:54.880705 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body= Nov 01 16:21:54.885039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:54.880763 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" podUID="77a6ec04-31e5-4d7e-a829-a4a0704de9db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" Nov 01 16:21:55.695477 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:55.695436 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Nov 01 16:21:55.695769 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:55.695734 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" podUID="1b180ef5-70b3-4bf2-901b-db280a232bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Nov 01 16:21:55.915597 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:55.915562 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" Nov 01 16:21:55.917493 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:55.917462 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" Nov 01 16:21:55.918412 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:55.918388 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" Nov 01 16:21:56.696132 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:56.696101 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" Nov 01 16:21:56.697371 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:56.697337 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" Nov 01 16:21:58.045610 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:58.045569 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-a4240a-745976f465-st69v" podUID="c8c062a2-3461-4bbe-ac11-81966a3cec16" containerName="payload-processing" containerID="cri-o://c9eabc2a9000378c0a532d6557de1af55e08161638f9cb28f7c4fed6f308a9ba" gracePeriod=25 Nov 01 16:21:58.053103 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:21:58.053067 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a4240a-86f876f9cf-9bqvq" podUID="1964cd1c-696f-4f45-83ba-8dec81dbc624" containerName="payload-pre-processing" containerID="cri-o://f45baca75c75cadf94e4a74a6f91e53a1304d6f802562fe14b7a39dd9e9b89ba" gracePeriod=25 Nov 01 16:22:00.300986 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:00.300945 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a4240a-745976f465-st69v" podUID="c8c062a2-3461-4bbe-ac11-81966a3cec16" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:22:00.310105 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:00.310067 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a4240a-86f876f9cf-9bqvq" podUID="1964cd1c-696f-4f45-83ba-8dec81dbc624" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:22:00.908070 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:00.902935 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf"] Nov 01 16:22:00.908559 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:00.908521 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" podUID="0e6063b9-04b2-42f5-ac6b-cea0dc65007a" containerName="istio-proxy" containerID="cri-o://1fbc16b99a880843de096ac7d24277b648d33dd2c94140dc3dbd165272de4977" gracePeriod=30 Nov 01 16:22:01.027148 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:01.027111 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5"] Nov 01 16:22:01.028378 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:01.028349 4363 cpu_manager.go:414] "containerMap: removing container" podUID="b39da192-a1b1-43f9-b6c4-b2186df9063f" containerName="payload-processing" Nov 01 16:22:01.028378 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:01.028376 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b39da192-a1b1-43f9-b6c4-b2186df9063f" containerName="payload-processing" Nov 01 16:22:01.028595 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:01.028417 4363 cpu_manager.go:414] "containerMap: removing container" podUID="d3007207-2dac-46d0-970c-8138373dc953" containerName="payload-pre-processing" Nov 01 16:22:01.028595 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:01.028426 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3007207-2dac-46d0-970c-8138373dc953" containerName="payload-pre-processing" Nov 01 16:22:01.028712 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:01.028647 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3007207-2dac-46d0-970c-8138373dc953" containerName="payload-pre-processing" Nov 01 16:22:01.028712 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:01.028673 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="b39da192-a1b1-43f9-b6c4-b2186df9063f" containerName="payload-processing" Nov 01 16:22:01.035896 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:01.035629 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" Nov 01 16:22:01.047644 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:01.047606 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-f01de5-openshift-default-dockercfg-czkss\"" Nov 01 16:22:01.054376 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:01.054352 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5"] Nov 01 16:22:01.161188 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:01.161092 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a850693a-07f7-4fdf-9ad3-4d08559c9737-istio-envoy\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5\" (UID: \"a850693a-07f7-4fdf-9ad3-4d08559c9737\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" Nov 01 16:22:01.161351 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:01.161215 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmk4k\" (UniqueName: \"kubernetes.io/projected/a850693a-07f7-4fdf-9ad3-4d08559c9737-kube-api-access-dmk4k\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5\" (UID: \"a850693a-07f7-4fdf-9ad3-4d08559c9737\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" Nov 01 16:22:01.161351 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:01.161289 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a850693a-07f7-4fdf-9ad3-4d08559c9737-workload-socket\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5\" (UID: \"a850693a-07f7-4fdf-9ad3-4d08559c9737\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" Nov 01 16:22:01.161463 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:01.161423 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a850693a-07f7-4fdf-9ad3-4d08559c9737-istio-podinfo\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5\" (UID: \"a850693a-07f7-4fdf-9ad3-4d08559c9737\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" Nov 01 16:22:01.161528 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:01.161479 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a850693a-07f7-4fdf-9ad3-4d08559c9737-istio-data\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5\" (UID: \"a850693a-07f7-4fdf-9ad3-4d08559c9737\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" Nov 01 16:22:01.161908 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:01.161511 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a850693a-07f7-4fdf-9ad3-4d08559c9737-credential-socket\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5\" (UID: \"a850693a-07f7-4fdf-9ad3-4d08559c9737\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" Nov 01 16:22:01.162267 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:01.162223 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a850693a-07f7-4fdf-9ad3-4d08559c9737-istiod-ca-cert\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5\" (UID: \"a850693a-07f7-4fdf-9ad3-4d08559c9737\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" Nov 01 16:22:01.162728 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:01.162706 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a850693a-07f7-4fdf-9ad3-4d08559c9737-istio-token\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5\" (UID: \"a850693a-07f7-4fdf-9ad3-4d08559c9737\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" Nov 01 16:22:01.162835 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:01.162791 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a850693a-07f7-4fdf-9ad3-4d08559c9737-workload-certs\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5\" (UID: \"a850693a-07f7-4fdf-9ad3-4d08559c9737\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" Nov 01 16:22:01.264502 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:01.264467 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a850693a-07f7-4fdf-9ad3-4d08559c9737-workload-certs\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5\" (UID: \"a850693a-07f7-4fdf-9ad3-4d08559c9737\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" Nov 01 16:22:01.264716 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:01.264524 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a850693a-07f7-4fdf-9ad3-4d08559c9737-istio-envoy\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5\" (UID: \"a850693a-07f7-4fdf-9ad3-4d08559c9737\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" Nov 01 16:22:01.264716 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:01.264594 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmk4k\" (UniqueName: \"kubernetes.io/projected/a850693a-07f7-4fdf-9ad3-4d08559c9737-kube-api-access-dmk4k\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5\" (UID: \"a850693a-07f7-4fdf-9ad3-4d08559c9737\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" Nov 01 16:22:01.264716 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:01.264654 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a850693a-07f7-4fdf-9ad3-4d08559c9737-workload-socket\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5\" (UID: \"a850693a-07f7-4fdf-9ad3-4d08559c9737\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" Nov 01 16:22:01.264896 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:01.264732 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a850693a-07f7-4fdf-9ad3-4d08559c9737-istio-podinfo\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5\" (UID: \"a850693a-07f7-4fdf-9ad3-4d08559c9737\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" Nov 01 16:22:01.264896 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:01.264779 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a850693a-07f7-4fdf-9ad3-4d08559c9737-istio-data\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5\" (UID: \"a850693a-07f7-4fdf-9ad3-4d08559c9737\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" Nov 01 16:22:01.264896 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:01.264804 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a850693a-07f7-4fdf-9ad3-4d08559c9737-credential-socket\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5\" (UID: \"a850693a-07f7-4fdf-9ad3-4d08559c9737\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" Nov 01 16:22:01.264896 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:01.264886 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a850693a-07f7-4fdf-9ad3-4d08559c9737-istiod-ca-cert\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5\" (UID: \"a850693a-07f7-4fdf-9ad3-4d08559c9737\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" Nov 01 16:22:01.265124 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:01.264918 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a850693a-07f7-4fdf-9ad3-4d08559c9737-workload-certs\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5\" (UID: \"a850693a-07f7-4fdf-9ad3-4d08559c9737\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" Nov 01 16:22:01.265124 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:01.264961 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a850693a-07f7-4fdf-9ad3-4d08559c9737-istio-token\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5\" (UID: \"a850693a-07f7-4fdf-9ad3-4d08559c9737\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" Nov 01 16:22:01.265124 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:01.265089 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a850693a-07f7-4fdf-9ad3-4d08559c9737-workload-socket\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5\" (UID: \"a850693a-07f7-4fdf-9ad3-4d08559c9737\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" Nov 01 16:22:01.265282 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:01.265244 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a850693a-07f7-4fdf-9ad3-4d08559c9737-credential-socket\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5\" (UID: \"a850693a-07f7-4fdf-9ad3-4d08559c9737\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" Nov 01 16:22:01.265338 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:01.265308 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a850693a-07f7-4fdf-9ad3-4d08559c9737-istio-data\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5\" (UID: \"a850693a-07f7-4fdf-9ad3-4d08559c9737\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" Nov 01 16:22:01.265568 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:01.265549 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a850693a-07f7-4fdf-9ad3-4d08559c9737-istiod-ca-cert\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5\" (UID: \"a850693a-07f7-4fdf-9ad3-4d08559c9737\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" Nov 01 16:22:01.267150 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:01.267116 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a850693a-07f7-4fdf-9ad3-4d08559c9737-istio-envoy\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5\" (UID: \"a850693a-07f7-4fdf-9ad3-4d08559c9737\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" Nov 01 16:22:01.269569 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:01.269545 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a850693a-07f7-4fdf-9ad3-4d08559c9737-istio-podinfo\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5\" (UID: \"a850693a-07f7-4fdf-9ad3-4d08559c9737\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" Nov 01 16:22:01.272681 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:01.272650 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a850693a-07f7-4fdf-9ad3-4d08559c9737-istio-token\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5\" (UID: \"a850693a-07f7-4fdf-9ad3-4d08559c9737\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" Nov 01 16:22:01.273110 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:01.273081 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmk4k\" (UniqueName: \"kubernetes.io/projected/a850693a-07f7-4fdf-9ad3-4d08559c9737-kube-api-access-dmk4k\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5\" (UID: \"a850693a-07f7-4fdf-9ad3-4d08559c9737\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" Nov 01 16:22:01.392055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:01.364338 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" Nov 01 16:22:01.829525 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:01.829271 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5"] Nov 01 16:22:01.830745 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:01.830705 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:22:01.830839 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:01.830791 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:22:01.830839 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:01.830831 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:22:02.959700 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:02.959654 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" event={"ID":"a850693a-07f7-4fdf-9ad3-4d08559c9737","Type":"ContainerStarted","Data":"50b4d3471a0b773905f9182ceb9f7b6c32429f63af7fbc8edd4e4281bc597a07"} Nov 01 16:22:02.966040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:02.960392 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" event={"ID":"a850693a-07f7-4fdf-9ad3-4d08559c9737","Type":"ContainerStarted","Data":"857eb2e018576cd281fb6632826220eb3dbb12d76d675f72671369ba84bb7bc7"} Nov 01 16:22:02.998042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:02.988930 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" podStartSLOduration=1.98891206 podStartE2EDuration="1.98891206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:22:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:22:02.985450934 +0000 UTC m=+2056.317188585" watchObservedRunningTime="2026-11-01 16:22:02.98891206 +0000 UTC m=+2056.320649711" Nov 01 16:22:03.383042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:03.381405 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" Nov 01 16:22:03.404120 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:03.391448 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Nov 01 16:22:03.404120 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:03.391508 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" podUID="a850693a-07f7-4fdf-9ad3-4d08559c9737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Nov 01 16:22:04.386041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:04.385563 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Nov 01 16:22:04.386041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:04.385635 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" podUID="a850693a-07f7-4fdf-9ad3-4d08559c9737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Nov 01 16:22:05.383130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:05.382404 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Nov 01 16:22:05.383130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:05.382463 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" podUID="a850693a-07f7-4fdf-9ad3-4d08559c9737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Nov 01 16:22:06.009763 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:06.009732 4363 generic.go:356] "Generic (PLEG): container finished" podID="0e6063b9-04b2-42f5-ac6b-cea0dc65007a" containerID="1fbc16b99a880843de096ac7d24277b648d33dd2c94140dc3dbd165272de4977" exitCode=0 Nov 01 16:22:06.010168 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:06.009805 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" event={"ID":"0e6063b9-04b2-42f5-ac6b-cea0dc65007a","Type":"ContainerDied","Data":"1fbc16b99a880843de096ac7d24277b648d33dd2c94140dc3dbd165272de4977"} Nov 01 16:22:06.378501 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:06.378466 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Nov 01 16:22:06.378861 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:06.378827 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" podUID="a850693a-07f7-4fdf-9ad3-4d08559c9737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Nov 01 16:22:06.831730 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:06.831394 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" Nov 01 16:22:06.868633 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:06.868596 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-credential-socket\") pod \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\" (UID: \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\") " Nov 01 16:22:06.869888 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:06.868674 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-kube-api-access-xgqwd\" (UniqueName: \"kubernetes.io/projected/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-kube-api-access-xgqwd\") pod \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\" (UID: \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\") " Nov 01 16:22:06.869888 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:06.868707 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-istiod-ca-cert\") pod \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\" (UID: \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\") " Nov 01 16:22:06.869888 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:06.868815 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-istio-envoy\") pod \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\" (UID: \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\") " Nov 01 16:22:06.869888 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:06.868852 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-workload-certs\") pod \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\" (UID: \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\") " Nov 01 16:22:06.869888 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:06.868898 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-istio-data\") pod \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\" (UID: \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\") " Nov 01 16:22:06.869888 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:06.868997 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-istio-token\" (UniqueName: \"kubernetes.io/projected/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-istio-token\") pod \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\" (UID: \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\") " Nov 01 16:22:06.869888 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:06.869102 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-workload-socket\") pod \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\" (UID: \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\") " Nov 01 16:22:06.869888 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:06.869184 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-istio-podinfo\") pod \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\" (UID: \"0e6063b9-04b2-42f5-ac6b-cea0dc65007a\") " Nov 01 16:22:06.870975 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:06.870489 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-credential-socket" pod "0e6063b9-04b2-42f5-ac6b-cea0dc65007a" (UID: "0e6063b9-04b2-42f5-ac6b-cea0dc65007a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:22:06.872289 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:06.871402 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-workload-certs" pod "0e6063b9-04b2-42f5-ac6b-cea0dc65007a" (UID: "0e6063b9-04b2-42f5-ac6b-cea0dc65007a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:22:06.872705 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:06.872501 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-workload-socket" pod "0e6063b9-04b2-42f5-ac6b-cea0dc65007a" (UID: "0e6063b9-04b2-42f5-ac6b-cea0dc65007a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:22:06.895165 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:06.872961 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-istio-data" pod "0e6063b9-04b2-42f5-ac6b-cea0dc65007a" (UID: "0e6063b9-04b2-42f5-ac6b-cea0dc65007a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:22:06.895165 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:06.873752 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-istio-token" pod "0e6063b9-04b2-42f5-ac6b-cea0dc65007a" (UID: "0e6063b9-04b2-42f5-ac6b-cea0dc65007a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:22:06.895165 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:06.873912 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-kube-api-access-xgqwd" pod "0e6063b9-04b2-42f5-ac6b-cea0dc65007a" (UID: "0e6063b9-04b2-42f5-ac6b-cea0dc65007a"). InnerVolumeSpecName "kube-api-access-xgqwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:22:06.928304 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:06.895548 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-istiod-ca-cert" pod "0e6063b9-04b2-42f5-ac6b-cea0dc65007a" (UID: "0e6063b9-04b2-42f5-ac6b-cea0dc65007a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:22:06.928304 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:06.920162 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-istio-envoy" pod "0e6063b9-04b2-42f5-ac6b-cea0dc65007a" (UID: "0e6063b9-04b2-42f5-ac6b-cea0dc65007a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:22:06.930750 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:06.930707 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-istio-podinfo" pod "0e6063b9-04b2-42f5-ac6b-cea0dc65007a" (UID: "0e6063b9-04b2-42f5-ac6b-cea0dc65007a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:22:06.972353 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:06.972323 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:06.972353 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:06.972351 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:06.972535 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:06.972364 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:06.972535 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:06.972373 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:06.972535 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:06.972382 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:06.972535 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:06.972390 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:06.972535 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:06.972400 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:06.972535 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:06.972411 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgqwd\" (UniqueName: \"kubernetes.io/projected/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-kube-api-access-xgqwd\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:06.972535 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:06.972423 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e6063b9-04b2-42f5-ac6b-cea0dc65007a-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:07.019161 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:07.019128 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" event={"ID":"0e6063b9-04b2-42f5-ac6b-cea0dc65007a","Type":"ContainerDied","Data":"76a1d3d52b46623aa7d39ff0aa07d325d0c237fd888aab0a88f3f7c8f57cc8b9"} Nov 01 16:22:07.019161 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:07.019156 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf" Nov 01 16:22:07.019676 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:07.019195 4363 scope.go:122] "RemoveContainer" containerID="1fbc16b99a880843de096ac7d24277b648d33dd2c94140dc3dbd165272de4977" Nov 01 16:22:07.103040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:07.084632 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf"] Nov 01 16:22:07.126106 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:07.126066 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-lb2mf"] Nov 01 16:22:07.382048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:07.381359 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Nov 01 16:22:07.382048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:07.381417 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" podUID="a850693a-07f7-4fdf-9ad3-4d08559c9737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Nov 01 16:22:08.503633 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:08.503100 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" Nov 01 16:22:08.504720 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:08.504209 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" Nov 01 16:22:08.512085 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:08.505224 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" Nov 01 16:22:08.801572 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:08.801494 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e6063b9-04b2-42f5-ac6b-cea0dc65007a" path="/var/lib/kubelet/pods/0e6063b9-04b2-42f5-ac6b-cea0dc65007a/volumes" Nov 01 16:22:09.529949 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:09.529916 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a4240a-5f8bf4f8dd-4fpd6"] Nov 01 16:22:09.530488 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:09.530163 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4240a-5f8bf4f8dd-4fpd6" podUID="9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c" containerName="maas-api" containerID="cri-o://10599dd59e2a6cff1983ba4b08d7492202c82e14221657e10559900129454bef" gracePeriod=30 Nov 01 16:22:10.126213 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:10.126155 4363 generic.go:356] "Generic (PLEG): container finished" podID="9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c" containerID="10599dd59e2a6cff1983ba4b08d7492202c82e14221657e10559900129454bef" exitCode=0 Nov 01 16:22:10.127451 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:10.127417 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4240a-5f8bf4f8dd-4fpd6" event={"ID":"9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c","Type":"ContainerDied","Data":"10599dd59e2a6cff1983ba4b08d7492202c82e14221657e10559900129454bef"} Nov 01 16:22:10.144789 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:10.144758 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4240a-5f8bf4f8dd-4fpd6" Nov 01 16:22:10.213343 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:10.213306 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c-metrics-certs\") pod \"9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c\" (UID: \"9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c\") " Nov 01 16:22:10.213559 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:10.213369 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c-kube-api-access-bllk9\" (UniqueName: \"kubernetes.io/projected/9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c-kube-api-access-bllk9\") pod \"9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c\" (UID: \"9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c\") " Nov 01 16:22:10.213559 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:10.213467 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c-maas-api-tls\") pod \"9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c\" (UID: \"9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c\") " Nov 01 16:22:10.213703 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:10.213640 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c-openshift-service-ca\") pod \"9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c\" (UID: \"9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c\") " Nov 01 16:22:10.215287 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:10.215257 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c-openshift-service-ca" pod "9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c" (UID: "9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:22:10.216891 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:10.216864 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c-metrics-certs" pod "9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c" (UID: "9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:22:10.217327 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:10.217304 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c-kube-api-access-bllk9" pod "9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c" (UID: "9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c"). InnerVolumeSpecName "kube-api-access-bllk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:22:10.218361 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:10.218342 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c-maas-api-tls" pod "9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c" (UID: "9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:22:10.321040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:10.320137 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a4240a-86f876f9cf-9bqvq" podUID="1964cd1c-696f-4f45-83ba-8dec81dbc624" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:22:10.326103 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:10.322362 4363 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c-metrics-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:10.326103 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:10.322396 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bllk9\" (UniqueName: \"kubernetes.io/projected/9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c-kube-api-access-bllk9\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:10.326103 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:10.322412 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c-maas-api-tls\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:10.326103 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:10.322425 4363 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c-openshift-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:10.326103 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:10.322558 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a4240a-745976f465-st69v" podUID="c8c062a2-3461-4bbe-ac11-81966a3cec16" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:22:11.135729 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:11.135698 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4240a-5f8bf4f8dd-4fpd6" Nov 01 16:22:11.135729 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:11.135709 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4240a-5f8bf4f8dd-4fpd6" event={"ID":"9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c","Type":"ContainerDied","Data":"89e6b8165b0ba1e523a6dfb8f13a0633fb921f2ad392fb5287f5fcbf17c05c54"} Nov 01 16:22:11.136369 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:11.135801 4363 scope.go:122] "RemoveContainer" containerID="10599dd59e2a6cff1983ba4b08d7492202c82e14221657e10559900129454bef" Nov 01 16:22:11.157551 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:11.157522 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a4240a-5f8bf4f8dd-4fpd6"] Nov 01 16:22:11.162051 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:11.161997 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a4240a-5f8bf4f8dd-4fpd6"] Nov 01 16:22:12.800557 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:12.800513 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c" path="/var/lib/kubelet/pods/9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c/volumes" Nov 01 16:22:16.867659 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:16.867629 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc"] Nov 01 16:22:16.868100 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:16.867908 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" podUID="23798655-3db1-4155-9f07-7c46e4fc7a14" containerName="istio-proxy" containerID="cri-o://a100941674dd5b6bdc0a4cc87c9dd372d7403e1afe33e84345f6c1b79f5acbdc" gracePeriod=30 Nov 01 16:22:17.252723 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:17.252638 4363 patch_prober.go:29] interesting pod/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:22:17.252954 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:17.252926 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" podUID="23798655-3db1-4155-9f07-7c46e4fc7a14" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:22:18.344493 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:18.344461 4363 generic.go:356] "Generic (PLEG): container finished" podID="c8c062a2-3461-4bbe-ac11-81966a3cec16" containerID="c9eabc2a9000378c0a532d6557de1af55e08161638f9cb28f7c4fed6f308a9ba" exitCode=0 Nov 01 16:22:18.345248 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:18.345219 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a4240a-745976f465-st69v" event={"ID":"c8c062a2-3461-4bbe-ac11-81966a3cec16","Type":"ContainerDied","Data":"c9eabc2a9000378c0a532d6557de1af55e08161638f9cb28f7c4fed6f308a9ba"} Nov 01 16:22:18.411576 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:22:18.411498 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:22:18.625067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:18.624992 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a4240a-745976f465-st69v" Nov 01 16:22:18.666000 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:18.665962 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a4240a-86f876f9cf-9bqvq" Nov 01 16:22:18.720225 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:18.720190 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1964cd1c-696f-4f45-83ba-8dec81dbc624-kube-api-access-7ftjc\" (UniqueName: \"kubernetes.io/projected/1964cd1c-696f-4f45-83ba-8dec81dbc624-kube-api-access-7ftjc\") pod \"1964cd1c-696f-4f45-83ba-8dec81dbc624\" (UID: \"1964cd1c-696f-4f45-83ba-8dec81dbc624\") " Nov 01 16:22:18.720421 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:18.720309 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8c062a2-3461-4bbe-ac11-81966a3cec16-kube-api-access-h4sfq\" (UniqueName: \"kubernetes.io/projected/c8c062a2-3461-4bbe-ac11-81966a3cec16-kube-api-access-h4sfq\") pod \"c8c062a2-3461-4bbe-ac11-81966a3cec16\" (UID: \"c8c062a2-3461-4bbe-ac11-81966a3cec16\") " Nov 01 16:22:18.720421 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:18.720395 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1964cd1c-696f-4f45-83ba-8dec81dbc624-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1964cd1c-696f-4f45-83ba-8dec81dbc624-plugins-config-volume\") pod \"1964cd1c-696f-4f45-83ba-8dec81dbc624\" (UID: \"1964cd1c-696f-4f45-83ba-8dec81dbc624\") " Nov 01 16:22:18.720635 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:18.720607 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8c062a2-3461-4bbe-ac11-81966a3cec16-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8c062a2-3461-4bbe-ac11-81966a3cec16-plugins-config-volume\") pod \"c8c062a2-3461-4bbe-ac11-81966a3cec16\" (UID: \"c8c062a2-3461-4bbe-ac11-81966a3cec16\") " Nov 01 16:22:18.729065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:18.729033 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8c062a2-3461-4bbe-ac11-81966a3cec16-plugins-config-volume" pod "c8c062a2-3461-4bbe-ac11-81966a3cec16" (UID: "c8c062a2-3461-4bbe-ac11-81966a3cec16"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:22:18.731839 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:18.730487 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1964cd1c-696f-4f45-83ba-8dec81dbc624-plugins-config-volume" pod "1964cd1c-696f-4f45-83ba-8dec81dbc624" (UID: "1964cd1c-696f-4f45-83ba-8dec81dbc624"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:22:18.733260 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:18.733236 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1964cd1c-696f-4f45-83ba-8dec81dbc624-kube-api-access-7ftjc" pod "1964cd1c-696f-4f45-83ba-8dec81dbc624" (UID: "1964cd1c-696f-4f45-83ba-8dec81dbc624"). InnerVolumeSpecName "kube-api-access-7ftjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:22:18.733370 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:18.733281 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8c062a2-3461-4bbe-ac11-81966a3cec16-kube-api-access-h4sfq" pod "c8c062a2-3461-4bbe-ac11-81966a3cec16" (UID: "c8c062a2-3461-4bbe-ac11-81966a3cec16"). InnerVolumeSpecName "kube-api-access-h4sfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:22:18.829564 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:18.829491 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8c062a2-3461-4bbe-ac11-81966a3cec16-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:18.829564 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:18.829521 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ftjc\" (UniqueName: \"kubernetes.io/projected/1964cd1c-696f-4f45-83ba-8dec81dbc624-kube-api-access-7ftjc\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:18.829564 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:18.829535 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4sfq\" (UniqueName: \"kubernetes.io/projected/c8c062a2-3461-4bbe-ac11-81966a3cec16-kube-api-access-h4sfq\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:18.829564 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:18.829547 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1964cd1c-696f-4f45-83ba-8dec81dbc624-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:19.354100 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.354064 4363 generic.go:356] "Generic (PLEG): container finished" podID="1964cd1c-696f-4f45-83ba-8dec81dbc624" containerID="f45baca75c75cadf94e4a74a6f91e53a1304d6f802562fe14b7a39dd9e9b89ba" exitCode=0 Nov 01 16:22:19.354596 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.354111 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a4240a-86f876f9cf-9bqvq" Nov 01 16:22:19.354596 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.354151 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a4240a-86f876f9cf-9bqvq" event={"ID":"1964cd1c-696f-4f45-83ba-8dec81dbc624","Type":"ContainerDied","Data":"f45baca75c75cadf94e4a74a6f91e53a1304d6f802562fe14b7a39dd9e9b89ba"} Nov 01 16:22:19.354596 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.354223 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a4240a-86f876f9cf-9bqvq" event={"ID":"1964cd1c-696f-4f45-83ba-8dec81dbc624","Type":"ContainerDied","Data":"69ded54c1b5da5010566cff659b28495d5ab0a72febad51350d7ee04e8d5e8e6"} Nov 01 16:22:19.354596 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.354247 4363 scope.go:122] "RemoveContainer" containerID="f45baca75c75cadf94e4a74a6f91e53a1304d6f802562fe14b7a39dd9e9b89ba" Nov 01 16:22:19.355946 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.355880 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a4240a-745976f465-st69v" Nov 01 16:22:19.355946 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.355924 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a4240a-745976f465-st69v" event={"ID":"c8c062a2-3461-4bbe-ac11-81966a3cec16","Type":"ContainerDied","Data":"a60b35030e1e00c63a411a9099bd566df633cfbcb23a61de9be2114388e4a81c"} Nov 01 16:22:19.365319 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.365303 4363 scope.go:122] "RemoveContainer" containerID="f45baca75c75cadf94e4a74a6f91e53a1304d6f802562fe14b7a39dd9e9b89ba" Nov 01 16:22:19.365571 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:22:19.365553 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f45baca75c75cadf94e4a74a6f91e53a1304d6f802562fe14b7a39dd9e9b89ba\": container with ID starting with f45baca75c75cadf94e4a74a6f91e53a1304d6f802562fe14b7a39dd9e9b89ba not found: ID does not exist" containerID="f45baca75c75cadf94e4a74a6f91e53a1304d6f802562fe14b7a39dd9e9b89ba" Nov 01 16:22:19.365656 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.365581 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f45baca75c75cadf94e4a74a6f91e53a1304d6f802562fe14b7a39dd9e9b89ba"} err="failed to get container status \"f45baca75c75cadf94e4a74a6f91e53a1304d6f802562fe14b7a39dd9e9b89ba\": rpc error: code = NotFound desc = could not find container \"f45baca75c75cadf94e4a74a6f91e53a1304d6f802562fe14b7a39dd9e9b89ba\": container with ID starting with f45baca75c75cadf94e4a74a6f91e53a1304d6f802562fe14b7a39dd9e9b89ba not found: ID does not exist" Nov 01 16:22:19.365656 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.365603 4363 scope.go:122] "RemoveContainer" containerID="c9eabc2a9000378c0a532d6557de1af55e08161638f9cb28f7c4fed6f308a9ba" Nov 01 16:22:19.388542 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.388517 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a4240a-86f876f9cf-9bqvq"] Nov 01 16:22:19.400850 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.400827 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a4240a-86f876f9cf-9bqvq"] Nov 01 16:22:19.424952 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.424925 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a4240a-745976f465-st69v"] Nov 01 16:22:19.434809 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.434770 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a4240a-745976f465-st69v"] Nov 01 16:22:19.655380 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.655299 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4"] Nov 01 16:22:19.656640 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.656604 4363 cpu_manager.go:414] "containerMap: removing container" podUID="1964cd1c-696f-4f45-83ba-8dec81dbc624" containerName="payload-pre-processing" Nov 01 16:22:19.656640 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.656639 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1964cd1c-696f-4f45-83ba-8dec81dbc624" containerName="payload-pre-processing" Nov 01 16:22:19.656795 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.656665 4363 cpu_manager.go:414] "containerMap: removing container" podUID="0e6063b9-04b2-42f5-ac6b-cea0dc65007a" containerName="istio-proxy" Nov 01 16:22:19.656795 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.656673 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e6063b9-04b2-42f5-ac6b-cea0dc65007a" containerName="istio-proxy" Nov 01 16:22:19.656795 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.656723 4363 cpu_manager.go:414] "containerMap: removing container" podUID="c8c062a2-3461-4bbe-ac11-81966a3cec16" containerName="payload-processing" Nov 01 16:22:19.656795 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.656732 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8c062a2-3461-4bbe-ac11-81966a3cec16" containerName="payload-processing" Nov 01 16:22:19.656795 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.656747 4363 cpu_manager.go:414] "containerMap: removing container" podUID="9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c" containerName="maas-api" Nov 01 16:22:19.656795 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.656755 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c" containerName="maas-api" Nov 01 16:22:19.656992 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.656940 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8c062a2-3461-4bbe-ac11-81966a3cec16" containerName="payload-processing" Nov 01 16:22:19.656992 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.656956 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dc8ab58-1f70-4ca9-a3df-bde7e6b7a61c" containerName="maas-api" Nov 01 16:22:19.656992 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.656978 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="1964cd1c-696f-4f45-83ba-8dec81dbc624" containerName="payload-pre-processing" Nov 01 16:22:19.656992 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.656988 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e6063b9-04b2-42f5-ac6b-cea0dc65007a" containerName="istio-proxy" Nov 01 16:22:19.664435 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.664270 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" Nov 01 16:22:19.668614 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.668587 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-c8537c-openshift-default-dockercfg-lhz4c\"" Nov 01 16:22:19.671150 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.671128 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4"] Nov 01 16:22:19.779047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.775350 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f58c340-e931-42d5-9322-2f60420216fb-istio-envoy\") pod \"e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4\" (UID: \"5f58c340-e931-42d5-9322-2f60420216fb\") " pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" Nov 01 16:22:19.779047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.775422 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f58c340-e931-42d5-9322-2f60420216fb-istio-podinfo\") pod \"e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4\" (UID: \"5f58c340-e931-42d5-9322-2f60420216fb\") " pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" Nov 01 16:22:19.779047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.775460 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f58c340-e931-42d5-9322-2f60420216fb-workload-certs\") pod \"e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4\" (UID: \"5f58c340-e931-42d5-9322-2f60420216fb\") " pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" Nov 01 16:22:19.779047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.775511 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f58c340-e931-42d5-9322-2f60420216fb-credential-socket\") pod \"e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4\" (UID: \"5f58c340-e931-42d5-9322-2f60420216fb\") " pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" Nov 01 16:22:19.779047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.775542 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j944s\" (UniqueName: \"kubernetes.io/projected/5f58c340-e931-42d5-9322-2f60420216fb-kube-api-access-j944s\") pod \"e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4\" (UID: \"5f58c340-e931-42d5-9322-2f60420216fb\") " pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" Nov 01 16:22:19.779047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.775582 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f58c340-e931-42d5-9322-2f60420216fb-workload-socket\") pod \"e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4\" (UID: \"5f58c340-e931-42d5-9322-2f60420216fb\") " pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" Nov 01 16:22:19.779047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.775740 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f58c340-e931-42d5-9322-2f60420216fb-istio-data\") pod \"e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4\" (UID: \"5f58c340-e931-42d5-9322-2f60420216fb\") " pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" Nov 01 16:22:19.779047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.775851 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f58c340-e931-42d5-9322-2f60420216fb-istiod-ca-cert\") pod \"e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4\" (UID: \"5f58c340-e931-42d5-9322-2f60420216fb\") " pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" Nov 01 16:22:19.779047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.775896 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f58c340-e931-42d5-9322-2f60420216fb-istio-token\") pod \"e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4\" (UID: \"5f58c340-e931-42d5-9322-2f60420216fb\") " pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" Nov 01 16:22:19.920089 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.919634 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f58c340-e931-42d5-9322-2f60420216fb-istio-envoy\") pod \"e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4\" (UID: \"5f58c340-e931-42d5-9322-2f60420216fb\") " pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" Nov 01 16:22:19.920089 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.919702 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f58c340-e931-42d5-9322-2f60420216fb-istio-podinfo\") pod \"e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4\" (UID: \"5f58c340-e931-42d5-9322-2f60420216fb\") " pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" Nov 01 16:22:19.920089 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.919740 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f58c340-e931-42d5-9322-2f60420216fb-workload-certs\") pod \"e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4\" (UID: \"5f58c340-e931-42d5-9322-2f60420216fb\") " pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" Nov 01 16:22:19.920089 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.919796 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f58c340-e931-42d5-9322-2f60420216fb-credential-socket\") pod \"e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4\" (UID: \"5f58c340-e931-42d5-9322-2f60420216fb\") " pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" Nov 01 16:22:19.920089 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.919827 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j944s\" (UniqueName: \"kubernetes.io/projected/5f58c340-e931-42d5-9322-2f60420216fb-kube-api-access-j944s\") pod \"e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4\" (UID: \"5f58c340-e931-42d5-9322-2f60420216fb\") " pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" Nov 01 16:22:19.920089 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.919867 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f58c340-e931-42d5-9322-2f60420216fb-workload-socket\") pod \"e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4\" (UID: \"5f58c340-e931-42d5-9322-2f60420216fb\") " pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" Nov 01 16:22:19.920089 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.920055 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f58c340-e931-42d5-9322-2f60420216fb-istio-data\") pod \"e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4\" (UID: \"5f58c340-e931-42d5-9322-2f60420216fb\") " pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" Nov 01 16:22:19.920589 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.920166 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f58c340-e931-42d5-9322-2f60420216fb-istiod-ca-cert\") pod \"e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4\" (UID: \"5f58c340-e931-42d5-9322-2f60420216fb\") " pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" Nov 01 16:22:19.920589 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.920216 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f58c340-e931-42d5-9322-2f60420216fb-istio-token\") pod \"e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4\" (UID: \"5f58c340-e931-42d5-9322-2f60420216fb\") " pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" Nov 01 16:22:19.945037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.920697 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f58c340-e931-42d5-9322-2f60420216fb-credential-socket\") pod \"e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4\" (UID: \"5f58c340-e931-42d5-9322-2f60420216fb\") " pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" Nov 01 16:22:19.945037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.939461 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f58c340-e931-42d5-9322-2f60420216fb-istio-data\") pod \"e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4\" (UID: \"5f58c340-e931-42d5-9322-2f60420216fb\") " pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" Nov 01 16:22:19.945037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.940366 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f58c340-e931-42d5-9322-2f60420216fb-workload-certs\") pod \"e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4\" (UID: \"5f58c340-e931-42d5-9322-2f60420216fb\") " pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" Nov 01 16:22:19.945037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.942210 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f58c340-e931-42d5-9322-2f60420216fb-istiod-ca-cert\") pod \"e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4\" (UID: \"5f58c340-e931-42d5-9322-2f60420216fb\") " pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" Nov 01 16:22:20.003042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.970791 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f58c340-e931-42d5-9322-2f60420216fb-istio-envoy\") pod \"e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4\" (UID: \"5f58c340-e931-42d5-9322-2f60420216fb\") " pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" Nov 01 16:22:20.003042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.981619 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f58c340-e931-42d5-9322-2f60420216fb-istio-podinfo\") pod \"e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4\" (UID: \"5f58c340-e931-42d5-9322-2f60420216fb\") " pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" Nov 01 16:22:20.003042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.981832 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f58c340-e931-42d5-9322-2f60420216fb-workload-socket\") pod \"e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4\" (UID: \"5f58c340-e931-42d5-9322-2f60420216fb\") " pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" Nov 01 16:22:20.003042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.987500 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j944s\" (UniqueName: \"kubernetes.io/projected/5f58c340-e931-42d5-9322-2f60420216fb-kube-api-access-j944s\") pod \"e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4\" (UID: \"5f58c340-e931-42d5-9322-2f60420216fb\") " pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" Nov 01 16:22:20.003042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:19.993185 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f58c340-e931-42d5-9322-2f60420216fb-istio-token\") pod \"e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4\" (UID: \"5f58c340-e931-42d5-9322-2f60420216fb\") " pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" Nov 01 16:22:20.287462 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:20.287366 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" Nov 01 16:22:20.577044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:20.577003 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4"] Nov 01 16:22:20.578620 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:22:20.578590 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f58c340_e931_42d5_9322_2f60420216fb.slice/crio-6633df2e2ab7d37c373d5f76fbf4e0f615a7557ef4b9eb2a32769af0e58310b1 WatchSource:0}: Error finding container 6633df2e2ab7d37c373d5f76fbf4e0f615a7557ef4b9eb2a32769af0e58310b1: Status 404 returned error can't find the container with id 6633df2e2ab7d37c373d5f76fbf4e0f615a7557ef4b9eb2a32769af0e58310b1 Nov 01 16:22:20.582206 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:20.582167 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:22:20.582312 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:20.582255 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:22:20.582312 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:20.582296 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:22:20.822122 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:20.822070 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1964cd1c-696f-4f45-83ba-8dec81dbc624" path="/var/lib/kubelet/pods/1964cd1c-696f-4f45-83ba-8dec81dbc624/volumes" Nov 01 16:22:20.823188 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:20.823146 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8c062a2-3461-4bbe-ac11-81966a3cec16" path="/var/lib/kubelet/pods/c8c062a2-3461-4bbe-ac11-81966a3cec16/volumes" Nov 01 16:22:21.380961 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:21.380923 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" event={"ID":"5f58c340-e931-42d5-9322-2f60420216fb","Type":"ContainerStarted","Data":"87a8952f9e0890633201b4cb35a1a240e85574976da72c9b8522e064ff91ab4e"} Nov 01 16:22:21.381218 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:21.380985 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" event={"ID":"5f58c340-e931-42d5-9322-2f60420216fb","Type":"ContainerStarted","Data":"6633df2e2ab7d37c373d5f76fbf4e0f615a7557ef4b9eb2a32769af0e58310b1"} Nov 01 16:22:21.405594 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:21.405532 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" podStartSLOduration=2.40551379 podStartE2EDuration="2.40551379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:22:21.401930803 +0000 UTC m=+2074.733668456" watchObservedRunningTime="2026-11-01 16:22:21.40551379 +0000 UTC m=+2074.737251439" Nov 01 16:22:22.179372 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.179348 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" Nov 01 16:22:22.288446 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.288416 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" Nov 01 16:22:22.294615 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.294583 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23798655-3db1-4155-9f07-7c46e4fc7a14-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23798655-3db1-4155-9f07-7c46e4fc7a14-workload-socket\") pod \"23798655-3db1-4155-9f07-7c46e4fc7a14\" (UID: \"23798655-3db1-4155-9f07-7c46e4fc7a14\") " Nov 01 16:22:22.294786 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.294680 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23798655-3db1-4155-9f07-7c46e4fc7a14-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23798655-3db1-4155-9f07-7c46e4fc7a14-istiod-ca-cert\") pod \"23798655-3db1-4155-9f07-7c46e4fc7a14\" (UID: \"23798655-3db1-4155-9f07-7c46e4fc7a14\") " Nov 01 16:22:22.294786 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.294707 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23798655-3db1-4155-9f07-7c46e4fc7a14-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23798655-3db1-4155-9f07-7c46e4fc7a14-istio-envoy\") pod \"23798655-3db1-4155-9f07-7c46e4fc7a14\" (UID: \"23798655-3db1-4155-9f07-7c46e4fc7a14\") " Nov 01 16:22:22.294786 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.294743 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23798655-3db1-4155-9f07-7c46e4fc7a14-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23798655-3db1-4155-9f07-7c46e4fc7a14-workload-certs\") pod \"23798655-3db1-4155-9f07-7c46e4fc7a14\" (UID: \"23798655-3db1-4155-9f07-7c46e4fc7a14\") " Nov 01 16:22:22.294959 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.294890 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23798655-3db1-4155-9f07-7c46e4fc7a14-workload-socket" pod "23798655-3db1-4155-9f07-7c46e4fc7a14" (UID: "23798655-3db1-4155-9f07-7c46e4fc7a14"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:22:22.294959 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.294912 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23798655-3db1-4155-9f07-7c46e4fc7a14-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23798655-3db1-4155-9f07-7c46e4fc7a14-credential-socket\") pod \"23798655-3db1-4155-9f07-7c46e4fc7a14\" (UID: \"23798655-3db1-4155-9f07-7c46e4fc7a14\") " Nov 01 16:22:22.294959 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.294946 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23798655-3db1-4155-9f07-7c46e4fc7a14-kube-api-access-zkpzf\" (UniqueName: \"kubernetes.io/projected/23798655-3db1-4155-9f07-7c46e4fc7a14-kube-api-access-zkpzf\") pod \"23798655-3db1-4155-9f07-7c46e4fc7a14\" (UID: \"23798655-3db1-4155-9f07-7c46e4fc7a14\") " Nov 01 16:22:22.295156 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.294976 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23798655-3db1-4155-9f07-7c46e4fc7a14-istio-token\" (UniqueName: \"kubernetes.io/projected/23798655-3db1-4155-9f07-7c46e4fc7a14-istio-token\") pod \"23798655-3db1-4155-9f07-7c46e4fc7a14\" (UID: \"23798655-3db1-4155-9f07-7c46e4fc7a14\") " Nov 01 16:22:22.295156 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.295081 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23798655-3db1-4155-9f07-7c46e4fc7a14-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23798655-3db1-4155-9f07-7c46e4fc7a14-istio-podinfo\") pod \"23798655-3db1-4155-9f07-7c46e4fc7a14\" (UID: \"23798655-3db1-4155-9f07-7c46e4fc7a14\") " Nov 01 16:22:22.295156 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.295109 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23798655-3db1-4155-9f07-7c46e4fc7a14-workload-certs" pod "23798655-3db1-4155-9f07-7c46e4fc7a14" (UID: "23798655-3db1-4155-9f07-7c46e4fc7a14"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:22:22.295321 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.295185 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23798655-3db1-4155-9f07-7c46e4fc7a14-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23798655-3db1-4155-9f07-7c46e4fc7a14-istio-data\") pod \"23798655-3db1-4155-9f07-7c46e4fc7a14\" (UID: \"23798655-3db1-4155-9f07-7c46e4fc7a14\") " Nov 01 16:22:22.295321 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.295277 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23798655-3db1-4155-9f07-7c46e4fc7a14-istiod-ca-cert" pod "23798655-3db1-4155-9f07-7c46e4fc7a14" (UID: "23798655-3db1-4155-9f07-7c46e4fc7a14"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:22:22.296303 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.295526 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23798655-3db1-4155-9f07-7c46e4fc7a14-credential-socket" pod "23798655-3db1-4155-9f07-7c46e4fc7a14" (UID: "23798655-3db1-4155-9f07-7c46e4fc7a14"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:22:22.296303 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.295781 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" Nov 01 16:22:22.296303 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.295841 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23798655-3db1-4155-9f07-7c46e4fc7a14-istio-data" pod "23798655-3db1-4155-9f07-7c46e4fc7a14" (UID: "23798655-3db1-4155-9f07-7c46e4fc7a14"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:22:22.296303 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.296086 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23798655-3db1-4155-9f07-7c46e4fc7a14-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:22.296303 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.296107 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23798655-3db1-4155-9f07-7c46e4fc7a14-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:22.296303 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.296119 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23798655-3db1-4155-9f07-7c46e4fc7a14-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:22.296303 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.296128 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23798655-3db1-4155-9f07-7c46e4fc7a14-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:22.296303 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.296137 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23798655-3db1-4155-9f07-7c46e4fc7a14-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:22.297739 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.297716 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23798655-3db1-4155-9f07-7c46e4fc7a14-istio-podinfo" pod "23798655-3db1-4155-9f07-7c46e4fc7a14" (UID: "23798655-3db1-4155-9f07-7c46e4fc7a14"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:22:22.297864 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.297752 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23798655-3db1-4155-9f07-7c46e4fc7a14-istio-token" pod "23798655-3db1-4155-9f07-7c46e4fc7a14" (UID: "23798655-3db1-4155-9f07-7c46e4fc7a14"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:22:22.298239 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.298208 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23798655-3db1-4155-9f07-7c46e4fc7a14-istio-envoy" pod "23798655-3db1-4155-9f07-7c46e4fc7a14" (UID: "23798655-3db1-4155-9f07-7c46e4fc7a14"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:22:22.298431 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.298398 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23798655-3db1-4155-9f07-7c46e4fc7a14-kube-api-access-zkpzf" pod "23798655-3db1-4155-9f07-7c46e4fc7a14" (UID: "23798655-3db1-4155-9f07-7c46e4fc7a14"). InnerVolumeSpecName "kube-api-access-zkpzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:22:22.392968 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.392933 4363 generic.go:356] "Generic (PLEG): container finished" podID="23798655-3db1-4155-9f07-7c46e4fc7a14" containerID="a100941674dd5b6bdc0a4cc87c9dd372d7403e1afe33e84345f6c1b79f5acbdc" exitCode=0 Nov 01 16:22:22.393151 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.392995 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" Nov 01 16:22:22.393151 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.393024 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" event={"ID":"23798655-3db1-4155-9f07-7c46e4fc7a14","Type":"ContainerDied","Data":"a100941674dd5b6bdc0a4cc87c9dd372d7403e1afe33e84345f6c1b79f5acbdc"} Nov 01 16:22:22.393151 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.393078 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc" event={"ID":"23798655-3db1-4155-9f07-7c46e4fc7a14","Type":"ContainerDied","Data":"aa8782129b74ab5d5ffc1bf2788f7a144aa03b4c80f3572d964528183ebc1d4f"} Nov 01 16:22:22.393151 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.393105 4363 scope.go:122] "RemoveContainer" containerID="a100941674dd5b6bdc0a4cc87c9dd372d7403e1afe33e84345f6c1b79f5acbdc" Nov 01 16:22:22.393648 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.393565 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" Nov 01 16:22:22.394552 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.394533 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" Nov 01 16:22:22.397545 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.397520 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23798655-3db1-4155-9f07-7c46e4fc7a14-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:22.397645 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.397551 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23798655-3db1-4155-9f07-7c46e4fc7a14-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:22.397645 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.397570 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkpzf\" (UniqueName: \"kubernetes.io/projected/23798655-3db1-4155-9f07-7c46e4fc7a14-kube-api-access-zkpzf\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:22.397645 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.397585 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23798655-3db1-4155-9f07-7c46e4fc7a14-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:22.406153 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.406133 4363 scope.go:122] "RemoveContainer" containerID="a0c6a86522b2aa01f5b55176dbc467ba847f72036ff9ec5cacac183d6d7faef8" Nov 01 16:22:22.431269 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.420713 4363 scope.go:122] "RemoveContainer" containerID="a100941674dd5b6bdc0a4cc87c9dd372d7403e1afe33e84345f6c1b79f5acbdc" Nov 01 16:22:22.431269 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:22:22.421561 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a100941674dd5b6bdc0a4cc87c9dd372d7403e1afe33e84345f6c1b79f5acbdc\": container with ID starting with a100941674dd5b6bdc0a4cc87c9dd372d7403e1afe33e84345f6c1b79f5acbdc not found: ID does not exist" containerID="a100941674dd5b6bdc0a4cc87c9dd372d7403e1afe33e84345f6c1b79f5acbdc" Nov 01 16:22:22.431269 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.421598 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a100941674dd5b6bdc0a4cc87c9dd372d7403e1afe33e84345f6c1b79f5acbdc"} err="failed to get container status \"a100941674dd5b6bdc0a4cc87c9dd372d7403e1afe33e84345f6c1b79f5acbdc\": rpc error: code = NotFound desc = could not find container \"a100941674dd5b6bdc0a4cc87c9dd372d7403e1afe33e84345f6c1b79f5acbdc\": container with ID starting with a100941674dd5b6bdc0a4cc87c9dd372d7403e1afe33e84345f6c1b79f5acbdc not found: ID does not exist" Nov 01 16:22:22.431269 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.421623 4363 scope.go:122] "RemoveContainer" containerID="a0c6a86522b2aa01f5b55176dbc467ba847f72036ff9ec5cacac183d6d7faef8" Nov 01 16:22:22.431269 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:22:22.421877 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0c6a86522b2aa01f5b55176dbc467ba847f72036ff9ec5cacac183d6d7faef8\": container with ID starting with a0c6a86522b2aa01f5b55176dbc467ba847f72036ff9ec5cacac183d6d7faef8 not found: ID does not exist" containerID="a0c6a86522b2aa01f5b55176dbc467ba847f72036ff9ec5cacac183d6d7faef8" Nov 01 16:22:22.431269 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.421903 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0c6a86522b2aa01f5b55176dbc467ba847f72036ff9ec5cacac183d6d7faef8"} err="failed to get container status \"a0c6a86522b2aa01f5b55176dbc467ba847f72036ff9ec5cacac183d6d7faef8\": rpc error: code = NotFound desc = could not find container \"a0c6a86522b2aa01f5b55176dbc467ba847f72036ff9ec5cacac183d6d7faef8\": container with ID starting with a0c6a86522b2aa01f5b55176dbc467ba847f72036ff9ec5cacac183d6d7faef8 not found: ID does not exist" Nov 01 16:22:22.542830 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.533297 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc"] Nov 01 16:22:22.553275 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.553241 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-zmwdc"] Nov 01 16:22:22.801721 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:22.801627 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23798655-3db1-4155-9f07-7c46e4fc7a14" path="/var/lib/kubelet/pods/23798655-3db1-4155-9f07-7c46e4fc7a14/volumes" Nov 01 16:22:33.474818 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:33.474781 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf"] Nov 01 16:22:33.475353 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:33.475043 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf" podUID="958066a5-3a94-4ed1-9801-9c011b5cf61c" containerName="maas-api" containerID="cri-o://a7de2c32a8369e1508612cfa0e4921e4b431f884999333812ae36514b277f9ac" gracePeriod=30 Nov 01 16:22:34.070055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:34.063267 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf" Nov 01 16:22:34.084920 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:34.075505 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/958066a5-3a94-4ed1-9801-9c011b5cf61c-kube-api-access-shjdk\" (UniqueName: \"kubernetes.io/projected/958066a5-3a94-4ed1-9801-9c011b5cf61c-kube-api-access-shjdk\") pod \"958066a5-3a94-4ed1-9801-9c011b5cf61c\" (UID: \"958066a5-3a94-4ed1-9801-9c011b5cf61c\") " Nov 01 16:22:34.084920 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:34.075603 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/958066a5-3a94-4ed1-9801-9c011b5cf61c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/958066a5-3a94-4ed1-9801-9c011b5cf61c-metrics-certs\") pod \"958066a5-3a94-4ed1-9801-9c011b5cf61c\" (UID: \"958066a5-3a94-4ed1-9801-9c011b5cf61c\") " Nov 01 16:22:34.084920 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:34.075678 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/958066a5-3a94-4ed1-9801-9c011b5cf61c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/958066a5-3a94-4ed1-9801-9c011b5cf61c-openshift-service-ca\") pod \"958066a5-3a94-4ed1-9801-9c011b5cf61c\" (UID: \"958066a5-3a94-4ed1-9801-9c011b5cf61c\") " Nov 01 16:22:34.084920 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:34.075780 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/958066a5-3a94-4ed1-9801-9c011b5cf61c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/958066a5-3a94-4ed1-9801-9c011b5cf61c-maas-api-tls\") pod \"958066a5-3a94-4ed1-9801-9c011b5cf61c\" (UID: \"958066a5-3a94-4ed1-9801-9c011b5cf61c\") " Nov 01 16:22:34.084920 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:34.076563 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/958066a5-3a94-4ed1-9801-9c011b5cf61c-openshift-service-ca" pod "958066a5-3a94-4ed1-9801-9c011b5cf61c" (UID: "958066a5-3a94-4ed1-9801-9c011b5cf61c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:22:34.084920 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:34.076817 4363 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/958066a5-3a94-4ed1-9801-9c011b5cf61c-openshift-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:34.176735 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:34.175425 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/958066a5-3a94-4ed1-9801-9c011b5cf61c-maas-api-tls" pod "958066a5-3a94-4ed1-9801-9c011b5cf61c" (UID: "958066a5-3a94-4ed1-9801-9c011b5cf61c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:22:34.178581 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:34.178549 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/958066a5-3a94-4ed1-9801-9c011b5cf61c-metrics-certs" pod "958066a5-3a94-4ed1-9801-9c011b5cf61c" (UID: "958066a5-3a94-4ed1-9801-9c011b5cf61c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:22:34.178725 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:34.178590 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/958066a5-3a94-4ed1-9801-9c011b5cf61c-maas-api-tls\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:34.184399 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:34.184368 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/958066a5-3a94-4ed1-9801-9c011b5cf61c-kube-api-access-shjdk" pod "958066a5-3a94-4ed1-9801-9c011b5cf61c" (UID: "958066a5-3a94-4ed1-9801-9c011b5cf61c"). InnerVolumeSpecName "kube-api-access-shjdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:22:34.280552 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:34.280476 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shjdk\" (UniqueName: \"kubernetes.io/projected/958066a5-3a94-4ed1-9801-9c011b5cf61c-kube-api-access-shjdk\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:34.280552 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:34.280503 4363 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/958066a5-3a94-4ed1-9801-9c011b5cf61c-metrics-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:34.539402 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:34.539317 4363 generic.go:356] "Generic (PLEG): container finished" podID="958066a5-3a94-4ed1-9801-9c011b5cf61c" containerID="a7de2c32a8369e1508612cfa0e4921e4b431f884999333812ae36514b277f9ac" exitCode=0 Nov 01 16:22:34.539402 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:34.539367 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf" Nov 01 16:22:34.539869 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:34.539405 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf" event={"ID":"958066a5-3a94-4ed1-9801-9c011b5cf61c","Type":"ContainerDied","Data":"a7de2c32a8369e1508612cfa0e4921e4b431f884999333812ae36514b277f9ac"} Nov 01 16:22:34.539869 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:34.539470 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf" event={"ID":"958066a5-3a94-4ed1-9801-9c011b5cf61c","Type":"ContainerDied","Data":"756b4ca1bd25ea5669247988401f2428bda4f25a28108452348159a7409e2016"} Nov 01 16:22:34.539869 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:34.539505 4363 scope.go:122] "RemoveContainer" containerID="a7de2c32a8369e1508612cfa0e4921e4b431f884999333812ae36514b277f9ac" Nov 01 16:22:34.550135 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:34.550116 4363 scope.go:122] "RemoveContainer" containerID="a7de2c32a8369e1508612cfa0e4921e4b431f884999333812ae36514b277f9ac" Nov 01 16:22:34.550738 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:22:34.550706 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7de2c32a8369e1508612cfa0e4921e4b431f884999333812ae36514b277f9ac\": container with ID starting with a7de2c32a8369e1508612cfa0e4921e4b431f884999333812ae36514b277f9ac not found: ID does not exist" containerID="a7de2c32a8369e1508612cfa0e4921e4b431f884999333812ae36514b277f9ac" Nov 01 16:22:34.550840 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:34.550745 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7de2c32a8369e1508612cfa0e4921e4b431f884999333812ae36514b277f9ac"} err="failed to get container status \"a7de2c32a8369e1508612cfa0e4921e4b431f884999333812ae36514b277f9ac\": rpc error: code = NotFound desc = could not find container \"a7de2c32a8369e1508612cfa0e4921e4b431f884999333812ae36514b277f9ac\": container with ID starting with a7de2c32a8369e1508612cfa0e4921e4b431f884999333812ae36514b277f9ac not found: ID does not exist" Nov 01 16:22:34.564649 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:34.564617 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf"] Nov 01 16:22:34.568920 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:34.568900 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0c091d-7d7b755c88-xzbpf"] Nov 01 16:22:34.934040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:34.928184 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="958066a5-3a94-4ed1-9801-9c011b5cf61c" path="/var/lib/kubelet/pods/958066a5-3a94-4ed1-9801-9c011b5cf61c/volumes" Nov 01 16:22:38.435176 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:38.435127 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w5-383fde/auth-test-model-383fde-kserve-57c55ccb8f-rqp6x" podUID="36f441c7-2672-4581-9a0e-7142326e172b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.209:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:22:38.672731 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:38.671125 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:22:38.695186 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:38.695093 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:22:38.831496 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:38.831442 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-7225e5/auth-test-model-7225e5-kserve-84f65d66df-jd8qc" podUID="901ec925-24b0-4f15-a356-b7a5da163158" containerName="main" probeResult="failure" output="Get \"https://10.134.0.215:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:22:38.952096 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:38.949959 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:22:46.788440 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:46.788387 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn"] Nov 01 16:22:46.830762 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:46.830729 4363 cpu_manager.go:414] "containerMap: removing container" podUID="23798655-3db1-4155-9f07-7c46e4fc7a14" containerName="istio-proxy" Nov 01 16:22:46.830762 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:46.830760 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23798655-3db1-4155-9f07-7c46e4fc7a14" containerName="istio-proxy" Nov 01 16:22:46.830961 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:46.830775 4363 cpu_manager.go:414] "containerMap: removing container" podUID="958066a5-3a94-4ed1-9801-9c011b5cf61c" containerName="maas-api" Nov 01 16:22:46.830961 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:46.830783 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="958066a5-3a94-4ed1-9801-9c011b5cf61c" containerName="maas-api" Nov 01 16:22:46.830961 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:46.830799 4363 cpu_manager.go:414] "containerMap: removing container" podUID="23798655-3db1-4155-9f07-7c46e4fc7a14" containerName="istio-proxy" Nov 01 16:22:46.830961 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:46.830807 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23798655-3db1-4155-9f07-7c46e4fc7a14" containerName="istio-proxy" Nov 01 16:22:46.831165 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:46.831145 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="23798655-3db1-4155-9f07-7c46e4fc7a14" containerName="istio-proxy" Nov 01 16:22:46.831206 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:46.831172 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="958066a5-3a94-4ed1-9801-9c011b5cf61c" containerName="maas-api" Nov 01 16:22:46.831206 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:46.831186 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="23798655-3db1-4155-9f07-7c46e4fc7a14" containerName="istio-proxy" Nov 01 16:22:46.837074 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:46.836252 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" Nov 01 16:22:46.887090 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:46.877420 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn"] Nov 01 16:22:46.913528 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:46.913443 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8744fda2-3d14-48d8-8426-323166cca7a3-istio-data\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn\" (UID: \"8744fda2-3d14-48d8-8426-323166cca7a3\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" Nov 01 16:22:46.913528 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:46.913503 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqkwv\" (UniqueName: \"kubernetes.io/projected/8744fda2-3d14-48d8-8426-323166cca7a3-kube-api-access-cqkwv\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn\" (UID: \"8744fda2-3d14-48d8-8426-323166cca7a3\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" Nov 01 16:22:46.913874 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:46.913849 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8744fda2-3d14-48d8-8426-323166cca7a3-istio-envoy\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn\" (UID: \"8744fda2-3d14-48d8-8426-323166cca7a3\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" Nov 01 16:22:46.913941 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:46.913910 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8744fda2-3d14-48d8-8426-323166cca7a3-istiod-ca-cert\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn\" (UID: \"8744fda2-3d14-48d8-8426-323166cca7a3\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" Nov 01 16:22:46.913941 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:46.913934 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8744fda2-3d14-48d8-8426-323166cca7a3-istio-token\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn\" (UID: \"8744fda2-3d14-48d8-8426-323166cca7a3\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" Nov 01 16:22:46.914048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:46.913973 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8744fda2-3d14-48d8-8426-323166cca7a3-istio-podinfo\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn\" (UID: \"8744fda2-3d14-48d8-8426-323166cca7a3\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" Nov 01 16:22:46.914048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:46.914028 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8744fda2-3d14-48d8-8426-323166cca7a3-workload-socket\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn\" (UID: \"8744fda2-3d14-48d8-8426-323166cca7a3\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" Nov 01 16:22:46.914180 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:46.914104 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8744fda2-3d14-48d8-8426-323166cca7a3-credential-socket\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn\" (UID: \"8744fda2-3d14-48d8-8426-323166cca7a3\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" Nov 01 16:22:46.914180 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:46.914128 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8744fda2-3d14-48d8-8426-323166cca7a3-workload-certs\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn\" (UID: \"8744fda2-3d14-48d8-8426-323166cca7a3\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" Nov 01 16:22:47.016398 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:47.016360 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8744fda2-3d14-48d8-8426-323166cca7a3-istio-data\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn\" (UID: \"8744fda2-3d14-48d8-8426-323166cca7a3\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" Nov 01 16:22:47.016616 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:47.016426 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqkwv\" (UniqueName: \"kubernetes.io/projected/8744fda2-3d14-48d8-8426-323166cca7a3-kube-api-access-cqkwv\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn\" (UID: \"8744fda2-3d14-48d8-8426-323166cca7a3\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" Nov 01 16:22:47.016616 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:47.016461 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8744fda2-3d14-48d8-8426-323166cca7a3-istio-envoy\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn\" (UID: \"8744fda2-3d14-48d8-8426-323166cca7a3\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" Nov 01 16:22:47.016616 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:47.016508 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8744fda2-3d14-48d8-8426-323166cca7a3-istiod-ca-cert\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn\" (UID: \"8744fda2-3d14-48d8-8426-323166cca7a3\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" Nov 01 16:22:47.016616 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:47.016540 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8744fda2-3d14-48d8-8426-323166cca7a3-istio-token\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn\" (UID: \"8744fda2-3d14-48d8-8426-323166cca7a3\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" Nov 01 16:22:47.016616 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:47.016589 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8744fda2-3d14-48d8-8426-323166cca7a3-istio-podinfo\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn\" (UID: \"8744fda2-3d14-48d8-8426-323166cca7a3\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" Nov 01 16:22:47.016923 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:47.016641 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8744fda2-3d14-48d8-8426-323166cca7a3-workload-socket\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn\" (UID: \"8744fda2-3d14-48d8-8426-323166cca7a3\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" Nov 01 16:22:47.016923 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:47.016727 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8744fda2-3d14-48d8-8426-323166cca7a3-credential-socket\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn\" (UID: \"8744fda2-3d14-48d8-8426-323166cca7a3\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" Nov 01 16:22:47.016923 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:47.016759 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8744fda2-3d14-48d8-8426-323166cca7a3-workload-certs\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn\" (UID: \"8744fda2-3d14-48d8-8426-323166cca7a3\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" Nov 01 16:22:47.016923 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:47.016851 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8744fda2-3d14-48d8-8426-323166cca7a3-istio-data\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn\" (UID: \"8744fda2-3d14-48d8-8426-323166cca7a3\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" Nov 01 16:22:47.017413 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:47.017375 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8744fda2-3d14-48d8-8426-323166cca7a3-workload-socket\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn\" (UID: \"8744fda2-3d14-48d8-8426-323166cca7a3\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" Nov 01 16:22:47.017594 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:47.017561 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8744fda2-3d14-48d8-8426-323166cca7a3-credential-socket\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn\" (UID: \"8744fda2-3d14-48d8-8426-323166cca7a3\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" Nov 01 16:22:47.017965 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:47.017922 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8744fda2-3d14-48d8-8426-323166cca7a3-workload-certs\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn\" (UID: \"8744fda2-3d14-48d8-8426-323166cca7a3\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" Nov 01 16:22:47.017965 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:47.017935 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8744fda2-3d14-48d8-8426-323166cca7a3-istiod-ca-cert\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn\" (UID: \"8744fda2-3d14-48d8-8426-323166cca7a3\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" Nov 01 16:22:47.019259 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:47.019222 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8744fda2-3d14-48d8-8426-323166cca7a3-istio-envoy\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn\" (UID: \"8744fda2-3d14-48d8-8426-323166cca7a3\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" Nov 01 16:22:47.020151 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:47.020123 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8744fda2-3d14-48d8-8426-323166cca7a3-istio-podinfo\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn\" (UID: \"8744fda2-3d14-48d8-8426-323166cca7a3\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" Nov 01 16:22:47.025698 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:47.025677 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8744fda2-3d14-48d8-8426-323166cca7a3-istio-token\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn\" (UID: \"8744fda2-3d14-48d8-8426-323166cca7a3\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" Nov 01 16:22:47.028750 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:47.028724 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqkwv\" (UniqueName: \"kubernetes.io/projected/8744fda2-3d14-48d8-8426-323166cca7a3-kube-api-access-cqkwv\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn\" (UID: \"8744fda2-3d14-48d8-8426-323166cca7a3\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" Nov 01 16:22:47.175199 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:47.175164 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" Nov 01 16:22:47.489533 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:47.488668 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn"] Nov 01 16:22:47.489533 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:22:47.488727 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8744fda2_3d14_48d8_8426_323166cca7a3.slice/crio-7285936034e376093598c2c930467616717dbefe0c0f84f62c67793efd1a8b36 WatchSource:0}: Error finding container 7285936034e376093598c2c930467616717dbefe0c0f84f62c67793efd1a8b36: Status 404 returned error can't find the container with id 7285936034e376093598c2c930467616717dbefe0c0f84f62c67793efd1a8b36 Nov 01 16:22:47.492666 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:47.492633 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:22:47.492791 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:47.492709 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:22:47.492791 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:47.492754 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:22:47.834082 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:47.833844 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtwhw_8254aafa-dd5b-4150-87e4-d165dab6bcb5/ovn-acl-logging/0.log" Nov 01 16:22:47.837509 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:47.837478 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtwhw_8254aafa-dd5b-4150-87e4-d165dab6bcb5/ovn-acl-logging/0.log" Nov 01 16:22:48.395981 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:22:48.395940 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:22:48.400340 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:48.399205 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" event={"ID":"8744fda2-3d14-48d8-8426-323166cca7a3","Type":"ContainerStarted","Data":"430f3237ed8e6bf67c1ea07120f26bb5947c6920b5934151c3541be54ff8ba96"} Nov 01 16:22:48.400340 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:48.399317 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" event={"ID":"8744fda2-3d14-48d8-8426-323166cca7a3","Type":"ContainerStarted","Data":"7285936034e376093598c2c930467616717dbefe0c0f84f62c67793efd1a8b36"} Nov 01 16:22:48.420471 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:48.420425 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" podStartSLOduration=2.420410868 podStartE2EDuration="2.420410868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:22:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:22:48.418560884 +0000 UTC m=+2101.750298535" watchObservedRunningTime="2026-11-01 16:22:48.420410868 +0000 UTC m=+2101.752148517" Nov 01 16:22:49.181040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:49.176496 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" Nov 01 16:22:49.229614 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:49.227192 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Nov 01 16:22:49.229614 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:49.227255 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" podUID="8744fda2-3d14-48d8-8426-323166cca7a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Nov 01 16:22:49.431259 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:49.430042 4363 generic.go:356] "Generic (PLEG): container finished" podID="f4b349f5-acb2-42b2-8d90-01b2029167f7" containerID="db8bd7437ddcdc6617b7e6c3dc9d860fab8ad93befb0669687b273a8d20eed82" exitCode=0 Nov 01 16:22:49.431259 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:49.431180 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" event={"ID":"f4b349f5-acb2-42b2-8d90-01b2029167f7","Type":"ContainerDied","Data":"db8bd7437ddcdc6617b7e6c3dc9d860fab8ad93befb0669687b273a8d20eed82"} Nov 01 16:22:49.431666 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:49.431641 4363 scope.go:122] "RemoveContainer" containerID="db8bd7437ddcdc6617b7e6c3dc9d860fab8ad93befb0669687b273a8d20eed82" Nov 01 16:22:49.434571 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:49.434540 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:22:49.434657 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:49.434629 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:22:49.434715 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:49.434672 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:22:49.770776 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:49.770634 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" Nov 01 16:22:50.175875 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:50.175840 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Nov 01 16:22:50.176073 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:50.175905 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" podUID="8744fda2-3d14-48d8-8426-323166cca7a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Nov 01 16:22:50.441468 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:50.441432 4363 generic.go:356] "Generic (PLEG): container finished" podID="5f69b8bb-4aa6-4713-8b2f-138c5d358858" containerID="68171b1bd2001cb00c9191f8272dfe0efca9dd69364dfbdaad7ff19e1bd3b8a4" exitCode=0 Nov 01 16:22:50.441883 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:50.441603 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" event={"ID":"5f69b8bb-4aa6-4713-8b2f-138c5d358858","Type":"ContainerDied","Data":"68171b1bd2001cb00c9191f8272dfe0efca9dd69364dfbdaad7ff19e1bd3b8a4"} Nov 01 16:22:50.442197 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:50.442177 4363 scope.go:122] "RemoveContainer" containerID="68171b1bd2001cb00c9191f8272dfe0efca9dd69364dfbdaad7ff19e1bd3b8a4" Nov 01 16:22:50.445878 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:50.445842 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:22:50.445970 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:50.445934 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:22:50.446009 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:50.445982 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:22:50.451250 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:50.451227 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" event={"ID":"f4b349f5-acb2-42b2-8d90-01b2029167f7","Type":"ContainerStarted","Data":"46f0f59d149b20a47b982a1e8d12504e3ff18c8b7e990e8cff3bb909efe59d41"} Nov 01 16:22:51.189548 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:51.185209 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Nov 01 16:22:51.189548 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:51.185275 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" podUID="8744fda2-3d14-48d8-8426-323166cca7a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Nov 01 16:22:51.454845 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:51.453375 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" Nov 01 16:22:51.455440 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:51.454836 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 01 16:22:51.455440 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:51.454913 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" podUID="f4b349f5-acb2-42b2-8d90-01b2029167f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 01 16:22:51.466980 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:51.466945 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" event={"ID":"5f69b8bb-4aa6-4713-8b2f-138c5d358858","Type":"ContainerStarted","Data":"5c0cf1794a919c29141a45a01845845fd0431e56c59229210667d72142b0010f"} Nov 01 16:22:51.833097 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:51.832382 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" Nov 01 16:22:51.833097 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:51.832505 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Nov 01 16:22:51.833097 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:51.832550 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" podUID="5f69b8bb-4aa6-4713-8b2f-138c5d358858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Nov 01 16:22:52.191083 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:52.176098 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Nov 01 16:22:52.191083 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:52.176158 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" podUID="8744fda2-3d14-48d8-8426-323166cca7a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Nov 01 16:22:52.443108 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:52.442357 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 01 16:22:52.443108 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:52.442422 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" podUID="f4b349f5-acb2-42b2-8d90-01b2029167f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 01 16:22:52.832101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:52.831595 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Nov 01 16:22:52.832101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:52.831679 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" podUID="5f69b8bb-4aa6-4713-8b2f-138c5d358858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Nov 01 16:22:53.227045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:53.223999 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Nov 01 16:22:53.227045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:53.224075 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" podUID="8744fda2-3d14-48d8-8426-323166cca7a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Nov 01 16:22:53.472445 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:53.472410 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 01 16:22:53.472821 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:53.472770 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" podUID="f4b349f5-acb2-42b2-8d90-01b2029167f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 01 16:22:53.825048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:53.821104 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-7225e5/auth-test-model-7225e5-kserve-84f65d66df-jd8qc" podUID="901ec925-24b0-4f15-a356-b7a5da163158" containerName="main" probeResult="failure" output="Get \"https://10.134.0.215:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:22:53.832260 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:53.832226 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Nov 01 16:22:53.832895 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:53.832863 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" podUID="5f69b8bb-4aa6-4713-8b2f-138c5d358858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Nov 01 16:22:54.200204 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:54.200165 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Nov 01 16:22:54.200508 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:54.200477 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" podUID="8744fda2-3d14-48d8-8426-323166cca7a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Nov 01 16:22:54.248041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:54.247225 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-5e2d03/auth-test-model-5e2d03-kserve-665774f749-75t2g" podUID="e66518b6-c569-4afd-9464-ba934cccbcf7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.204:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:22:54.283678 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:54.278302 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:22:54.285301 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:54.283991 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-ae79df/test-model-ae79df-kserve-7c554b5c5-k248t" podUID="251044fd-4f4d-4423-9f9d-685c35f259a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.203:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:22:54.453418 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:54.453320 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 01 16:22:54.453418 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:54.453388 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" podUID="f4b349f5-acb2-42b2-8d90-01b2029167f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 01 16:22:54.457070 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:54.455513 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w5-383fde/auth-test-model-383fde-kserve-57c55ccb8f-rqp6x" podUID="36f441c7-2672-4581-9a0e-7142326e172b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.209:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:22:54.563295 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:54.563245 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w0-5e2d03/auth-test-model-5e2d03-kserve-665774f749-75t2g" podUID="e66518b6-c569-4afd-9464-ba934cccbcf7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.204:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:22:54.683530 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:54.683485 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:22:54.775711 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:54.775606 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:22:54.824114 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:54.824069 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-7225e5/auth-test-model-7225e5-kserve-84f65d66df-jd8qc" podUID="901ec925-24b0-4f15-a356-b7a5da163158" containerName="main" probeResult="failure" output="Get \"https://10.134.0.215:8000/ready\": context deadline exceeded" Nov 01 16:22:54.855838 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:54.855805 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Nov 01 16:22:54.856477 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:54.856445 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" podUID="5f69b8bb-4aa6-4713-8b2f-138c5d358858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Nov 01 16:22:54.881041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:54.876130 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-7225e5/auth-test-model-7225e5-kserve-84f65d66df-jd8qc" Nov 01 16:22:54.905039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:54.901491 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-7225e5/auth-test-model-7225e5-kserve-84f65d66df-jd8qc" Nov 01 16:22:55.472041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:55.448128 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 01 16:22:55.496141 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:55.496091 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" podUID="f4b349f5-acb2-42b2-8d90-01b2029167f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 01 16:22:55.755993 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:55.755432 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" Nov 01 16:22:55.758535 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:55.758512 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" Nov 01 16:22:55.769652 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:55.769259 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" Nov 01 16:22:55.867048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:55.866392 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Nov 01 16:22:55.867048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:55.866448 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" podUID="5f69b8bb-4aa6-4713-8b2f-138c5d358858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Nov 01 16:22:56.035786 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:56.035047 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:22:56.166165 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:56.166117 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-ae79df/test-model-ae79df-kserve-7c554b5c5-k248t" podUID="251044fd-4f4d-4423-9f9d-685c35f259a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.203:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:22:56.429729 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:56.429689 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 01 16:22:56.430101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:56.430066 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" podUID="f4b349f5-acb2-42b2-8d90-01b2029167f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 01 16:22:56.852044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:56.848754 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Nov 01 16:22:56.852044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:56.848813 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" podUID="5f69b8bb-4aa6-4713-8b2f-138c5d358858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Nov 01 16:22:57.070775 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:57.070751 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" Nov 01 16:22:57.426479 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:57.426438 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 01 16:22:57.429768 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:57.429732 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" podUID="f4b349f5-acb2-42b2-8d90-01b2029167f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 01 16:22:57.832188 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:57.832104 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Nov 01 16:22:57.832188 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:57.832163 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" podUID="5f69b8bb-4aa6-4713-8b2f-138c5d358858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Nov 01 16:22:58.431505 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:58.431469 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" Nov 01 16:22:58.431977 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:58.431959 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" Nov 01 16:22:58.432735 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:58.432718 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" Nov 01 16:22:58.835991 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:58.835886 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" Nov 01 16:22:58.837101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:22:58.837078 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" Nov 01 16:23:00.807350 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:00.807249 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw"] Nov 01 16:23:01.098393 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:01.098349 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" podUID="674b7965-e091-422f-b128-35635859883f" containerName="istio-proxy" containerID="cri-o://bf72c11e6f0a4c21909cb35bc0e5f34e511c723707e46605498c6d3fc98224c0" gracePeriod=30 Nov 01 16:23:06.393935 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:06.393906 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" Nov 01 16:23:06.451345 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:06.451309 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/674b7965-e091-422f-b128-35635859883f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/674b7965-e091-422f-b128-35635859883f-workload-certs\") pod \"674b7965-e091-422f-b128-35635859883f\" (UID: \"674b7965-e091-422f-b128-35635859883f\") " Nov 01 16:23:06.451756 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:06.451360 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/674b7965-e091-422f-b128-35635859883f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/674b7965-e091-422f-b128-35635859883f-istiod-ca-cert\") pod \"674b7965-e091-422f-b128-35635859883f\" (UID: \"674b7965-e091-422f-b128-35635859883f\") " Nov 01 16:23:06.451756 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:06.451512 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/674b7965-e091-422f-b128-35635859883f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/674b7965-e091-422f-b128-35635859883f-workload-socket\") pod \"674b7965-e091-422f-b128-35635859883f\" (UID: \"674b7965-e091-422f-b128-35635859883f\") " Nov 01 16:23:06.451756 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:06.451539 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/674b7965-e091-422f-b128-35635859883f-kube-api-access-hnpfs\" (UniqueName: \"kubernetes.io/projected/674b7965-e091-422f-b128-35635859883f-kube-api-access-hnpfs\") pod \"674b7965-e091-422f-b128-35635859883f\" (UID: \"674b7965-e091-422f-b128-35635859883f\") " Nov 01 16:23:06.451756 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:06.451599 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/674b7965-e091-422f-b128-35635859883f-workload-certs" pod "674b7965-e091-422f-b128-35635859883f" (UID: "674b7965-e091-422f-b128-35635859883f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:23:06.451756 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:06.451614 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/674b7965-e091-422f-b128-35635859883f-istio-token\" (UniqueName: \"kubernetes.io/projected/674b7965-e091-422f-b128-35635859883f-istio-token\") pod \"674b7965-e091-422f-b128-35635859883f\" (UID: \"674b7965-e091-422f-b128-35635859883f\") " Nov 01 16:23:06.451756 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:06.451724 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/674b7965-e091-422f-b128-35635859883f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/674b7965-e091-422f-b128-35635859883f-istio-data\") pod \"674b7965-e091-422f-b128-35635859883f\" (UID: \"674b7965-e091-422f-b128-35635859883f\") " Nov 01 16:23:06.452273 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:06.451786 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/674b7965-e091-422f-b128-35635859883f-workload-socket" pod "674b7965-e091-422f-b128-35635859883f" (UID: "674b7965-e091-422f-b128-35635859883f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:23:06.452273 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:06.451803 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/674b7965-e091-422f-b128-35635859883f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/674b7965-e091-422f-b128-35635859883f-istio-podinfo\") pod \"674b7965-e091-422f-b128-35635859883f\" (UID: \"674b7965-e091-422f-b128-35635859883f\") " Nov 01 16:23:06.452273 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:06.451870 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/674b7965-e091-422f-b128-35635859883f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/674b7965-e091-422f-b128-35635859883f-istio-envoy\") pod \"674b7965-e091-422f-b128-35635859883f\" (UID: \"674b7965-e091-422f-b128-35635859883f\") " Nov 01 16:23:06.452273 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:06.452037 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/674b7965-e091-422f-b128-35635859883f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/674b7965-e091-422f-b128-35635859883f-credential-socket\") pod \"674b7965-e091-422f-b128-35635859883f\" (UID: \"674b7965-e091-422f-b128-35635859883f\") " Nov 01 16:23:06.452273 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:06.452115 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/674b7965-e091-422f-b128-35635859883f-istiod-ca-cert" pod "674b7965-e091-422f-b128-35635859883f" (UID: "674b7965-e091-422f-b128-35635859883f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:23:06.452564 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:06.452349 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/674b7965-e091-422f-b128-35635859883f-istio-data" pod "674b7965-e091-422f-b128-35635859883f" (UID: "674b7965-e091-422f-b128-35635859883f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:23:06.452897 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:06.452851 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/674b7965-e091-422f-b128-35635859883f-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:23:06.452897 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:06.452876 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/674b7965-e091-422f-b128-35635859883f-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:23:06.452897 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:06.452890 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/674b7965-e091-422f-b128-35635859883f-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:23:06.453140 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:06.452903 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/674b7965-e091-422f-b128-35635859883f-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:23:06.453295 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:06.453252 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/674b7965-e091-422f-b128-35635859883f-credential-socket" pod "674b7965-e091-422f-b128-35635859883f" (UID: "674b7965-e091-422f-b128-35635859883f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:23:06.455441 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:06.455410 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/674b7965-e091-422f-b128-35635859883f-kube-api-access-hnpfs" pod "674b7965-e091-422f-b128-35635859883f" (UID: "674b7965-e091-422f-b128-35635859883f"). InnerVolumeSpecName "kube-api-access-hnpfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:23:06.456202 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:06.456178 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/674b7965-e091-422f-b128-35635859883f-istio-token" pod "674b7965-e091-422f-b128-35635859883f" (UID: "674b7965-e091-422f-b128-35635859883f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:23:06.457396 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:06.457369 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/674b7965-e091-422f-b128-35635859883f-istio-podinfo" pod "674b7965-e091-422f-b128-35635859883f" (UID: "674b7965-e091-422f-b128-35635859883f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:23:06.457656 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:06.457633 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/674b7965-e091-422f-b128-35635859883f-istio-envoy" pod "674b7965-e091-422f-b128-35635859883f" (UID: "674b7965-e091-422f-b128-35635859883f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:23:06.584347 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:06.558735 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/674b7965-e091-422f-b128-35635859883f-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:23:06.584347 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:06.558781 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnpfs\" (UniqueName: \"kubernetes.io/projected/674b7965-e091-422f-b128-35635859883f-kube-api-access-hnpfs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:23:06.584347 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:06.558799 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/674b7965-e091-422f-b128-35635859883f-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:23:06.584347 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:06.558814 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/674b7965-e091-422f-b128-35635859883f-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:23:06.584347 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:06.558829 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/674b7965-e091-422f-b128-35635859883f-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:23:07.151106 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:07.151047 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" event={"ID":"674b7965-e091-422f-b128-35635859883f","Type":"ContainerDied","Data":"bf72c11e6f0a4c21909cb35bc0e5f34e511c723707e46605498c6d3fc98224c0"} Nov 01 16:23:07.151284 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:07.151141 4363 scope.go:122] "RemoveContainer" containerID="bf72c11e6f0a4c21909cb35bc0e5f34e511c723707e46605498c6d3fc98224c0" Nov 01 16:23:07.151284 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:07.151053 4363 generic.go:356] "Generic (PLEG): container finished" podID="674b7965-e091-422f-b128-35635859883f" containerID="bf72c11e6f0a4c21909cb35bc0e5f34e511c723707e46605498c6d3fc98224c0" exitCode=0 Nov 01 16:23:07.151284 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:07.151080 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" Nov 01 16:23:07.151474 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:07.151314 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw" event={"ID":"674b7965-e091-422f-b128-35635859883f","Type":"ContainerDied","Data":"aca1e4213d6cc5c2d45444dfaae7b14625584c8ca1aee41e6503bd49063eafae"} Nov 01 16:23:07.163600 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:07.163582 4363 scope.go:122] "RemoveContainer" containerID="bf72c11e6f0a4c21909cb35bc0e5f34e511c723707e46605498c6d3fc98224c0" Nov 01 16:23:07.163868 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:23:07.163852 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf72c11e6f0a4c21909cb35bc0e5f34e511c723707e46605498c6d3fc98224c0\": container with ID starting with bf72c11e6f0a4c21909cb35bc0e5f34e511c723707e46605498c6d3fc98224c0 not found: ID does not exist" containerID="bf72c11e6f0a4c21909cb35bc0e5f34e511c723707e46605498c6d3fc98224c0" Nov 01 16:23:07.163924 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:07.163874 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf72c11e6f0a4c21909cb35bc0e5f34e511c723707e46605498c6d3fc98224c0"} err="failed to get container status \"bf72c11e6f0a4c21909cb35bc0e5f34e511c723707e46605498c6d3fc98224c0\": rpc error: code = NotFound desc = could not find container \"bf72c11e6f0a4c21909cb35bc0e5f34e511c723707e46605498c6d3fc98224c0\": container with ID starting with bf72c11e6f0a4c21909cb35bc0e5f34e511c723707e46605498c6d3fc98224c0 not found: ID does not exist" Nov 01 16:23:07.170532 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:07.170500 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw"] Nov 01 16:23:07.175542 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:07.175519 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-z4hmw"] Nov 01 16:23:08.805494 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:08.805453 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="674b7965-e091-422f-b128-35635859883f" path="/var/lib/kubelet/pods/674b7965-e091-422f-b128-35635859883f/volumes" Nov 01 16:23:18.313142 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:23:18.313094 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:23:18.878514 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:18.878473 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9e044c/test-model-9e044c-kserve-588c95d597-r2hz8"] Nov 01 16:23:26.339739 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.339698 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf"] Nov 01 16:23:26.341394 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.341359 4363 cpu_manager.go:414] "containerMap: removing container" podUID="674b7965-e091-422f-b128-35635859883f" containerName="istio-proxy" Nov 01 16:23:26.341394 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.341386 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="674b7965-e091-422f-b128-35635859883f" containerName="istio-proxy" Nov 01 16:23:26.341791 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.341772 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="674b7965-e091-422f-b128-35635859883f" containerName="istio-proxy" Nov 01 16:23:26.348006 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.347982 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" Nov 01 16:23:26.352777 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.352751 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-9a482b-openshift-default-dockercfg-bmbzh\"" Nov 01 16:23:26.354717 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.354694 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf"] Nov 01 16:23:26.360492 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.359605 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq"] Nov 01 16:23:26.386153 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.385236 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq"] Nov 01 16:23:26.386153 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.385358 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" Nov 01 16:23:26.432593 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.432551 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-istio-data\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq\" (UID: \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" Nov 01 16:23:26.432751 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.432617 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-istio-token\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq\" (UID: \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" Nov 01 16:23:26.432751 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.432667 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkbjm\" (UniqueName: \"kubernetes.io/projected/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-kube-api-access-xkbjm\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq\" (UID: \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" Nov 01 16:23:26.432849 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.432794 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7babd48-8700-4398-b87b-c282cbc99c4e-credential-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf\" (UID: \"d7babd48-8700-4398-b87b-c282cbc99c4e\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" Nov 01 16:23:26.432890 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.432851 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-istiod-ca-cert\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq\" (UID: \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" Nov 01 16:23:26.432890 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.432883 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7babd48-8700-4398-b87b-c282cbc99c4e-workload-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf\" (UID: \"d7babd48-8700-4398-b87b-c282cbc99c4e\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" Nov 01 16:23:26.432956 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.432945 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-istio-envoy\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq\" (UID: \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" Nov 01 16:23:26.432991 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.432981 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7babd48-8700-4398-b87b-c282cbc99c4e-istio-podinfo\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf\" (UID: \"d7babd48-8700-4398-b87b-c282cbc99c4e\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" Nov 01 16:23:26.433094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.433075 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7babd48-8700-4398-b87b-c282cbc99c4e-istiod-ca-cert\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf\" (UID: \"d7babd48-8700-4398-b87b-c282cbc99c4e\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" Nov 01 16:23:26.433303 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.433283 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7babd48-8700-4398-b87b-c282cbc99c4e-workload-certs\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf\" (UID: \"d7babd48-8700-4398-b87b-c282cbc99c4e\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" Nov 01 16:23:26.433382 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.433324 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7babd48-8700-4398-b87b-c282cbc99c4e-istio-token\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf\" (UID: \"d7babd48-8700-4398-b87b-c282cbc99c4e\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" Nov 01 16:23:26.433382 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.433374 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-istio-podinfo\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq\" (UID: \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" Nov 01 16:23:26.433452 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.433409 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-workload-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq\" (UID: \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" Nov 01 16:23:26.433495 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.433449 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7babd48-8700-4398-b87b-c282cbc99c4e-istio-envoy\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf\" (UID: \"d7babd48-8700-4398-b87b-c282cbc99c4e\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" Nov 01 16:23:26.433495 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.433482 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmrqk\" (UniqueName: \"kubernetes.io/projected/d7babd48-8700-4398-b87b-c282cbc99c4e-kube-api-access-zmrqk\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf\" (UID: \"d7babd48-8700-4398-b87b-c282cbc99c4e\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" Nov 01 16:23:26.433561 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.433520 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-workload-certs\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq\" (UID: \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" Nov 01 16:23:26.433667 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.433649 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-credential-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq\" (UID: \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" Nov 01 16:23:26.433891 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.433866 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7babd48-8700-4398-b87b-c282cbc99c4e-istio-data\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf\" (UID: \"d7babd48-8700-4398-b87b-c282cbc99c4e\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" Nov 01 16:23:26.534822 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.534781 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7babd48-8700-4398-b87b-c282cbc99c4e-istio-token\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf\" (UID: \"d7babd48-8700-4398-b87b-c282cbc99c4e\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" Nov 01 16:23:26.535062 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.534845 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-istio-podinfo\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq\" (UID: \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" Nov 01 16:23:26.535062 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.534878 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-workload-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq\" (UID: \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" Nov 01 16:23:26.535062 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.534914 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7babd48-8700-4398-b87b-c282cbc99c4e-istio-envoy\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf\" (UID: \"d7babd48-8700-4398-b87b-c282cbc99c4e\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" Nov 01 16:23:26.535062 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.534955 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmrqk\" (UniqueName: \"kubernetes.io/projected/d7babd48-8700-4398-b87b-c282cbc99c4e-kube-api-access-zmrqk\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf\" (UID: \"d7babd48-8700-4398-b87b-c282cbc99c4e\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" Nov 01 16:23:26.535062 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.534993 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-workload-certs\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq\" (UID: \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" Nov 01 16:23:26.535388 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.535132 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-credential-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq\" (UID: \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" Nov 01 16:23:26.535388 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.535227 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7babd48-8700-4398-b87b-c282cbc99c4e-istio-data\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf\" (UID: \"d7babd48-8700-4398-b87b-c282cbc99c4e\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" Nov 01 16:23:26.535388 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.535315 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-istio-data\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq\" (UID: \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" Nov 01 16:23:26.535388 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.535363 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-istio-token\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq\" (UID: \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" Nov 01 16:23:26.535607 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.535415 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkbjm\" (UniqueName: \"kubernetes.io/projected/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-kube-api-access-xkbjm\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq\" (UID: \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" Nov 01 16:23:26.535607 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.535462 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7babd48-8700-4398-b87b-c282cbc99c4e-credential-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf\" (UID: \"d7babd48-8700-4398-b87b-c282cbc99c4e\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" Nov 01 16:23:26.535607 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.535487 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-workload-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq\" (UID: \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" Nov 01 16:23:26.535607 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.535529 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-istiod-ca-cert\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq\" (UID: \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" Nov 01 16:23:26.535607 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.535555 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7babd48-8700-4398-b87b-c282cbc99c4e-istio-data\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf\" (UID: \"d7babd48-8700-4398-b87b-c282cbc99c4e\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" Nov 01 16:23:26.535818 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.535803 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7babd48-8700-4398-b87b-c282cbc99c4e-credential-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf\" (UID: \"d7babd48-8700-4398-b87b-c282cbc99c4e\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" Nov 01 16:23:26.535859 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.535561 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7babd48-8700-4398-b87b-c282cbc99c4e-workload-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf\" (UID: \"d7babd48-8700-4398-b87b-c282cbc99c4e\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" Nov 01 16:23:26.535929 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.535909 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-istio-envoy\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq\" (UID: \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" Nov 01 16:23:26.535989 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.535957 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7babd48-8700-4398-b87b-c282cbc99c4e-istio-podinfo\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf\" (UID: \"d7babd48-8700-4398-b87b-c282cbc99c4e\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" Nov 01 16:23:26.536068 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.536031 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7babd48-8700-4398-b87b-c282cbc99c4e-istiod-ca-cert\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf\" (UID: \"d7babd48-8700-4398-b87b-c282cbc99c4e\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" Nov 01 16:23:26.536134 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.536066 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7babd48-8700-4398-b87b-c282cbc99c4e-workload-certs\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf\" (UID: \"d7babd48-8700-4398-b87b-c282cbc99c4e\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" Nov 01 16:23:26.536134 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.536117 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-credential-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq\" (UID: \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" Nov 01 16:23:26.536251 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.536187 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-workload-certs\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq\" (UID: \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" Nov 01 16:23:26.536251 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.536213 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-istio-data\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq\" (UID: \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" Nov 01 16:23:26.536367 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.536244 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7babd48-8700-4398-b87b-c282cbc99c4e-workload-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf\" (UID: \"d7babd48-8700-4398-b87b-c282cbc99c4e\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" Nov 01 16:23:26.536510 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.536486 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7babd48-8700-4398-b87b-c282cbc99c4e-workload-certs\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf\" (UID: \"d7babd48-8700-4398-b87b-c282cbc99c4e\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" Nov 01 16:23:26.536970 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.536947 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-istiod-ca-cert\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq\" (UID: \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" Nov 01 16:23:26.537089 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.536975 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7babd48-8700-4398-b87b-c282cbc99c4e-istiod-ca-cert\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf\" (UID: \"d7babd48-8700-4398-b87b-c282cbc99c4e\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" Nov 01 16:23:26.537440 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.537421 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7babd48-8700-4398-b87b-c282cbc99c4e-istio-envoy\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf\" (UID: \"d7babd48-8700-4398-b87b-c282cbc99c4e\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" Nov 01 16:23:26.537849 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.537827 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-istio-podinfo\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq\" (UID: \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" Nov 01 16:23:26.538579 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.538557 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-istio-envoy\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq\" (UID: \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" Nov 01 16:23:26.538905 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.538883 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7babd48-8700-4398-b87b-c282cbc99c4e-istio-podinfo\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf\" (UID: \"d7babd48-8700-4398-b87b-c282cbc99c4e\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" Nov 01 16:23:26.542803 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.542782 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7babd48-8700-4398-b87b-c282cbc99c4e-istio-token\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf\" (UID: \"d7babd48-8700-4398-b87b-c282cbc99c4e\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" Nov 01 16:23:26.543009 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.542984 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkbjm\" (UniqueName: \"kubernetes.io/projected/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-kube-api-access-xkbjm\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq\" (UID: \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" Nov 01 16:23:26.543188 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.543170 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-istio-token\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq\" (UID: \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" Nov 01 16:23:26.543301 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.543261 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmrqk\" (UniqueName: \"kubernetes.io/projected/d7babd48-8700-4398-b87b-c282cbc99c4e-kube-api-access-zmrqk\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf\" (UID: \"d7babd48-8700-4398-b87b-c282cbc99c4e\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" Nov 01 16:23:26.674942 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.674912 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" Nov 01 16:23:26.723293 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.723258 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" Nov 01 16:23:26.997726 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:26.997693 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf"] Nov 01 16:23:26.997887 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:23:26.997787 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7babd48_8700_4398_b87b_c282cbc99c4e.slice/crio-9393fb2e8ee6cb11848b8e86a550a319241dc47e90f12cc9def5469ec4cb3993 WatchSource:0}: Error finding container 9393fb2e8ee6cb11848b8e86a550a319241dc47e90f12cc9def5469ec4cb3993: Status 404 returned error can't find the container with id 9393fb2e8ee6cb11848b8e86a550a319241dc47e90f12cc9def5469ec4cb3993 Nov 01 16:23:27.001603 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:27.001556 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:23:27.001727 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:27.001647 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:23:27.001727 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:27.001694 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:23:27.110055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:27.109214 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:23:27.110055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:27.109286 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:23:27.110055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:27.109317 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:23:27.110055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:27.109652 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq"] Nov 01 16:23:27.354150 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:27.354109 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" event={"ID":"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95","Type":"ContainerStarted","Data":"e53343934f58a222255dca8b3bf908e425bb95ccac95033c90c154ea64b54c59"} Nov 01 16:23:27.354639 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:27.354176 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" event={"ID":"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95","Type":"ContainerStarted","Data":"287a7142d645ddba079c56c24bfaa9e134ac02b0c711a160083b7c7ad87102c8"} Nov 01 16:23:27.356580 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:27.356545 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" event={"ID":"d7babd48-8700-4398-b87b-c282cbc99c4e","Type":"ContainerStarted","Data":"3d297cf78921df8bc352150246b9065dfbdb424a0fa844ea5e1a94143043b5de"} Nov 01 16:23:27.356580 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:27.356601 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" event={"ID":"d7babd48-8700-4398-b87b-c282cbc99c4e","Type":"ContainerStarted","Data":"9393fb2e8ee6cb11848b8e86a550a319241dc47e90f12cc9def5469ec4cb3993"} Nov 01 16:23:27.373848 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:27.373807 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" podStartSLOduration=1.3737933820000001 podStartE2EDuration="1.373793382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:23:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:23:27.37190327 +0000 UTC m=+2140.703640957" watchObservedRunningTime="2026-11-01 16:23:27.373793382 +0000 UTC m=+2140.705531031" Nov 01 16:23:27.391933 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:27.391889 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" podStartSLOduration=1.39187398 podStartE2EDuration="1.39187398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:23:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:23:27.391156908 +0000 UTC m=+2140.722894559" watchObservedRunningTime="2026-11-01 16:23:27.39187398 +0000 UTC m=+2140.723611689" Nov 01 16:23:27.675916 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:27.675838 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" Nov 01 16:23:27.724114 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:27.724077 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" Nov 01 16:23:28.730040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:28.700960 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Nov 01 16:23:28.740047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:28.730667 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" podUID="d7babd48-8700-4398-b87b-c282cbc99c4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Nov 01 16:23:28.744043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:28.740571 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Nov 01 16:23:28.744043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:28.740651 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" podUID="9c37d94d-deb1-4a00-9d58-d91bb4b2dd95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Nov 01 16:23:29.675953 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:29.675898 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Nov 01 16:23:29.676155 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:29.675959 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" podUID="d7babd48-8700-4398-b87b-c282cbc99c4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Nov 01 16:23:29.731932 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:29.731452 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Nov 01 16:23:29.731932 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:29.731553 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" podUID="9c37d94d-deb1-4a00-9d58-d91bb4b2dd95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Nov 01 16:23:30.676494 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:30.676461 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Nov 01 16:23:30.698043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:30.694255 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" podUID="d7babd48-8700-4398-b87b-c282cbc99c4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Nov 01 16:23:30.731041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:30.730415 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Nov 01 16:23:30.731041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:30.730478 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" podUID="9c37d94d-deb1-4a00-9d58-d91bb4b2dd95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Nov 01 16:23:31.680042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:31.679361 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Nov 01 16:23:31.680042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:31.679440 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" podUID="d7babd48-8700-4398-b87b-c282cbc99c4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Nov 01 16:23:31.725040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:31.724442 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Nov 01 16:23:31.725040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:31.724506 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" podUID="9c37d94d-deb1-4a00-9d58-d91bb4b2dd95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Nov 01 16:23:32.358361 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:32.358324 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ae79df/test-model-ae79df-kserve-7c554b5c5-k248t"] Nov 01 16:23:32.824097 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:32.823377 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Nov 01 16:23:32.824097 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:32.823443 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" podUID="9c37d94d-deb1-4a00-9d58-d91bb4b2dd95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Nov 01 16:23:32.884126 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:32.876184 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Nov 01 16:23:32.884126 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:32.876247 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" podUID="d7babd48-8700-4398-b87b-c282cbc99c4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Nov 01 16:23:33.686068 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:33.677644 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Nov 01 16:23:33.686068 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:33.677719 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" podUID="d7babd48-8700-4398-b87b-c282cbc99c4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Nov 01 16:23:33.728154 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:33.728117 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Nov 01 16:23:33.728446 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:33.728415 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" podUID="9c37d94d-deb1-4a00-9d58-d91bb4b2dd95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Nov 01 16:23:33.938096 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:33.928934 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-9e044c/test-model-9e044c-kserve-588c95d597-r2hz8" podUID="6a04c3a0-59d4-4214-81a4-5d840c85ce35" containerName="main" containerID="cri-o://8c024ef8e3bd9207a7bcc7b8561ab997e958ca3ffc7b0cd8e94fb00bfd3e383c" gracePeriod=45 Nov 01 16:23:34.426215 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:34.420655 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9e044c/test-model-9e044c-kserve-588c95d597-r2hz8" Nov 01 16:23:34.574167 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:34.574135 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a04c3a0-59d4-4214-81a4-5d840c85ce35-tls-certs\" (UniqueName: \"kubernetes.io/secret/6a04c3a0-59d4-4214-81a4-5d840c85ce35-tls-certs\") pod \"6a04c3a0-59d4-4214-81a4-5d840c85ce35\" (UID: \"6a04c3a0-59d4-4214-81a4-5d840c85ce35\") " Nov 01 16:23:34.574388 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:34.574197 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a04c3a0-59d4-4214-81a4-5d840c85ce35-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a04c3a0-59d4-4214-81a4-5d840c85ce35-tmp-dir\") pod \"6a04c3a0-59d4-4214-81a4-5d840c85ce35\" (UID: \"6a04c3a0-59d4-4214-81a4-5d840c85ce35\") " Nov 01 16:23:34.574456 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:34.574394 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a04c3a0-59d4-4214-81a4-5d840c85ce35-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a04c3a0-59d4-4214-81a4-5d840c85ce35-dshm\") pod \"6a04c3a0-59d4-4214-81a4-5d840c85ce35\" (UID: \"6a04c3a0-59d4-4214-81a4-5d840c85ce35\") " Nov 01 16:23:34.574548 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:34.574524 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a04c3a0-59d4-4214-81a4-5d840c85ce35-tmp-dir" pod "6a04c3a0-59d4-4214-81a4-5d840c85ce35" (UID: "6a04c3a0-59d4-4214-81a4-5d840c85ce35"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:23:34.574763 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:34.574743 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a04c3a0-59d4-4214-81a4-5d840c85ce35-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a04c3a0-59d4-4214-81a4-5d840c85ce35-model-cache\") pod \"6a04c3a0-59d4-4214-81a4-5d840c85ce35\" (UID: \"6a04c3a0-59d4-4214-81a4-5d840c85ce35\") " Nov 01 16:23:34.574930 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:34.574792 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a04c3a0-59d4-4214-81a4-5d840c85ce35-home\" (UniqueName: \"kubernetes.io/empty-dir/6a04c3a0-59d4-4214-81a4-5d840c85ce35-home\") pod \"6a04c3a0-59d4-4214-81a4-5d840c85ce35\" (UID: \"6a04c3a0-59d4-4214-81a4-5d840c85ce35\") " Nov 01 16:23:34.574930 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:34.574814 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a04c3a0-59d4-4214-81a4-5d840c85ce35-kube-api-access-szjzn\" (UniqueName: \"kubernetes.io/projected/6a04c3a0-59d4-4214-81a4-5d840c85ce35-kube-api-access-szjzn\") pod \"6a04c3a0-59d4-4214-81a4-5d840c85ce35\" (UID: \"6a04c3a0-59d4-4214-81a4-5d840c85ce35\") " Nov 01 16:23:34.575105 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:34.574994 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a04c3a0-59d4-4214-81a4-5d840c85ce35-model-cache" pod "6a04c3a0-59d4-4214-81a4-5d840c85ce35" (UID: "6a04c3a0-59d4-4214-81a4-5d840c85ce35"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:23:34.575243 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:34.575221 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a04c3a0-59d4-4214-81a4-5d840c85ce35-home" pod "6a04c3a0-59d4-4214-81a4-5d840c85ce35" (UID: "6a04c3a0-59d4-4214-81a4-5d840c85ce35"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:23:34.575737 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:34.575715 4363 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a04c3a0-59d4-4214-81a4-5d840c85ce35-tmp-dir\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:23:34.575737 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:34.575734 4363 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a04c3a0-59d4-4214-81a4-5d840c85ce35-model-cache\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:23:34.575896 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:34.575744 4363 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a04c3a0-59d4-4214-81a4-5d840c85ce35-home\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:23:34.576895 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:34.576867 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a04c3a0-59d4-4214-81a4-5d840c85ce35-tls-certs" pod "6a04c3a0-59d4-4214-81a4-5d840c85ce35" (UID: "6a04c3a0-59d4-4214-81a4-5d840c85ce35"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:23:34.577049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:34.576959 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a04c3a0-59d4-4214-81a4-5d840c85ce35-dshm" pod "6a04c3a0-59d4-4214-81a4-5d840c85ce35" (UID: "6a04c3a0-59d4-4214-81a4-5d840c85ce35"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:23:34.581499 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:34.581475 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a04c3a0-59d4-4214-81a4-5d840c85ce35-kube-api-access-szjzn" pod "6a04c3a0-59d4-4214-81a4-5d840c85ce35" (UID: "6a04c3a0-59d4-4214-81a4-5d840c85ce35"). InnerVolumeSpecName "kube-api-access-szjzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:23:34.684216 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:34.677347 4363 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a04c3a0-59d4-4214-81a4-5d840c85ce35-tls-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:23:34.684216 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:34.677380 4363 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a04c3a0-59d4-4214-81a4-5d840c85ce35-dshm\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:23:34.684216 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:34.677394 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szjzn\" (UniqueName: \"kubernetes.io/projected/6a04c3a0-59d4-4214-81a4-5d840c85ce35-kube-api-access-szjzn\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:23:34.691738 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:34.689441 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" Nov 01 16:23:34.691738 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:34.690609 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" Nov 01 16:23:34.691738 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:34.690799 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" Nov 01 16:23:34.706175 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:34.703517 4363 generic.go:356] "Generic (PLEG): container finished" podID="6a04c3a0-59d4-4214-81a4-5d840c85ce35" containerID="8c024ef8e3bd9207a7bcc7b8561ab997e958ca3ffc7b0cd8e94fb00bfd3e383c" exitCode=0 Nov 01 16:23:34.706175 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:34.703782 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9e044c/test-model-9e044c-kserve-588c95d597-r2hz8" event={"ID":"6a04c3a0-59d4-4214-81a4-5d840c85ce35","Type":"ContainerDied","Data":"8c024ef8e3bd9207a7bcc7b8561ab997e958ca3ffc7b0cd8e94fb00bfd3e383c"} Nov 01 16:23:34.706175 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:34.703824 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9e044c/test-model-9e044c-kserve-588c95d597-r2hz8" event={"ID":"6a04c3a0-59d4-4214-81a4-5d840c85ce35","Type":"ContainerDied","Data":"4a5e37ac717bfdb356c96e6b65e87df98860d94f40649b030e6c190ddc5c6391"} Nov 01 16:23:34.706175 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:34.703846 4363 scope.go:122] "RemoveContainer" containerID="8c024ef8e3bd9207a7bcc7b8561ab997e958ca3ffc7b0cd8e94fb00bfd3e383c" Nov 01 16:23:34.706175 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:34.704011 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9e044c/test-model-9e044c-kserve-588c95d597-r2hz8" Nov 01 16:23:34.730771 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:34.730746 4363 scope.go:122] "RemoveContainer" containerID="8c024ef8e3bd9207a7bcc7b8561ab997e958ca3ffc7b0cd8e94fb00bfd3e383c" Nov 01 16:23:34.731163 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:23:34.731134 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c024ef8e3bd9207a7bcc7b8561ab997e958ca3ffc7b0cd8e94fb00bfd3e383c\": container with ID starting with 8c024ef8e3bd9207a7bcc7b8561ab997e958ca3ffc7b0cd8e94fb00bfd3e383c not found: ID does not exist" containerID="8c024ef8e3bd9207a7bcc7b8561ab997e958ca3ffc7b0cd8e94fb00bfd3e383c" Nov 01 16:23:34.731275 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:34.731170 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c024ef8e3bd9207a7bcc7b8561ab997e958ca3ffc7b0cd8e94fb00bfd3e383c"} err="failed to get container status \"8c024ef8e3bd9207a7bcc7b8561ab997e958ca3ffc7b0cd8e94fb00bfd3e383c\": rpc error: code = NotFound desc = could not find container \"8c024ef8e3bd9207a7bcc7b8561ab997e958ca3ffc7b0cd8e94fb00bfd3e383c\": container with ID starting with 8c024ef8e3bd9207a7bcc7b8561ab997e958ca3ffc7b0cd8e94fb00bfd3e383c not found: ID does not exist" Nov 01 16:23:34.756810 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:34.756774 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9e044c/test-model-9e044c-kserve-588c95d597-r2hz8"] Nov 01 16:23:34.764495 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:34.764460 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-9e044c/test-model-9e044c-kserve-588c95d597-r2hz8"] Nov 01 16:23:34.815600 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:34.815567 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a04c3a0-59d4-4214-81a4-5d840c85ce35" path="/var/lib/kubelet/pods/6a04c3a0-59d4-4214-81a4-5d840c85ce35/volumes" Nov 01 16:23:34.927104 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:34.926881 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" Nov 01 16:23:34.927807 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:34.927785 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" Nov 01 16:23:34.929370 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:34.929343 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" Nov 01 16:23:41.394394 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:41.394350 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg"] Nov 01 16:23:41.395753 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:41.395724 4363 cpu_manager.go:414] "containerMap: removing container" podUID="6a04c3a0-59d4-4214-81a4-5d840c85ce35" containerName="main" Nov 01 16:23:41.395753 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:41.395750 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a04c3a0-59d4-4214-81a4-5d840c85ce35" containerName="main" Nov 01 16:23:41.396047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:41.395967 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a04c3a0-59d4-4214-81a4-5d840c85ce35" containerName="main" Nov 01 16:23:41.407906 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:41.405894 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" Nov 01 16:23:41.688266 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:41.688108 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg"] Nov 01 16:23:41.703771 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:41.697256 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6505861-2940-41bb-8dbd-d71b797cd2c6-workload-certs\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg\" (UID: \"a6505861-2940-41bb-8dbd-d71b797cd2c6\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" Nov 01 16:23:41.703771 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:41.697315 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6505861-2940-41bb-8dbd-d71b797cd2c6-istiod-ca-cert\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg\" (UID: \"a6505861-2940-41bb-8dbd-d71b797cd2c6\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" Nov 01 16:23:41.703771 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:41.697422 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6505861-2940-41bb-8dbd-d71b797cd2c6-istio-podinfo\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg\" (UID: \"a6505861-2940-41bb-8dbd-d71b797cd2c6\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" Nov 01 16:23:41.703771 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:41.697449 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6505861-2940-41bb-8dbd-d71b797cd2c6-istio-token\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg\" (UID: \"a6505861-2940-41bb-8dbd-d71b797cd2c6\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" Nov 01 16:23:41.703771 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:41.697491 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6505861-2940-41bb-8dbd-d71b797cd2c6-istio-envoy\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg\" (UID: \"a6505861-2940-41bb-8dbd-d71b797cd2c6\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" Nov 01 16:23:41.703771 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:41.697758 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6505861-2940-41bb-8dbd-d71b797cd2c6-credential-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg\" (UID: \"a6505861-2940-41bb-8dbd-d71b797cd2c6\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" Nov 01 16:23:41.703771 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:41.697788 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w69hx\" (UniqueName: \"kubernetes.io/projected/a6505861-2940-41bb-8dbd-d71b797cd2c6-kube-api-access-w69hx\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg\" (UID: \"a6505861-2940-41bb-8dbd-d71b797cd2c6\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" Nov 01 16:23:41.703771 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:41.697820 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6505861-2940-41bb-8dbd-d71b797cd2c6-istio-data\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg\" (UID: \"a6505861-2940-41bb-8dbd-d71b797cd2c6\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" Nov 01 16:23:41.703771 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:41.697876 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6505861-2940-41bb-8dbd-d71b797cd2c6-workload-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg\" (UID: \"a6505861-2940-41bb-8dbd-d71b797cd2c6\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" Nov 01 16:23:41.799034 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:41.798979 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6505861-2940-41bb-8dbd-d71b797cd2c6-workload-certs\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg\" (UID: \"a6505861-2940-41bb-8dbd-d71b797cd2c6\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" Nov 01 16:23:41.799251 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:41.799063 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6505861-2940-41bb-8dbd-d71b797cd2c6-istiod-ca-cert\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg\" (UID: \"a6505861-2940-41bb-8dbd-d71b797cd2c6\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" Nov 01 16:23:41.799251 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:41.799159 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6505861-2940-41bb-8dbd-d71b797cd2c6-istio-podinfo\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg\" (UID: \"a6505861-2940-41bb-8dbd-d71b797cd2c6\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" Nov 01 16:23:41.799251 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:41.799187 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6505861-2940-41bb-8dbd-d71b797cd2c6-istio-token\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg\" (UID: \"a6505861-2940-41bb-8dbd-d71b797cd2c6\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" Nov 01 16:23:41.799251 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:41.799222 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6505861-2940-41bb-8dbd-d71b797cd2c6-istio-envoy\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg\" (UID: \"a6505861-2940-41bb-8dbd-d71b797cd2c6\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" Nov 01 16:23:41.799499 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:41.799388 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6505861-2940-41bb-8dbd-d71b797cd2c6-workload-certs\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg\" (UID: \"a6505861-2940-41bb-8dbd-d71b797cd2c6\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" Nov 01 16:23:41.799499 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:41.799472 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6505861-2940-41bb-8dbd-d71b797cd2c6-credential-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg\" (UID: \"a6505861-2940-41bb-8dbd-d71b797cd2c6\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" Nov 01 16:23:41.799499 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:41.799496 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w69hx\" (UniqueName: \"kubernetes.io/projected/a6505861-2940-41bb-8dbd-d71b797cd2c6-kube-api-access-w69hx\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg\" (UID: \"a6505861-2940-41bb-8dbd-d71b797cd2c6\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" Nov 01 16:23:41.799672 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:41.799518 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6505861-2940-41bb-8dbd-d71b797cd2c6-istio-data\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg\" (UID: \"a6505861-2940-41bb-8dbd-d71b797cd2c6\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" Nov 01 16:23:41.799672 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:41.799553 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6505861-2940-41bb-8dbd-d71b797cd2c6-workload-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg\" (UID: \"a6505861-2940-41bb-8dbd-d71b797cd2c6\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" Nov 01 16:23:41.800174 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:41.799823 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6505861-2940-41bb-8dbd-d71b797cd2c6-istiod-ca-cert\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg\" (UID: \"a6505861-2940-41bb-8dbd-d71b797cd2c6\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" Nov 01 16:23:41.800174 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:41.799922 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6505861-2940-41bb-8dbd-d71b797cd2c6-credential-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg\" (UID: \"a6505861-2940-41bb-8dbd-d71b797cd2c6\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" Nov 01 16:23:41.800174 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:41.800094 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6505861-2940-41bb-8dbd-d71b797cd2c6-workload-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg\" (UID: \"a6505861-2940-41bb-8dbd-d71b797cd2c6\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" Nov 01 16:23:41.800174 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:41.800128 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6505861-2940-41bb-8dbd-d71b797cd2c6-istio-data\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg\" (UID: \"a6505861-2940-41bb-8dbd-d71b797cd2c6\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" Nov 01 16:23:41.802083 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:41.802058 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6505861-2940-41bb-8dbd-d71b797cd2c6-istio-podinfo\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg\" (UID: \"a6505861-2940-41bb-8dbd-d71b797cd2c6\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" Nov 01 16:23:41.802194 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:41.802134 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6505861-2940-41bb-8dbd-d71b797cd2c6-istio-envoy\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg\" (UID: \"a6505861-2940-41bb-8dbd-d71b797cd2c6\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" Nov 01 16:23:41.806788 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:41.806763 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6505861-2940-41bb-8dbd-d71b797cd2c6-istio-token\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg\" (UID: \"a6505861-2940-41bb-8dbd-d71b797cd2c6\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" Nov 01 16:23:41.806959 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:41.806942 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w69hx\" (UniqueName: \"kubernetes.io/projected/a6505861-2940-41bb-8dbd-d71b797cd2c6-kube-api-access-w69hx\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg\" (UID: \"a6505861-2940-41bb-8dbd-d71b797cd2c6\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" Nov 01 16:23:42.015140 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:42.014343 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" Nov 01 16:23:42.336501 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:42.336465 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg"] Nov 01 16:23:42.337425 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:23:42.337383 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6505861_2940_41bb_8dbd_d71b797cd2c6.slice/crio-f13bd2fa60d457df0157d11cd5f7ad57d7ddd4da30cf62867dd4f7e1364a2822 WatchSource:0}: Error finding container f13bd2fa60d457df0157d11cd5f7ad57d7ddd4da30cf62867dd4f7e1364a2822: Status 404 returned error can't find the container with id f13bd2fa60d457df0157d11cd5f7ad57d7ddd4da30cf62867dd4f7e1364a2822 Nov 01 16:23:42.341104 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:42.341064 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:23:42.341215 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:42.341142 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:23:42.341215 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:42.341179 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:23:42.737707 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:42.737669 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9e044c-668c79d887-78jg4"] Nov 01 16:23:42.859062 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:42.857983 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" event={"ID":"a6505861-2940-41bb-8dbd-d71b797cd2c6","Type":"ContainerStarted","Data":"9582032c5133eac85e5b978a2473e0a66e33af27b2bd142e961cbea19ae34faa"} Nov 01 16:23:42.859062 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:42.858104 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" event={"ID":"a6505861-2940-41bb-8dbd-d71b797cd2c6","Type":"ContainerStarted","Data":"f13bd2fa60d457df0157d11cd5f7ad57d7ddd4da30cf62867dd4f7e1364a2822"} Nov 01 16:23:42.893411 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:42.893352 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" podStartSLOduration=1.8933336920000001 podStartE2EDuration="1.893333692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:23:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:23:42.8907823 +0000 UTC m=+2156.222519963" watchObservedRunningTime="2026-11-01 16:23:42.893333692 +0000 UTC m=+2156.225071343" Nov 01 16:23:43.015257 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:43.015165 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" Nov 01 16:23:43.016953 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:43.016747 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Nov 01 16:23:43.016953 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:43.016814 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" podUID="a6505861-2940-41bb-8dbd-d71b797cd2c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Nov 01 16:23:44.023042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:44.019557 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Nov 01 16:23:44.023042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:44.019635 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" podUID="a6505861-2940-41bb-8dbd-d71b797cd2c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Nov 01 16:23:45.019041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:45.018136 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Nov 01 16:23:45.019041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:45.018208 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" podUID="a6505861-2940-41bb-8dbd-d71b797cd2c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Nov 01 16:23:46.055051 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:46.028402 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Nov 01 16:23:46.055051 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:46.028463 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" podUID="a6505861-2940-41bb-8dbd-d71b797cd2c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Nov 01 16:23:47.034490 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:47.034453 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Nov 01 16:23:47.034775 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:47.034746 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" podUID="a6505861-2940-41bb-8dbd-d71b797cd2c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Nov 01 16:23:47.659257 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:47.659151 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ae79df/test-model-ae79df-kserve-7c554b5c5-k248t" podUID="251044fd-4f4d-4423-9f9d-685c35f259a9" containerName="main" containerID="cri-o://18fea58b64ec153d557805b6a0835c2e0f5c360a66e945f244666491cfb5c2fe" gracePeriod=45 Nov 01 16:23:47.661121 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:47.661087 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp"] Nov 01 16:23:47.672771 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:47.672747 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp"] Nov 01 16:23:47.673043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:47.672999 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp" Nov 01 16:23:47.720423 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:47.720393 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-c8537c\"/\"kube-root-ca.crt\"" Nov 01 16:23:47.725220 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:47.725192 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-c8537c\"/\"default-dockercfg-dgbbr\"" Nov 01 16:23:47.725518 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:47.725498 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-c8537c\"/\"auth-test-model-c8537c-kserve-self-signed-certs\"" Nov 01 16:23:47.727739 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:47.727717 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-c8537c\"/\"openshift-service-ca.crt\"" Nov 01 16:23:47.741069 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:47.738114 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e044c-668c79d887-78jg4" podUID="79acdb79-6330-4958-a12d-d42d9d5c8d4e" containerName="payload-pre-processing" containerID="cri-o://95e8035c38cc902452b8d1cf36881d5003e298ef2e9fc05a55fe25f66db2073e" gracePeriod=25 Nov 01 16:23:47.776499 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:47.776467 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8ca3021-8cbd-4cfc-aab7-33bb656f4c01-tmp-dir\") pod \"auth-test-model-c8537c-kserve-5859bdfb6f-kblvp\" (UID: \"f8ca3021-8cbd-4cfc-aab7-33bb656f4c01\") " pod="ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp" Nov 01 16:23:47.776658 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:47.776598 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8ca3021-8cbd-4cfc-aab7-33bb656f4c01-dshm\") pod \"auth-test-model-c8537c-kserve-5859bdfb6f-kblvp\" (UID: \"f8ca3021-8cbd-4cfc-aab7-33bb656f4c01\") " pod="ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp" Nov 01 16:23:47.776711 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:47.776654 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8ca3021-8cbd-4cfc-aab7-33bb656f4c01-home\") pod \"auth-test-model-c8537c-kserve-5859bdfb6f-kblvp\" (UID: \"f8ca3021-8cbd-4cfc-aab7-33bb656f4c01\") " pod="ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp" Nov 01 16:23:47.776711 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:47.776682 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g66mz\" (UniqueName: \"kubernetes.io/projected/f8ca3021-8cbd-4cfc-aab7-33bb656f4c01-kube-api-access-g66mz\") pod \"auth-test-model-c8537c-kserve-5859bdfb6f-kblvp\" (UID: \"f8ca3021-8cbd-4cfc-aab7-33bb656f4c01\") " pod="ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp" Nov 01 16:23:47.777053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:47.776978 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8ca3021-8cbd-4cfc-aab7-33bb656f4c01-tls-certs\") pod \"auth-test-model-c8537c-kserve-5859bdfb6f-kblvp\" (UID: \"f8ca3021-8cbd-4cfc-aab7-33bb656f4c01\") " pod="ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp" Nov 01 16:23:47.805075 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:47.777201 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8ca3021-8cbd-4cfc-aab7-33bb656f4c01-model-cache\") pod \"auth-test-model-c8537c-kserve-5859bdfb6f-kblvp\" (UID: \"f8ca3021-8cbd-4cfc-aab7-33bb656f4c01\") " pod="ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp" Nov 01 16:23:47.969649 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:47.886365 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8ca3021-8cbd-4cfc-aab7-33bb656f4c01-model-cache\") pod \"auth-test-model-c8537c-kserve-5859bdfb6f-kblvp\" (UID: \"f8ca3021-8cbd-4cfc-aab7-33bb656f4c01\") " pod="ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp" Nov 01 16:23:47.969649 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:47.886484 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8ca3021-8cbd-4cfc-aab7-33bb656f4c01-tmp-dir\") pod \"auth-test-model-c8537c-kserve-5859bdfb6f-kblvp\" (UID: \"f8ca3021-8cbd-4cfc-aab7-33bb656f4c01\") " pod="ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp" Nov 01 16:23:47.969649 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:47.886553 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8ca3021-8cbd-4cfc-aab7-33bb656f4c01-dshm\") pod \"auth-test-model-c8537c-kserve-5859bdfb6f-kblvp\" (UID: \"f8ca3021-8cbd-4cfc-aab7-33bb656f4c01\") " pod="ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp" Nov 01 16:23:47.969649 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:47.886582 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8ca3021-8cbd-4cfc-aab7-33bb656f4c01-home\") pod \"auth-test-model-c8537c-kserve-5859bdfb6f-kblvp\" (UID: \"f8ca3021-8cbd-4cfc-aab7-33bb656f4c01\") " pod="ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp" Nov 01 16:23:47.969649 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:47.886608 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g66mz\" (UniqueName: \"kubernetes.io/projected/f8ca3021-8cbd-4cfc-aab7-33bb656f4c01-kube-api-access-g66mz\") pod \"auth-test-model-c8537c-kserve-5859bdfb6f-kblvp\" (UID: \"f8ca3021-8cbd-4cfc-aab7-33bb656f4c01\") " pod="ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp" Nov 01 16:23:47.969649 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:47.886750 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8ca3021-8cbd-4cfc-aab7-33bb656f4c01-tls-certs\") pod \"auth-test-model-c8537c-kserve-5859bdfb6f-kblvp\" (UID: \"f8ca3021-8cbd-4cfc-aab7-33bb656f4c01\") " pod="ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp" Nov 01 16:23:47.969649 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:47.913412 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8ca3021-8cbd-4cfc-aab7-33bb656f4c01-home\") pod \"auth-test-model-c8537c-kserve-5859bdfb6f-kblvp\" (UID: \"f8ca3021-8cbd-4cfc-aab7-33bb656f4c01\") " pod="ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp" Nov 01 16:23:47.969649 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:47.913983 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8ca3021-8cbd-4cfc-aab7-33bb656f4c01-model-cache\") pod \"auth-test-model-c8537c-kserve-5859bdfb6f-kblvp\" (UID: \"f8ca3021-8cbd-4cfc-aab7-33bb656f4c01\") " pod="ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp" Nov 01 16:23:47.969649 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:47.914271 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8ca3021-8cbd-4cfc-aab7-33bb656f4c01-tmp-dir\") pod \"auth-test-model-c8537c-kserve-5859bdfb6f-kblvp\" (UID: \"f8ca3021-8cbd-4cfc-aab7-33bb656f4c01\") " pod="ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp" Nov 01 16:23:48.044788 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:48.042817 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8ca3021-8cbd-4cfc-aab7-33bb656f4c01-dshm\") pod \"auth-test-model-c8537c-kserve-5859bdfb6f-kblvp\" (UID: \"f8ca3021-8cbd-4cfc-aab7-33bb656f4c01\") " pod="ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp" Nov 01 16:23:48.044788 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:48.043846 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8ca3021-8cbd-4cfc-aab7-33bb656f4c01-tls-certs\") pod \"auth-test-model-c8537c-kserve-5859bdfb6f-kblvp\" (UID: \"f8ca3021-8cbd-4cfc-aab7-33bb656f4c01\") " pod="ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp" Nov 01 16:23:48.044788 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:48.044337 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Nov 01 16:23:48.044788 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:48.044398 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" podUID="a6505861-2940-41bb-8dbd-d71b797cd2c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Nov 01 16:23:48.088232 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:48.088198 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g66mz\" (UniqueName: \"kubernetes.io/projected/f8ca3021-8cbd-4cfc-aab7-33bb656f4c01-kube-api-access-g66mz\") pod \"auth-test-model-c8537c-kserve-5859bdfb6f-kblvp\" (UID: \"f8ca3021-8cbd-4cfc-aab7-33bb656f4c01\") " pod="ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp" Nov 01 16:23:48.160504 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:48.160462 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-ae79df/test-model-ae79df-kserve-7c554b5c5-k248t" podUID="251044fd-4f4d-4423-9f9d-685c35f259a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.203:8000/ready\": dial tcp 10.134.0.203:8000: connect: connection refused" Nov 01 16:23:48.175788 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:48.175755 4363 generic.go:356] "Generic (PLEG): container finished" podID="251044fd-4f4d-4423-9f9d-685c35f259a9" containerID="18fea58b64ec153d557805b6a0835c2e0f5c360a66e945f244666491cfb5c2fe" exitCode=0 Nov 01 16:23:48.175967 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:48.175831 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ae79df/test-model-ae79df-kserve-7c554b5c5-k248t" event={"ID":"251044fd-4f4d-4423-9f9d-685c35f259a9","Type":"ContainerDied","Data":"18fea58b64ec153d557805b6a0835c2e0f5c360a66e945f244666491cfb5c2fe"} Nov 01 16:23:48.375982 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:48.375946 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp" Nov 01 16:23:48.440456 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:48.440428 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ae79df/test-model-ae79df-kserve-7c554b5c5-k248t" Nov 01 16:23:48.451192 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:23:48.451163 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:23:48.591178 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:48.581088 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/251044fd-4f4d-4423-9f9d-685c35f259a9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/251044fd-4f4d-4423-9f9d-685c35f259a9-dshm\") pod \"251044fd-4f4d-4423-9f9d-685c35f259a9\" (UID: \"251044fd-4f4d-4423-9f9d-685c35f259a9\") " Nov 01 16:23:48.591178 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:48.581349 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/251044fd-4f4d-4423-9f9d-685c35f259a9-home\" (UniqueName: \"kubernetes.io/empty-dir/251044fd-4f4d-4423-9f9d-685c35f259a9-home\") pod \"251044fd-4f4d-4423-9f9d-685c35f259a9\" (UID: \"251044fd-4f4d-4423-9f9d-685c35f259a9\") " Nov 01 16:23:48.591178 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:48.581459 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/251044fd-4f4d-4423-9f9d-685c35f259a9-kube-api-access-smfgs\" (UniqueName: \"kubernetes.io/projected/251044fd-4f4d-4423-9f9d-685c35f259a9-kube-api-access-smfgs\") pod \"251044fd-4f4d-4423-9f9d-685c35f259a9\" (UID: \"251044fd-4f4d-4423-9f9d-685c35f259a9\") " Nov 01 16:23:48.591178 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:48.581490 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/251044fd-4f4d-4423-9f9d-685c35f259a9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/251044fd-4f4d-4423-9f9d-685c35f259a9-model-cache\") pod \"251044fd-4f4d-4423-9f9d-685c35f259a9\" (UID: \"251044fd-4f4d-4423-9f9d-685c35f259a9\") " Nov 01 16:23:48.591178 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:48.581539 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/251044fd-4f4d-4423-9f9d-685c35f259a9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/251044fd-4f4d-4423-9f9d-685c35f259a9-tmp-dir\") pod \"251044fd-4f4d-4423-9f9d-685c35f259a9\" (UID: \"251044fd-4f4d-4423-9f9d-685c35f259a9\") " Nov 01 16:23:48.591178 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:48.581594 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/251044fd-4f4d-4423-9f9d-685c35f259a9-tls-certs\" (UniqueName: \"kubernetes.io/secret/251044fd-4f4d-4423-9f9d-685c35f259a9-tls-certs\") pod \"251044fd-4f4d-4423-9f9d-685c35f259a9\" (UID: \"251044fd-4f4d-4423-9f9d-685c35f259a9\") " Nov 01 16:23:48.595656 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:48.595294 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/251044fd-4f4d-4423-9f9d-685c35f259a9-tmp-dir" pod "251044fd-4f4d-4423-9f9d-685c35f259a9" (UID: "251044fd-4f4d-4423-9f9d-685c35f259a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:23:48.616094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:48.595860 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/251044fd-4f4d-4423-9f9d-685c35f259a9-model-cache" pod "251044fd-4f4d-4423-9f9d-685c35f259a9" (UID: "251044fd-4f4d-4423-9f9d-685c35f259a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:23:48.616842 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:48.616423 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/251044fd-4f4d-4423-9f9d-685c35f259a9-home" pod "251044fd-4f4d-4423-9f9d-685c35f259a9" (UID: "251044fd-4f4d-4423-9f9d-685c35f259a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:23:48.721053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:48.703334 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/251044fd-4f4d-4423-9f9d-685c35f259a9-kube-api-access-smfgs" pod "251044fd-4f4d-4423-9f9d-685c35f259a9" (UID: "251044fd-4f4d-4423-9f9d-685c35f259a9"). InnerVolumeSpecName "kube-api-access-smfgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:23:48.721845 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:48.721759 4363 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/251044fd-4f4d-4423-9f9d-685c35f259a9-home\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:23:48.721845 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:48.721792 4363 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/251044fd-4f4d-4423-9f9d-685c35f259a9-model-cache\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:23:48.721845 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:48.721810 4363 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/251044fd-4f4d-4423-9f9d-685c35f259a9-tmp-dir\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:23:48.774906 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:48.774418 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/251044fd-4f4d-4423-9f9d-685c35f259a9-tls-certs" pod "251044fd-4f4d-4423-9f9d-685c35f259a9" (UID: "251044fd-4f4d-4423-9f9d-685c35f259a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:23:48.777461 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:48.775626 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/251044fd-4f4d-4423-9f9d-685c35f259a9-dshm" pod "251044fd-4f4d-4423-9f9d-685c35f259a9" (UID: "251044fd-4f4d-4423-9f9d-685c35f259a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:23:48.862068 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:48.861986 4363 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/251044fd-4f4d-4423-9f9d-685c35f259a9-dshm\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:23:48.862068 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:48.862048 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smfgs\" (UniqueName: \"kubernetes.io/projected/251044fd-4f4d-4423-9f9d-685c35f259a9-kube-api-access-smfgs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:23:48.862068 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:48.862064 4363 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/251044fd-4f4d-4423-9f9d-685c35f259a9-tls-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:23:49.052556 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:49.052521 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Nov 01 16:23:49.052814 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:49.052787 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" podUID="a6505861-2940-41bb-8dbd-d71b797cd2c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Nov 01 16:23:49.201446 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:49.193690 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp"] Nov 01 16:23:49.249153 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:49.249124 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ae79df/test-model-ae79df-kserve-7c554b5c5-k248t" event={"ID":"251044fd-4f4d-4423-9f9d-685c35f259a9","Type":"ContainerDied","Data":"5f1d5d0ac3c02fa1d349117a3137ad505a0fb5fc5163ca61d1abed33d10cf913"} Nov 01 16:23:49.249354 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:49.249338 4363 scope.go:122] "RemoveContainer" containerID="18fea58b64ec153d557805b6a0835c2e0f5c360a66e945f244666491cfb5c2fe" Nov 01 16:23:49.249680 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:49.249185 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ae79df/test-model-ae79df-kserve-7c554b5c5-k248t" Nov 01 16:23:49.321228 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:49.306077 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ae79df/test-model-ae79df-kserve-7c554b5c5-k248t"] Nov 01 16:23:49.328216 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:49.328167 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ae79df/test-model-ae79df-kserve-7c554b5c5-k248t"] Nov 01 16:23:50.146909 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:50.146876 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" Nov 01 16:23:50.148573 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:50.148544 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" Nov 01 16:23:50.149975 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:50.149947 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" Nov 01 16:23:50.271747 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:50.271707 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp" event={"ID":"f8ca3021-8cbd-4cfc-aab7-33bb656f4c01","Type":"ContainerStarted","Data":"1f83e3929c29391804dfd6954867e2dd6cdf5ab8f08d70e2ee812dd3f5a51446"} Nov 01 16:23:50.272136 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:50.271763 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp" event={"ID":"f8ca3021-8cbd-4cfc-aab7-33bb656f4c01","Type":"ContainerStarted","Data":"8916bb946ec624f1b7024d6451ca0cacbd7d9d12fcdef0eccb8dd2d19e6d3b40"} Nov 01 16:23:50.297779 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:50.297724 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp" podStartSLOduration=3.29771004 podStartE2EDuration="3.29771004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:23:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:23:50.293304461 +0000 UTC m=+2163.625042109" watchObservedRunningTime="2026-11-01 16:23:50.29771004 +0000 UTC m=+2163.629447689" Nov 01 16:23:50.376176 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:50.376139 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp" Nov 01 16:23:50.804726 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:50.804639 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="251044fd-4f4d-4423-9f9d-685c35f259a9" path="/var/lib/kubelet/pods/251044fd-4f4d-4423-9f9d-685c35f259a9/volumes" Nov 01 16:23:51.672254 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:51.672213 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e044c-668c79d887-78jg4" podUID="79acdb79-6330-4958-a12d-d42d9d5c8d4e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:23:53.436160 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:53.436115 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:23:53.436678 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:53.436165 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" podUID="5f58c340-e931-42d5-9322-2f60420216fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:23:55.320631 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:55.320598 4363 generic.go:356] "Generic (PLEG): container finished" podID="5f58c340-e931-42d5-9322-2f60420216fb" containerID="87a8952f9e0890633201b4cb35a1a240e85574976da72c9b8522e064ff91ab4e" exitCode=0 Nov 01 16:23:55.321123 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:55.320672 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" event={"ID":"5f58c340-e931-42d5-9322-2f60420216fb","Type":"ContainerDied","Data":"87a8952f9e0890633201b4cb35a1a240e85574976da72c9b8522e064ff91ab4e"} Nov 01 16:23:55.321311 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:55.321293 4363 scope.go:122] "RemoveContainer" containerID="87a8952f9e0890633201b4cb35a1a240e85574976da72c9b8522e064ff91ab4e" Nov 01 16:23:55.323692 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:55.323660 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:23:55.323794 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:55.323752 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:23:55.323840 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:55.323799 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:23:56.333674 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:56.333632 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" event={"ID":"5f58c340-e931-42d5-9322-2f60420216fb","Type":"ContainerStarted","Data":"6cda8bb655a705c48dbdeb3c9ed430060e56dbe87850ef256169e37234f887dc"} Nov 01 16:23:56.429040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:56.426491 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7"] Nov 01 16:23:56.429040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:56.427330 4363 cpu_manager.go:414] "containerMap: removing container" podUID="251044fd-4f4d-4423-9f9d-685c35f259a9" containerName="main" Nov 01 16:23:56.429040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:56.427344 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="251044fd-4f4d-4423-9f9d-685c35f259a9" containerName="main" Nov 01 16:23:56.429344 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:56.429152 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="251044fd-4f4d-4423-9f9d-685c35f259a9" containerName="main" Nov 01 16:23:56.437301 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:56.437271 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7"] Nov 01 16:23:56.437479 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:56.437403 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" Nov 01 16:23:56.472155 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:56.472121 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-istiod-ca-cert\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7\" (UID: \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" Nov 01 16:23:56.472324 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:56.472293 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-istio-envoy\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7\" (UID: \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" Nov 01 16:23:56.472375 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:56.472331 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-workload-certs\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7\" (UID: \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" Nov 01 16:23:56.472462 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:56.472442 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-istio-token\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7\" (UID: \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" Nov 01 16:23:56.472505 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:56.472497 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-istio-podinfo\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7\" (UID: \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" Nov 01 16:23:56.472615 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:56.472602 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-workload-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7\" (UID: \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" Nov 01 16:23:56.472654 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:56.472630 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgkqv\" (UniqueName: \"kubernetes.io/projected/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-kube-api-access-fgkqv\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7\" (UID: \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" Nov 01 16:23:56.472654 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:56.472648 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-credential-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7\" (UID: \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" Nov 01 16:23:56.472744 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:56.472733 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-istio-data\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7\" (UID: \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" Nov 01 16:23:56.574356 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:56.574322 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-workload-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7\" (UID: \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" Nov 01 16:23:56.574533 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:56.574369 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgkqv\" (UniqueName: \"kubernetes.io/projected/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-kube-api-access-fgkqv\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7\" (UID: \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" Nov 01 16:23:56.574533 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:56.574484 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-credential-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7\" (UID: \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" Nov 01 16:23:56.574620 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:56.574533 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-istio-data\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7\" (UID: \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" Nov 01 16:23:56.574723 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:56.574706 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-istiod-ca-cert\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7\" (UID: \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" Nov 01 16:23:56.574798 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:56.574773 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-workload-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7\" (UID: \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" Nov 01 16:23:56.574845 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:56.574795 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-credential-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7\" (UID: \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" Nov 01 16:23:56.574881 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:56.574861 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-istio-envoy\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7\" (UID: \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" Nov 01 16:23:56.574921 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:56.574895 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-workload-certs\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7\" (UID: \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" Nov 01 16:23:56.574921 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:56.574909 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-istio-data\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7\" (UID: \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" Nov 01 16:23:56.575010 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:56.574993 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-istio-token\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7\" (UID: \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" Nov 01 16:23:56.575117 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:56.575101 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-istio-podinfo\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7\" (UID: \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" Nov 01 16:23:56.575201 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:56.575185 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-workload-certs\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7\" (UID: \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" Nov 01 16:23:56.575463 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:56.575442 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-istiod-ca-cert\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7\" (UID: \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" Nov 01 16:23:56.587083 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:56.577061 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-istio-envoy\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7\" (UID: \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" Nov 01 16:23:56.614586 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:56.614549 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-istio-token\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7\" (UID: \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" Nov 01 16:23:56.615354 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:56.615278 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-istio-podinfo\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7\" (UID: \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" Nov 01 16:23:56.618144 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:56.618125 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgkqv\" (UniqueName: \"kubernetes.io/projected/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-kube-api-access-fgkqv\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7\" (UID: \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" Nov 01 16:23:56.754708 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:56.754673 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" Nov 01 16:23:57.230951 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:57.230911 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7"] Nov 01 16:23:57.237787 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:23:57.237757 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf32c7fe0_caa2_4a6d_80db_aec48cd9ddda.slice/crio-b2d5d9326a0e87086f7dc5a454879d2c3ea916ecec8c59f6651b4407469ceb8b WatchSource:0}: Error finding container b2d5d9326a0e87086f7dc5a454879d2c3ea916ecec8c59f6651b4407469ceb8b: Status 404 returned error can't find the container with id b2d5d9326a0e87086f7dc5a454879d2c3ea916ecec8c59f6651b4407469ceb8b Nov 01 16:23:57.240995 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:57.240963 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:23:57.241121 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:57.241047 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:23:57.241121 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:57.241079 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:23:57.308517 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:57.308490 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" Nov 01 16:23:57.308875 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:57.308780 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Nov 01 16:23:57.309087 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:57.309062 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" podUID="5f58c340-e931-42d5-9322-2f60420216fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Nov 01 16:23:57.344930 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:57.344586 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" event={"ID":"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda","Type":"ContainerStarted","Data":"b2d5d9326a0e87086f7dc5a454879d2c3ea916ecec8c59f6651b4407469ceb8b"} Nov 01 16:23:58.295049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:58.294617 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Nov 01 16:23:58.295049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:58.294695 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" podUID="5f58c340-e931-42d5-9322-2f60420216fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Nov 01 16:23:58.365744 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:58.365706 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" event={"ID":"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda","Type":"ContainerStarted","Data":"eb89bd1fca927c7b5dd5be025577f78abb01193fdf94913158b4335fc43bea95"} Nov 01 16:23:58.381865 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:58.381834 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp" Nov 01 16:23:58.398618 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:58.398555 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" podStartSLOduration=2.398535433 podStartE2EDuration="2.398535433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:23:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:23:58.398446116 +0000 UTC m=+2171.730183766" watchObservedRunningTime="2026-11-01 16:23:58.398535433 +0000 UTC m=+2171.730273084" Nov 01 16:23:58.458361 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:58.458329 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp" Nov 01 16:23:58.755530 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:58.755495 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" Nov 01 16:23:58.757200 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:58.757172 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Nov 01 16:23:58.757285 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:58.757234 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" podUID="f32c7fe0-caa2-4a6d-80db-aec48cd9ddda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Nov 01 16:23:59.293425 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:59.293387 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Nov 01 16:23:59.293733 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:59.293701 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" podUID="5f58c340-e931-42d5-9322-2f60420216fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Nov 01 16:23:59.459585 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:59.459547 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp" Nov 01 16:23:59.758474 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:59.758439 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Nov 01 16:23:59.758836 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:23:59.758792 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" podUID="f32c7fe0-caa2-4a6d-80db-aec48cd9ddda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Nov 01 16:24:00.340429 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:00.340240 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Nov 01 16:24:00.340737 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:00.340704 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" podUID="5f58c340-e931-42d5-9322-2f60420216fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Nov 01 16:24:00.755674 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:00.755638 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Nov 01 16:24:00.756092 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:00.755694 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" podUID="f32c7fe0-caa2-4a6d-80db-aec48cd9ddda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Nov 01 16:24:01.305180 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:01.305143 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Nov 01 16:24:01.309068 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:01.305464 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" podUID="5f58c340-e931-42d5-9322-2f60420216fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Nov 01 16:24:01.700174 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:01.700132 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e044c-668c79d887-78jg4" podUID="79acdb79-6330-4958-a12d-d42d9d5c8d4e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:24:01.755485 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:01.755449 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Nov 01 16:24:01.755807 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:01.755775 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" podUID="f32c7fe0-caa2-4a6d-80db-aec48cd9ddda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Nov 01 16:24:02.290050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:02.289370 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Nov 01 16:24:02.290050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:02.289433 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" podUID="5f58c340-e931-42d5-9322-2f60420216fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Nov 01 16:24:02.755532 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:02.755498 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Nov 01 16:24:02.755725 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:02.755553 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" podUID="f32c7fe0-caa2-4a6d-80db-aec48cd9ddda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Nov 01 16:24:03.300043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:03.299808 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" Nov 01 16:24:03.313041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:03.301579 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" Nov 01 16:24:03.313041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:03.303209 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" Nov 01 16:24:03.755234 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:03.755201 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Nov 01 16:24:03.755409 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:03.755256 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" podUID="f32c7fe0-caa2-4a6d-80db-aec48cd9ddda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Nov 01 16:24:04.981273 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:04.981228 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" Nov 01 16:24:04.984273 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:04.984244 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" Nov 01 16:24:04.986878 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:04.986853 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" Nov 01 16:24:06.657240 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:06.657184 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl"] Nov 01 16:24:06.877099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:06.876964 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" podUID="22eae6cc-4752-4174-a0f0-573adc82e32b" containerName="istio-proxy" containerID="cri-o://931a83ee2d5ef1d73bdeec852b45c33a754b9fe3ca32df298f4970f18e95938c" gracePeriod=30 Nov 01 16:24:07.153005 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.150855 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv"] Nov 01 16:24:07.157239 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.157215 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv"] Nov 01 16:24:07.157423 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.157263 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd"] Nov 01 16:24:07.158483 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.158457 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" Nov 01 16:24:07.161587 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.161565 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-734859-openshift-default-dockercfg-gwnw2\"" Nov 01 16:24:07.162407 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.162389 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd"] Nov 01 16:24:07.162525 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.162484 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" Nov 01 16:24:07.165199 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.165173 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-f47890-openshift-default-dockercfg-qpfmq\"" Nov 01 16:24:07.297493 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.297400 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-662ph\" (UniqueName: \"kubernetes.io/projected/fcd61588-08cf-49c1-9dab-febc5c612ca8-kube-api-access-662ph\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv\" (UID: \"fcd61588-08cf-49c1-9dab-febc5c612ca8\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" Nov 01 16:24:07.297493 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.297470 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-istio-envoy\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd\" (UID: \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" Nov 01 16:24:07.297713 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.297631 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd61588-08cf-49c1-9dab-febc5c612ca8-credential-socket\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv\" (UID: \"fcd61588-08cf-49c1-9dab-febc5c612ca8\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" Nov 01 16:24:07.297713 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.297691 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84q4q\" (UniqueName: \"kubernetes.io/projected/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-kube-api-access-84q4q\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd\" (UID: \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" Nov 01 16:24:07.297798 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.297718 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-credential-socket\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd\" (UID: \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" Nov 01 16:24:07.297798 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.297743 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-workload-socket\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd\" (UID: \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" Nov 01 16:24:07.297798 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.297779 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd61588-08cf-49c1-9dab-febc5c612ca8-workload-socket\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv\" (UID: \"fcd61588-08cf-49c1-9dab-febc5c612ca8\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" Nov 01 16:24:07.297947 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.297826 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-istio-podinfo\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd\" (UID: \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" Nov 01 16:24:07.298009 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.297949 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-istio-data\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd\" (UID: \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" Nov 01 16:24:07.298009 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.297977 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcd61588-08cf-49c1-9dab-febc5c612ca8-istiod-ca-cert\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv\" (UID: \"fcd61588-08cf-49c1-9dab-febc5c612ca8\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" Nov 01 16:24:07.298173 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.298084 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-istio-token\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd\" (UID: \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" Nov 01 16:24:07.298173 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.298107 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-workload-certs\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd\" (UID: \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" Nov 01 16:24:07.298325 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.298304 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcd61588-08cf-49c1-9dab-febc5c612ca8-istio-data\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv\" (UID: \"fcd61588-08cf-49c1-9dab-febc5c612ca8\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" Nov 01 16:24:07.298378 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.298365 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcd61588-08cf-49c1-9dab-febc5c612ca8-istio-podinfo\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv\" (UID: \"fcd61588-08cf-49c1-9dab-febc5c612ca8\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" Nov 01 16:24:07.298422 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.298390 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcd61588-08cf-49c1-9dab-febc5c612ca8-istio-token\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv\" (UID: \"fcd61588-08cf-49c1-9dab-febc5c612ca8\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" Nov 01 16:24:07.298566 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.298545 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcd61588-08cf-49c1-9dab-febc5c612ca8-istio-envoy\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv\" (UID: \"fcd61588-08cf-49c1-9dab-febc5c612ca8\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" Nov 01 16:24:07.298687 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.298672 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcd61588-08cf-49c1-9dab-febc5c612ca8-workload-certs\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv\" (UID: \"fcd61588-08cf-49c1-9dab-febc5c612ca8\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" Nov 01 16:24:07.298764 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.298694 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-istiod-ca-cert\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd\" (UID: \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" Nov 01 16:24:07.400584 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.400547 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-istio-data\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd\" (UID: \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" Nov 01 16:24:07.400584 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.400595 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcd61588-08cf-49c1-9dab-febc5c612ca8-istiod-ca-cert\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv\" (UID: \"fcd61588-08cf-49c1-9dab-febc5c612ca8\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" Nov 01 16:24:07.400825 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.400665 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-istio-token\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd\" (UID: \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" Nov 01 16:24:07.400825 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.400698 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-workload-certs\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd\" (UID: \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" Nov 01 16:24:07.400950 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.400825 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcd61588-08cf-49c1-9dab-febc5c612ca8-istio-data\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv\" (UID: \"fcd61588-08cf-49c1-9dab-febc5c612ca8\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" Nov 01 16:24:07.400950 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.400881 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcd61588-08cf-49c1-9dab-febc5c612ca8-istio-podinfo\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv\" (UID: \"fcd61588-08cf-49c1-9dab-febc5c612ca8\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" Nov 01 16:24:07.400950 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.400926 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcd61588-08cf-49c1-9dab-febc5c612ca8-istio-token\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv\" (UID: \"fcd61588-08cf-49c1-9dab-febc5c612ca8\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" Nov 01 16:24:07.401160 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.400986 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcd61588-08cf-49c1-9dab-febc5c612ca8-istio-envoy\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv\" (UID: \"fcd61588-08cf-49c1-9dab-febc5c612ca8\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" Nov 01 16:24:07.401160 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.400996 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-istio-data\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd\" (UID: \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" Nov 01 16:24:07.401160 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.401109 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcd61588-08cf-49c1-9dab-febc5c612ca8-workload-certs\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv\" (UID: \"fcd61588-08cf-49c1-9dab-febc5c612ca8\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" Nov 01 16:24:07.401160 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.401135 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-istiod-ca-cert\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd\" (UID: \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" Nov 01 16:24:07.401388 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.401183 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-662ph\" (UniqueName: \"kubernetes.io/projected/fcd61588-08cf-49c1-9dab-febc5c612ca8-kube-api-access-662ph\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv\" (UID: \"fcd61588-08cf-49c1-9dab-febc5c612ca8\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" Nov 01 16:24:07.401388 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.401222 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-istio-envoy\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd\" (UID: \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" Nov 01 16:24:07.401388 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.401233 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-workload-certs\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd\" (UID: \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" Nov 01 16:24:07.401388 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.401256 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd61588-08cf-49c1-9dab-febc5c612ca8-credential-socket\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv\" (UID: \"fcd61588-08cf-49c1-9dab-febc5c612ca8\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" Nov 01 16:24:07.401388 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.401300 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcd61588-08cf-49c1-9dab-febc5c612ca8-istio-data\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv\" (UID: \"fcd61588-08cf-49c1-9dab-febc5c612ca8\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" Nov 01 16:24:07.401388 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.401326 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84q4q\" (UniqueName: \"kubernetes.io/projected/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-kube-api-access-84q4q\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd\" (UID: \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" Nov 01 16:24:07.401388 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.401387 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-credential-socket\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd\" (UID: \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" Nov 01 16:24:07.401845 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.401444 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-workload-socket\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd\" (UID: \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" Nov 01 16:24:07.401845 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.401452 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcd61588-08cf-49c1-9dab-febc5c612ca8-istiod-ca-cert\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv\" (UID: \"fcd61588-08cf-49c1-9dab-febc5c612ca8\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" Nov 01 16:24:07.401845 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.401474 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd61588-08cf-49c1-9dab-febc5c612ca8-workload-socket\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv\" (UID: \"fcd61588-08cf-49c1-9dab-febc5c612ca8\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" Nov 01 16:24:07.401845 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.401529 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-istio-podinfo\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd\" (UID: \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" Nov 01 16:24:07.401845 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.401527 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd61588-08cf-49c1-9dab-febc5c612ca8-credential-socket\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv\" (UID: \"fcd61588-08cf-49c1-9dab-febc5c612ca8\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" Nov 01 16:24:07.402135 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.401920 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-istiod-ca-cert\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd\" (UID: \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" Nov 01 16:24:07.402235 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.402210 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcd61588-08cf-49c1-9dab-febc5c612ca8-workload-certs\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv\" (UID: \"fcd61588-08cf-49c1-9dab-febc5c612ca8\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" Nov 01 16:24:07.402347 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.402322 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-workload-socket\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd\" (UID: \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" Nov 01 16:24:07.402425 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.402390 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd61588-08cf-49c1-9dab-febc5c612ca8-workload-socket\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv\" (UID: \"fcd61588-08cf-49c1-9dab-febc5c612ca8\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" Nov 01 16:24:07.402744 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.402724 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-credential-socket\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd\" (UID: \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" Nov 01 16:24:07.403960 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.403928 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcd61588-08cf-49c1-9dab-febc5c612ca8-istio-envoy\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv\" (UID: \"fcd61588-08cf-49c1-9dab-febc5c612ca8\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" Nov 01 16:24:07.404380 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.404356 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcd61588-08cf-49c1-9dab-febc5c612ca8-istio-podinfo\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv\" (UID: \"fcd61588-08cf-49c1-9dab-febc5c612ca8\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" Nov 01 16:24:07.404505 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.404483 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-istio-envoy\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd\" (UID: \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" Nov 01 16:24:07.404982 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.404959 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-istio-podinfo\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd\" (UID: \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" Nov 01 16:24:07.409459 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.409433 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-istio-token\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd\" (UID: \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" Nov 01 16:24:07.410823 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.410199 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-662ph\" (UniqueName: \"kubernetes.io/projected/fcd61588-08cf-49c1-9dab-febc5c612ca8-kube-api-access-662ph\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv\" (UID: \"fcd61588-08cf-49c1-9dab-febc5c612ca8\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" Nov 01 16:24:07.410823 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.410401 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcd61588-08cf-49c1-9dab-febc5c612ca8-istio-token\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv\" (UID: \"fcd61588-08cf-49c1-9dab-febc5c612ca8\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" Nov 01 16:24:07.411421 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.411398 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84q4q\" (UniqueName: \"kubernetes.io/projected/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-kube-api-access-84q4q\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd\" (UID: \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" Nov 01 16:24:07.482976 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.482392 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" Nov 01 16:24:07.502588 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.500391 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" Nov 01 16:24:07.788099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.788067 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv"] Nov 01 16:24:07.791836 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.791799 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:24:07.791962 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.791915 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:24:07.791962 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.791952 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:24:07.826727 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.826696 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd"] Nov 01 16:24:07.846042 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:24:07.846001 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf9b2de9_f52e_4580_af9e_d77e06ee6e64.slice/crio-be586793e31a557aa9ae900a9c4d1267086d5b4ba93db804210484bab9831ec4 WatchSource:0}: Error finding container be586793e31a557aa9ae900a9c4d1267086d5b4ba93db804210484bab9831ec4: Status 404 returned error can't find the container with id be586793e31a557aa9ae900a9c4d1267086d5b4ba93db804210484bab9831ec4 Nov 01 16:24:07.849462 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.849364 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:24:07.849462 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.849445 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:24:07.849700 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.849488 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:24:07.972939 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:07.972916 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e044c-668c79d887-78jg4" Nov 01 16:24:08.116864 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:08.114799 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79acdb79-6330-4958-a12d-d42d9d5c8d4e-kube-api-access-x7lj4\" (UniqueName: \"kubernetes.io/projected/79acdb79-6330-4958-a12d-d42d9d5c8d4e-kube-api-access-x7lj4\") pod \"79acdb79-6330-4958-a12d-d42d9d5c8d4e\" (UID: \"79acdb79-6330-4958-a12d-d42d9d5c8d4e\") " Nov 01 16:24:08.116864 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:08.114908 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79acdb79-6330-4958-a12d-d42d9d5c8d4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79acdb79-6330-4958-a12d-d42d9d5c8d4e-plugins-config-volume\") pod \"79acdb79-6330-4958-a12d-d42d9d5c8d4e\" (UID: \"79acdb79-6330-4958-a12d-d42d9d5c8d4e\") " Nov 01 16:24:08.116864 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:08.116158 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79acdb79-6330-4958-a12d-d42d9d5c8d4e-plugins-config-volume" pod "79acdb79-6330-4958-a12d-d42d9d5c8d4e" (UID: "79acdb79-6330-4958-a12d-d42d9d5c8d4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:24:08.116864 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:08.116442 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" event={"ID":"fcd61588-08cf-49c1-9dab-febc5c612ca8","Type":"ContainerStarted","Data":"1b6fd1f6379c2ca60d8ee0af7b8bd2dd0870bec9c1f4cc2e89721e16010d4d1b"} Nov 01 16:24:08.116864 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:08.116499 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" event={"ID":"fcd61588-08cf-49c1-9dab-febc5c612ca8","Type":"ContainerStarted","Data":"c70e4c82de8d2dbd78948603489ed46a97974541d015f1b49070e0f34cae135f"} Nov 01 16:24:08.121463 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:08.121432 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79acdb79-6330-4958-a12d-d42d9d5c8d4e-kube-api-access-x7lj4" pod "79acdb79-6330-4958-a12d-d42d9d5c8d4e" (UID: "79acdb79-6330-4958-a12d-d42d9d5c8d4e"). InnerVolumeSpecName "kube-api-access-x7lj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:24:08.124656 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:08.123822 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" event={"ID":"cf9b2de9-f52e-4580-af9e-d77e06ee6e64","Type":"ContainerStarted","Data":"9012a4038cbd6b621e94b09b6f15ba48bac9d607395e4dd1937895ee00ecd6e2"} Nov 01 16:24:08.124656 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:08.123878 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" event={"ID":"cf9b2de9-f52e-4580-af9e-d77e06ee6e64","Type":"ContainerStarted","Data":"be586793e31a557aa9ae900a9c4d1267086d5b4ba93db804210484bab9831ec4"} Nov 01 16:24:08.126866 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:08.126842 4363 generic.go:356] "Generic (PLEG): container finished" podID="79acdb79-6330-4958-a12d-d42d9d5c8d4e" containerID="95e8035c38cc902452b8d1cf36881d5003e298ef2e9fc05a55fe25f66db2073e" exitCode=0 Nov 01 16:24:08.127178 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:08.127154 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e044c-668c79d887-78jg4" Nov 01 16:24:08.127299 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:08.127275 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e044c-668c79d887-78jg4" event={"ID":"79acdb79-6330-4958-a12d-d42d9d5c8d4e","Type":"ContainerDied","Data":"95e8035c38cc902452b8d1cf36881d5003e298ef2e9fc05a55fe25f66db2073e"} Nov 01 16:24:08.127375 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:08.127336 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e044c-668c79d887-78jg4" event={"ID":"79acdb79-6330-4958-a12d-d42d9d5c8d4e","Type":"ContainerDied","Data":"c5723460f035dcd5c55693c981dd5f32923a1fa1150ec2c4a9e6516510286742"} Nov 01 16:24:08.127375 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:08.127370 4363 scope.go:122] "RemoveContainer" containerID="95e8035c38cc902452b8d1cf36881d5003e298ef2e9fc05a55fe25f66db2073e" Nov 01 16:24:08.140252 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:08.140162 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" podStartSLOduration=2.140146371 podStartE2EDuration="2.140146371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:24:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:24:08.135653242 +0000 UTC m=+2181.467390891" watchObservedRunningTime="2026-11-01 16:24:08.140146371 +0000 UTC m=+2181.471884021" Nov 01 16:24:08.148468 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:08.148445 4363 scope.go:122] "RemoveContainer" containerID="95e8035c38cc902452b8d1cf36881d5003e298ef2e9fc05a55fe25f66db2073e" Nov 01 16:24:08.148878 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:24:08.148850 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95e8035c38cc902452b8d1cf36881d5003e298ef2e9fc05a55fe25f66db2073e\": container with ID starting with 95e8035c38cc902452b8d1cf36881d5003e298ef2e9fc05a55fe25f66db2073e not found: ID does not exist" containerID="95e8035c38cc902452b8d1cf36881d5003e298ef2e9fc05a55fe25f66db2073e" Nov 01 16:24:08.148951 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:08.148893 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95e8035c38cc902452b8d1cf36881d5003e298ef2e9fc05a55fe25f66db2073e"} err="failed to get container status \"95e8035c38cc902452b8d1cf36881d5003e298ef2e9fc05a55fe25f66db2073e\": rpc error: code = NotFound desc = could not find container \"95e8035c38cc902452b8d1cf36881d5003e298ef2e9fc05a55fe25f66db2073e\": container with ID starting with 95e8035c38cc902452b8d1cf36881d5003e298ef2e9fc05a55fe25f66db2073e not found: ID does not exist" Nov 01 16:24:08.160085 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:08.159465 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" podStartSLOduration=1.159447375 podStartE2EDuration="1.159447375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:24:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:24:08.157763019 +0000 UTC m=+2181.489500679" watchObservedRunningTime="2026-11-01 16:24:08.159447375 +0000 UTC m=+2181.491185027" Nov 01 16:24:08.174787 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:08.174758 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9e044c-668c79d887-78jg4"] Nov 01 16:24:08.180548 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:08.180519 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9e044c-668c79d887-78jg4"] Nov 01 16:24:08.220375 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:08.219697 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7lj4\" (UniqueName: \"kubernetes.io/projected/79acdb79-6330-4958-a12d-d42d9d5c8d4e-kube-api-access-x7lj4\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:08.220375 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:08.219726 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79acdb79-6330-4958-a12d-d42d9d5c8d4e-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:08.483864 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:08.483832 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" Nov 01 16:24:08.485384 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:08.485354 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Nov 01 16:24:08.485535 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:08.485414 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" podUID="fcd61588-08cf-49c1-9dab-febc5c612ca8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Nov 01 16:24:08.500781 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:08.500753 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" Nov 01 16:24:08.502274 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:08.502246 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Nov 01 16:24:08.502369 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:08.502300 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" podUID="cf9b2de9-f52e-4580-af9e-d77e06ee6e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Nov 01 16:24:08.836117 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:08.835280 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79acdb79-6330-4958-a12d-d42d9d5c8d4e" path="/var/lib/kubelet/pods/79acdb79-6330-4958-a12d-d42d9d5c8d4e/volumes" Nov 01 16:24:09.491077 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:09.491036 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Nov 01 16:24:09.491387 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:09.491356 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" podUID="fcd61588-08cf-49c1-9dab-febc5c612ca8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Nov 01 16:24:09.504051 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:09.503998 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Nov 01 16:24:09.504326 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:09.504296 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" podUID="cf9b2de9-f52e-4580-af9e-d77e06ee6e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Nov 01 16:24:10.520043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:10.512159 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Nov 01 16:24:10.520043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:10.512223 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" podUID="cf9b2de9-f52e-4580-af9e-d77e06ee6e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Nov 01 16:24:10.520043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:10.512705 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Nov 01 16:24:10.520043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:10.512748 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" podUID="fcd61588-08cf-49c1-9dab-febc5c612ca8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Nov 01 16:24:11.485416 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:11.485379 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Nov 01 16:24:11.485720 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:11.485693 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" podUID="fcd61588-08cf-49c1-9dab-febc5c612ca8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Nov 01 16:24:11.536200 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:11.535167 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Nov 01 16:24:11.536200 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:11.535232 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" podUID="cf9b2de9-f52e-4580-af9e-d77e06ee6e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Nov 01 16:24:12.354826 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:12.354713 4363 generic.go:356] "Generic (PLEG): container finished" podID="22eae6cc-4752-4174-a0f0-573adc82e32b" containerID="931a83ee2d5ef1d73bdeec852b45c33a754b9fe3ca32df298f4970f18e95938c" exitCode=0 Nov 01 16:24:12.355266 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:12.355238 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" event={"ID":"22eae6cc-4752-4174-a0f0-573adc82e32b","Type":"ContainerDied","Data":"931a83ee2d5ef1d73bdeec852b45c33a754b9fe3ca32df298f4970f18e95938c"} Nov 01 16:24:12.355550 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:12.355533 4363 scope.go:122] "RemoveContainer" containerID="9b3bbf5f34103ac74f32d032cca159897c7f64f6dff7ccea3bd6567ccf289c19" Nov 01 16:24:12.494042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:12.491395 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Nov 01 16:24:12.494042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:12.491453 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" podUID="fcd61588-08cf-49c1-9dab-febc5c612ca8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Nov 01 16:24:12.541117 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:12.533357 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Nov 01 16:24:12.546640 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:12.545248 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" podUID="cf9b2de9-f52e-4580-af9e-d77e06ee6e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Nov 01 16:24:12.645115 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:12.645035 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" Nov 01 16:24:12.652751 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:12.652722 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22eae6cc-4752-4174-a0f0-573adc82e32b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22eae6cc-4752-4174-a0f0-573adc82e32b-credential-socket\") pod \"22eae6cc-4752-4174-a0f0-573adc82e32b\" (UID: \"22eae6cc-4752-4174-a0f0-573adc82e32b\") " Nov 01 16:24:12.652942 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:12.652758 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22eae6cc-4752-4174-a0f0-573adc82e32b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22eae6cc-4752-4174-a0f0-573adc82e32b-istiod-ca-cert\") pod \"22eae6cc-4752-4174-a0f0-573adc82e32b\" (UID: \"22eae6cc-4752-4174-a0f0-573adc82e32b\") " Nov 01 16:24:12.652942 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:12.652809 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22eae6cc-4752-4174-a0f0-573adc82e32b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22eae6cc-4752-4174-a0f0-573adc82e32b-istio-envoy\") pod \"22eae6cc-4752-4174-a0f0-573adc82e32b\" (UID: \"22eae6cc-4752-4174-a0f0-573adc82e32b\") " Nov 01 16:24:12.652942 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:12.652873 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22eae6cc-4752-4174-a0f0-573adc82e32b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22eae6cc-4752-4174-a0f0-573adc82e32b-istio-podinfo\") pod \"22eae6cc-4752-4174-a0f0-573adc82e32b\" (UID: \"22eae6cc-4752-4174-a0f0-573adc82e32b\") " Nov 01 16:24:12.652942 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:12.652910 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22eae6cc-4752-4174-a0f0-573adc82e32b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22eae6cc-4752-4174-a0f0-573adc82e32b-istio-data\") pod \"22eae6cc-4752-4174-a0f0-573adc82e32b\" (UID: \"22eae6cc-4752-4174-a0f0-573adc82e32b\") " Nov 01 16:24:12.653202 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:12.652984 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22eae6cc-4752-4174-a0f0-573adc82e32b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22eae6cc-4752-4174-a0f0-573adc82e32b-workload-socket\") pod \"22eae6cc-4752-4174-a0f0-573adc82e32b\" (UID: \"22eae6cc-4752-4174-a0f0-573adc82e32b\") " Nov 01 16:24:12.653202 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:12.653057 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22eae6cc-4752-4174-a0f0-573adc82e32b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22eae6cc-4752-4174-a0f0-573adc82e32b-workload-certs\") pod \"22eae6cc-4752-4174-a0f0-573adc82e32b\" (UID: \"22eae6cc-4752-4174-a0f0-573adc82e32b\") " Nov 01 16:24:12.653202 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:12.653090 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22eae6cc-4752-4174-a0f0-573adc82e32b-credential-socket" pod "22eae6cc-4752-4174-a0f0-573adc82e32b" (UID: "22eae6cc-4752-4174-a0f0-573adc82e32b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:24:12.653202 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:12.653162 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22eae6cc-4752-4174-a0f0-573adc82e32b-istio-token\" (UniqueName: \"kubernetes.io/projected/22eae6cc-4752-4174-a0f0-573adc82e32b-istio-token\") pod \"22eae6cc-4752-4174-a0f0-573adc82e32b\" (UID: \"22eae6cc-4752-4174-a0f0-573adc82e32b\") " Nov 01 16:24:12.653430 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:12.653225 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22eae6cc-4752-4174-a0f0-573adc82e32b-istiod-ca-cert" pod "22eae6cc-4752-4174-a0f0-573adc82e32b" (UID: "22eae6cc-4752-4174-a0f0-573adc82e32b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:24:12.653430 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:12.653360 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22eae6cc-4752-4174-a0f0-573adc82e32b-kube-api-access-ttxrr\" (UniqueName: \"kubernetes.io/projected/22eae6cc-4752-4174-a0f0-573adc82e32b-kube-api-access-ttxrr\") pod \"22eae6cc-4752-4174-a0f0-573adc82e32b\" (UID: \"22eae6cc-4752-4174-a0f0-573adc82e32b\") " Nov 01 16:24:12.653648 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:12.653624 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22eae6cc-4752-4174-a0f0-573adc82e32b-workload-socket" pod "22eae6cc-4752-4174-a0f0-573adc82e32b" (UID: "22eae6cc-4752-4174-a0f0-573adc82e32b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:24:12.653840 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:12.653819 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22eae6cc-4752-4174-a0f0-573adc82e32b-istio-data" pod "22eae6cc-4752-4174-a0f0-573adc82e32b" (UID: "22eae6cc-4752-4174-a0f0-573adc82e32b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:24:12.654005 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:12.653987 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22eae6cc-4752-4174-a0f0-573adc82e32b-workload-certs" pod "22eae6cc-4752-4174-a0f0-573adc82e32b" (UID: "22eae6cc-4752-4174-a0f0-573adc82e32b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:24:12.654149 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:12.654132 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22eae6cc-4752-4174-a0f0-573adc82e32b-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:12.654232 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:12.654150 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22eae6cc-4752-4174-a0f0-573adc82e32b-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:12.654232 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:12.654159 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22eae6cc-4752-4174-a0f0-573adc82e32b-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:12.654232 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:12.654170 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22eae6cc-4752-4174-a0f0-573adc82e32b-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:12.654232 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:12.654182 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22eae6cc-4752-4174-a0f0-573adc82e32b-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:12.656645 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:12.656604 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22eae6cc-4752-4174-a0f0-573adc82e32b-istio-envoy" pod "22eae6cc-4752-4174-a0f0-573adc82e32b" (UID: "22eae6cc-4752-4174-a0f0-573adc82e32b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:24:12.656764 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:12.656729 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22eae6cc-4752-4174-a0f0-573adc82e32b-istio-token" pod "22eae6cc-4752-4174-a0f0-573adc82e32b" (UID: "22eae6cc-4752-4174-a0f0-573adc82e32b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:24:12.656855 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:12.656833 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22eae6cc-4752-4174-a0f0-573adc82e32b-kube-api-access-ttxrr" pod "22eae6cc-4752-4174-a0f0-573adc82e32b" (UID: "22eae6cc-4752-4174-a0f0-573adc82e32b"). InnerVolumeSpecName "kube-api-access-ttxrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:24:12.658767 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:12.658740 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22eae6cc-4752-4174-a0f0-573adc82e32b-istio-podinfo" pod "22eae6cc-4752-4174-a0f0-573adc82e32b" (UID: "22eae6cc-4752-4174-a0f0-573adc82e32b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:24:12.794120 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:12.786461 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22eae6cc-4752-4174-a0f0-573adc82e32b-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:12.794373 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:12.794347 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22eae6cc-4752-4174-a0f0-573adc82e32b-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:12.794529 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:12.794509 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttxrr\" (UniqueName: \"kubernetes.io/projected/22eae6cc-4752-4174-a0f0-573adc82e32b-kube-api-access-ttxrr\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:12.794636 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:12.794623 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22eae6cc-4752-4174-a0f0-573adc82e32b-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:13.364584 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:13.364547 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" Nov 01 16:24:13.364803 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:13.364584 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl" event={"ID":"22eae6cc-4752-4174-a0f0-573adc82e32b","Type":"ContainerDied","Data":"71e289895986bc067ad57e4014d4df29022d4a01dd7ed50ca9629758786d03fd"} Nov 01 16:24:13.364803 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:13.364643 4363 scope.go:122] "RemoveContainer" containerID="931a83ee2d5ef1d73bdeec852b45c33a754b9fe3ca32df298f4970f18e95938c" Nov 01 16:24:13.444042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:13.431075 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl"] Nov 01 16:24:13.448268 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:13.448243 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-4zmbl"] Nov 01 16:24:13.486147 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:13.486116 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Nov 01 16:24:13.486394 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:13.486365 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" podUID="fcd61588-08cf-49c1-9dab-febc5c612ca8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Nov 01 16:24:13.551127 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:13.537412 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Nov 01 16:24:13.551127 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:13.537568 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" podUID="cf9b2de9-f52e-4580-af9e-d77e06ee6e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Nov 01 16:24:14.484536 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:14.484259 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Nov 01 16:24:14.484536 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:14.484321 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" podUID="fcd61588-08cf-49c1-9dab-febc5c612ca8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Nov 01 16:24:14.502170 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:14.502103 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Nov 01 16:24:14.502356 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:14.502214 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" podUID="cf9b2de9-f52e-4580-af9e-d77e06ee6e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Nov 01 16:24:14.809090 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:14.808920 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22eae6cc-4752-4174-a0f0-573adc82e32b" path="/var/lib/kubelet/pods/22eae6cc-4752-4174-a0f0-573adc82e32b/volumes" Nov 01 16:24:15.490327 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:15.490295 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" Nov 01 16:24:15.490843 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:15.490816 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" Nov 01 16:24:15.491724 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:15.491701 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" Nov 01 16:24:15.506282 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:15.506249 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" Nov 01 16:24:15.506657 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:15.506639 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" Nov 01 16:24:15.507514 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:15.507493 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" Nov 01 16:24:18.312114 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:24:18.312066 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:24:30.707189 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:30.707151 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl"] Nov 01 16:24:30.707722 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:30.707402 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" podUID="46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e" containerName="istio-proxy" containerID="cri-o://e4b4be64281d6c8c85fa6b77e79170b4da2abada82b76cf8e6ff679c6cfa106c" gracePeriod=30 Nov 01 16:24:32.061823 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.061792 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn"] Nov 01 16:24:32.062655 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.062636 4363 cpu_manager.go:414] "containerMap: removing container" podUID="22eae6cc-4752-4174-a0f0-573adc82e32b" containerName="istio-proxy" Nov 01 16:24:32.062655 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.062654 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22eae6cc-4752-4174-a0f0-573adc82e32b" containerName="istio-proxy" Nov 01 16:24:32.062837 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.062689 4363 cpu_manager.go:414] "containerMap: removing container" podUID="22eae6cc-4752-4174-a0f0-573adc82e32b" containerName="istio-proxy" Nov 01 16:24:32.062837 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.062697 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22eae6cc-4752-4174-a0f0-573adc82e32b" containerName="istio-proxy" Nov 01 16:24:32.062837 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.062728 4363 cpu_manager.go:414] "containerMap: removing container" podUID="79acdb79-6330-4958-a12d-d42d9d5c8d4e" containerName="payload-pre-processing" Nov 01 16:24:32.062837 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.062733 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79acdb79-6330-4958-a12d-d42d9d5c8d4e" containerName="payload-pre-processing" Nov 01 16:24:32.063063 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.062863 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="22eae6cc-4752-4174-a0f0-573adc82e32b" containerName="istio-proxy" Nov 01 16:24:32.063063 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.062879 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="22eae6cc-4752-4174-a0f0-573adc82e32b" containerName="istio-proxy" Nov 01 16:24:32.063063 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.062899 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="79acdb79-6330-4958-a12d-d42d9d5c8d4e" containerName="payload-pre-processing" Nov 01 16:24:32.079620 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.079591 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" Nov 01 16:24:32.087473 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.087442 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-e1e630-openshift-default-dockercfg-ndttc\"" Nov 01 16:24:32.118262 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.118107 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn"] Nov 01 16:24:32.146414 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.146372 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a4c04ae-eac9-46b2-b5fb-b36585723722-istio-envoy\") pod \"e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn\" (UID: \"9a4c04ae-eac9-46b2-b5fb-b36585723722\") " pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" Nov 01 16:24:32.146584 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.146568 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a4c04ae-eac9-46b2-b5fb-b36585723722-istio-token\") pod \"e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn\" (UID: \"9a4c04ae-eac9-46b2-b5fb-b36585723722\") " pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" Nov 01 16:24:32.146641 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.146602 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpfs5\" (UniqueName: \"kubernetes.io/projected/9a4c04ae-eac9-46b2-b5fb-b36585723722-kube-api-access-wpfs5\") pod \"e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn\" (UID: \"9a4c04ae-eac9-46b2-b5fb-b36585723722\") " pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" Nov 01 16:24:32.146691 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.146639 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a4c04ae-eac9-46b2-b5fb-b36585723722-istio-data\") pod \"e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn\" (UID: \"9a4c04ae-eac9-46b2-b5fb-b36585723722\") " pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" Nov 01 16:24:32.146738 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.146725 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a4c04ae-eac9-46b2-b5fb-b36585723722-credential-socket\") pod \"e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn\" (UID: \"9a4c04ae-eac9-46b2-b5fb-b36585723722\") " pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" Nov 01 16:24:32.146785 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.146773 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a4c04ae-eac9-46b2-b5fb-b36585723722-workload-certs\") pod \"e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn\" (UID: \"9a4c04ae-eac9-46b2-b5fb-b36585723722\") " pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" Nov 01 16:24:32.146822 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.146814 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a4c04ae-eac9-46b2-b5fb-b36585723722-istio-podinfo\") pod \"e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn\" (UID: \"9a4c04ae-eac9-46b2-b5fb-b36585723722\") " pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" Nov 01 16:24:32.146872 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.146855 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a4c04ae-eac9-46b2-b5fb-b36585723722-workload-socket\") pod \"e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn\" (UID: \"9a4c04ae-eac9-46b2-b5fb-b36585723722\") " pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" Nov 01 16:24:32.146906 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.146873 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a4c04ae-eac9-46b2-b5fb-b36585723722-istiod-ca-cert\") pod \"e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn\" (UID: \"9a4c04ae-eac9-46b2-b5fb-b36585723722\") " pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" Nov 01 16:24:32.258100 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.258062 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a4c04ae-eac9-46b2-b5fb-b36585723722-istio-token\") pod \"e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn\" (UID: \"9a4c04ae-eac9-46b2-b5fb-b36585723722\") " pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" Nov 01 16:24:32.278046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.258405 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpfs5\" (UniqueName: \"kubernetes.io/projected/9a4c04ae-eac9-46b2-b5fb-b36585723722-kube-api-access-wpfs5\") pod \"e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn\" (UID: \"9a4c04ae-eac9-46b2-b5fb-b36585723722\") " pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" Nov 01 16:24:32.278403 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.278373 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a4c04ae-eac9-46b2-b5fb-b36585723722-istio-data\") pod \"e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn\" (UID: \"9a4c04ae-eac9-46b2-b5fb-b36585723722\") " pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" Nov 01 16:24:32.278680 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.278662 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a4c04ae-eac9-46b2-b5fb-b36585723722-credential-socket\") pod \"e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn\" (UID: \"9a4c04ae-eac9-46b2-b5fb-b36585723722\") " pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" Nov 01 16:24:32.278875 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.278857 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a4c04ae-eac9-46b2-b5fb-b36585723722-workload-certs\") pod \"e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn\" (UID: \"9a4c04ae-eac9-46b2-b5fb-b36585723722\") " pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" Nov 01 16:24:32.279072 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.279054 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a4c04ae-eac9-46b2-b5fb-b36585723722-istio-podinfo\") pod \"e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn\" (UID: \"9a4c04ae-eac9-46b2-b5fb-b36585723722\") " pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" Nov 01 16:24:32.279263 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.279246 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a4c04ae-eac9-46b2-b5fb-b36585723722-workload-socket\") pod \"e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn\" (UID: \"9a4c04ae-eac9-46b2-b5fb-b36585723722\") " pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" Nov 01 16:24:32.279387 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.279372 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a4c04ae-eac9-46b2-b5fb-b36585723722-istiod-ca-cert\") pod \"e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn\" (UID: \"9a4c04ae-eac9-46b2-b5fb-b36585723722\") " pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" Nov 01 16:24:32.279510 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.279496 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a4c04ae-eac9-46b2-b5fb-b36585723722-istio-envoy\") pod \"e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn\" (UID: \"9a4c04ae-eac9-46b2-b5fb-b36585723722\") " pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" Nov 01 16:24:32.294094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.280812 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a4c04ae-eac9-46b2-b5fb-b36585723722-workload-certs\") pod \"e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn\" (UID: \"9a4c04ae-eac9-46b2-b5fb-b36585723722\") " pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" Nov 01 16:24:32.294353 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.282645 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a4c04ae-eac9-46b2-b5fb-b36585723722-istio-data\") pod \"e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn\" (UID: \"9a4c04ae-eac9-46b2-b5fb-b36585723722\") " pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" Nov 01 16:24:32.294481 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.282879 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a4c04ae-eac9-46b2-b5fb-b36585723722-credential-socket\") pod \"e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn\" (UID: \"9a4c04ae-eac9-46b2-b5fb-b36585723722\") " pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" Nov 01 16:24:32.294610 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.283164 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a4c04ae-eac9-46b2-b5fb-b36585723722-workload-socket\") pod \"e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn\" (UID: \"9a4c04ae-eac9-46b2-b5fb-b36585723722\") " pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" Nov 01 16:24:32.295634 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.295606 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a4c04ae-eac9-46b2-b5fb-b36585723722-istiod-ca-cert\") pod \"e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn\" (UID: \"9a4c04ae-eac9-46b2-b5fb-b36585723722\") " pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" Nov 01 16:24:32.328270 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.328178 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a4c04ae-eac9-46b2-b5fb-b36585723722-istio-envoy\") pod \"e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn\" (UID: \"9a4c04ae-eac9-46b2-b5fb-b36585723722\") " pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" Nov 01 16:24:32.339044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.331276 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a4c04ae-eac9-46b2-b5fb-b36585723722-istio-token\") pod \"e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn\" (UID: \"9a4c04ae-eac9-46b2-b5fb-b36585723722\") " pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" Nov 01 16:24:32.339044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.331481 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpfs5\" (UniqueName: \"kubernetes.io/projected/9a4c04ae-eac9-46b2-b5fb-b36585723722-kube-api-access-wpfs5\") pod \"e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn\" (UID: \"9a4c04ae-eac9-46b2-b5fb-b36585723722\") " pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" Nov 01 16:24:32.339044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.333505 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a4c04ae-eac9-46b2-b5fb-b36585723722-istio-podinfo\") pod \"e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn\" (UID: \"9a4c04ae-eac9-46b2-b5fb-b36585723722\") " pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" Nov 01 16:24:32.424122 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.420401 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" Nov 01 16:24:32.897562 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:24:32.896839 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a4c04ae_eac9_46b2_b5fb_b36585723722.slice/crio-1d49f97fd7b2941c4fc39580ddd3b4a7cb95abae3194fec1693cd9e9a8da1877 WatchSource:0}: Error finding container 1d49f97fd7b2941c4fc39580ddd3b4a7cb95abae3194fec1693cd9e9a8da1877: Status 404 returned error can't find the container with id 1d49f97fd7b2941c4fc39580ddd3b4a7cb95abae3194fec1693cd9e9a8da1877 Nov 01 16:24:32.898137 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.898111 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn"] Nov 01 16:24:32.902094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.902058 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:24:32.902199 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.902139 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:24:32.902199 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:32.902183 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:24:33.784874 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:33.784841 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" event={"ID":"9a4c04ae-eac9-46b2-b5fb-b36585723722","Type":"ContainerStarted","Data":"02be29b57b17261b3ec54fb84642abd5c33b125e81098185a462576cbf9b4c9b"} Nov 01 16:24:33.785430 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:33.784899 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" event={"ID":"9a4c04ae-eac9-46b2-b5fb-b36585723722","Type":"ContainerStarted","Data":"1d49f97fd7b2941c4fc39580ddd3b4a7cb95abae3194fec1693cd9e9a8da1877"} Nov 01 16:24:33.818101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:33.818051 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" podStartSLOduration=1.8180095569999999 podStartE2EDuration="1.818009557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:24:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:24:33.816439892 +0000 UTC m=+2207.148177579" watchObservedRunningTime="2026-11-01 16:24:33.818009557 +0000 UTC m=+2207.149747207" Nov 01 16:24:34.421346 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:34.421312 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" Nov 01 16:24:34.427502 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:34.427473 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" Nov 01 16:24:34.805556 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:34.805478 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" Nov 01 16:24:34.805945 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:34.805575 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" Nov 01 16:24:36.028627 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:36.028598 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" Nov 01 16:24:36.163048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:36.158440 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-kube-api-access-b8xph\" (UniqueName: \"kubernetes.io/projected/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-kube-api-access-b8xph\") pod \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\" (UID: \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\") " Nov 01 16:24:36.163048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:36.158544 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-istio-envoy\") pod \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\" (UID: \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\") " Nov 01 16:24:36.163048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:36.158578 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-istio-podinfo\") pod \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\" (UID: \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\") " Nov 01 16:24:36.163048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:36.158618 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-workload-socket\") pod \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\" (UID: \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\") " Nov 01 16:24:36.163048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:36.158660 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-istiod-ca-cert\") pod \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\" (UID: \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\") " Nov 01 16:24:36.171099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:36.167558 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-istiod-ca-cert" pod "46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e" (UID: "46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:24:36.197046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:36.196258 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-istio-data\") pod \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\" (UID: \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\") " Nov 01 16:24:36.197046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:36.196439 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-workload-certs\") pod \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\" (UID: \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\") " Nov 01 16:24:36.197046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:36.196552 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-credential-socket\") pod \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\" (UID: \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\") " Nov 01 16:24:36.197046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:36.196697 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-istio-token\" (UniqueName: \"kubernetes.io/projected/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-istio-token\") pod \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\" (UID: \"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e\") " Nov 01 16:24:36.215163 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:36.197816 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:36.215163 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:36.214788 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-workload-socket" pod "46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e" (UID: "46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:24:36.224258 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:36.224212 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-workload-certs" pod "46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e" (UID: "46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:24:36.224526 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:36.224503 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-istio-data" pod "46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e" (UID: "46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:24:36.224768 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:36.224747 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-credential-socket" pod "46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e" (UID: "46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:24:36.257178 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:36.257089 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-kube-api-access-b8xph" pod "46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e" (UID: "46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e"). InnerVolumeSpecName "kube-api-access-b8xph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:24:36.260338 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:36.260302 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-istio-envoy" pod "46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e" (UID: "46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:24:36.261803 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:36.261760 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-istio-podinfo" pod "46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e" (UID: "46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:24:36.263566 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:36.263537 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-istio-token" pod "46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e" (UID: "46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:24:36.299923 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:36.299888 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:36.299923 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:36.299917 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:36.299923 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:36.299928 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:36.300219 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:36.299937 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8xph\" (UniqueName: \"kubernetes.io/projected/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-kube-api-access-b8xph\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:36.300219 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:36.299946 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:36.300219 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:36.299955 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:36.300219 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:36.299963 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:36.300219 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:36.299971 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:36.817545 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:36.817518 4363 generic.go:356] "Generic (PLEG): container finished" podID="46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e" containerID="e4b4be64281d6c8c85fa6b77e79170b4da2abada82b76cf8e6ff679c6cfa106c" exitCode=0 Nov 01 16:24:36.817675 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:36.817579 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" Nov 01 16:24:36.817675 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:36.817599 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" event={"ID":"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e","Type":"ContainerDied","Data":"e4b4be64281d6c8c85fa6b77e79170b4da2abada82b76cf8e6ff679c6cfa106c"} Nov 01 16:24:36.817675 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:36.817668 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl" event={"ID":"46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e","Type":"ContainerDied","Data":"6a2903101daf1c515b92be3331cb903d2a1eba896ba1f73b3961d892693e72ac"} Nov 01 16:24:36.817808 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:36.817699 4363 scope.go:122] "RemoveContainer" containerID="e4b4be64281d6c8c85fa6b77e79170b4da2abada82b76cf8e6ff679c6cfa106c" Nov 01 16:24:36.829594 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:36.829565 4363 scope.go:122] "RemoveContainer" containerID="ec7d89b3ee03862754bd3ca26d3f980319834332ecc5d21c73ca9f51fb7ebc85" Nov 01 16:24:36.840695 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:36.840358 4363 scope.go:122] "RemoveContainer" containerID="e4b4be64281d6c8c85fa6b77e79170b4da2abada82b76cf8e6ff679c6cfa106c" Nov 01 16:24:36.840780 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:36.840710 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl"] Nov 01 16:24:36.840857 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:24:36.840834 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4b4be64281d6c8c85fa6b77e79170b4da2abada82b76cf8e6ff679c6cfa106c\": container with ID starting with e4b4be64281d6c8c85fa6b77e79170b4da2abada82b76cf8e6ff679c6cfa106c not found: ID does not exist" containerID="e4b4be64281d6c8c85fa6b77e79170b4da2abada82b76cf8e6ff679c6cfa106c" Nov 01 16:24:36.840897 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:36.840866 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4b4be64281d6c8c85fa6b77e79170b4da2abada82b76cf8e6ff679c6cfa106c"} err="failed to get container status \"e4b4be64281d6c8c85fa6b77e79170b4da2abada82b76cf8e6ff679c6cfa106c\": rpc error: code = NotFound desc = could not find container \"e4b4be64281d6c8c85fa6b77e79170b4da2abada82b76cf8e6ff679c6cfa106c\": container with ID starting with e4b4be64281d6c8c85fa6b77e79170b4da2abada82b76cf8e6ff679c6cfa106c not found: ID does not exist" Nov 01 16:24:36.840897 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:36.840884 4363 scope.go:122] "RemoveContainer" containerID="ec7d89b3ee03862754bd3ca26d3f980319834332ecc5d21c73ca9f51fb7ebc85" Nov 01 16:24:36.841188 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:24:36.841172 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec7d89b3ee03862754bd3ca26d3f980319834332ecc5d21c73ca9f51fb7ebc85\": container with ID starting with ec7d89b3ee03862754bd3ca26d3f980319834332ecc5d21c73ca9f51fb7ebc85 not found: ID does not exist" containerID="ec7d89b3ee03862754bd3ca26d3f980319834332ecc5d21c73ca9f51fb7ebc85" Nov 01 16:24:36.841269 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:36.841191 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec7d89b3ee03862754bd3ca26d3f980319834332ecc5d21c73ca9f51fb7ebc85"} err="failed to get container status \"ec7d89b3ee03862754bd3ca26d3f980319834332ecc5d21c73ca9f51fb7ebc85\": rpc error: code = NotFound desc = could not find container \"ec7d89b3ee03862754bd3ca26d3f980319834332ecc5d21c73ca9f51fb7ebc85\": container with ID starting with ec7d89b3ee03862754bd3ca26d3f980319834332ecc5d21c73ca9f51fb7ebc85 not found: ID does not exist" Nov 01 16:24:36.844989 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:36.844969 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ae79df-openshift-default-777549df6f-n7nfl"] Nov 01 16:24:38.800977 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:24:38.800946 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e" path="/var/lib/kubelet/pods/46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e/volumes" Nov 01 16:24:48.320590 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:24:48.320542 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:25:18.312388 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:25:18.312340 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:25:19.302986 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:19.302935 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4"] Nov 01 16:25:19.303968 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:19.303934 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" podUID="5eb560c5-c5f0-497b-a30d-fa3a44caae3d" containerName="istio-proxy" containerID="cri-o://35f9a6e42cb1d896a5c0ad69e76af50c30844e172d884b5b0e44037c6e61a4f9" gracePeriod=30 Nov 01 16:25:24.535849 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:24.535810 4363 generic.go:356] "Generic (PLEG): container finished" podID="5eb560c5-c5f0-497b-a30d-fa3a44caae3d" containerID="35f9a6e42cb1d896a5c0ad69e76af50c30844e172d884b5b0e44037c6e61a4f9" exitCode=0 Nov 01 16:25:24.536277 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:24.535884 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" event={"ID":"5eb560c5-c5f0-497b-a30d-fa3a44caae3d","Type":"ContainerDied","Data":"35f9a6e42cb1d896a5c0ad69e76af50c30844e172d884b5b0e44037c6e61a4f9"} Nov 01 16:25:25.230146 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:25.228241 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" Nov 01 16:25:25.257846 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:25.257786 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-istio-podinfo\") pod \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\" (UID: \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\") " Nov 01 16:25:25.258033 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:25.257937 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-workload-certs\") pod \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\" (UID: \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\") " Nov 01 16:25:25.258179 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:25.258158 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-workload-socket\") pod \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\" (UID: \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\") " Nov 01 16:25:25.258247 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:25.258233 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-istio-token\" (UniqueName: \"kubernetes.io/projected/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-istio-token\") pod \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\" (UID: \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\") " Nov 01 16:25:25.258309 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:25.258260 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-istio-data\") pod \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\" (UID: \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\") " Nov 01 16:25:25.258421 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:25.258405 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-istio-envoy\") pod \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\" (UID: \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\") " Nov 01 16:25:25.258480 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:25.258462 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-istiod-ca-cert\") pod \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\" (UID: \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\") " Nov 01 16:25:25.258554 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:25.258539 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-kube-api-access-7z4ck\" (UniqueName: \"kubernetes.io/projected/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-kube-api-access-7z4ck\") pod \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\" (UID: \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\") " Nov 01 16:25:25.258620 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:25.258582 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-credential-socket\") pod \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\" (UID: \"5eb560c5-c5f0-497b-a30d-fa3a44caae3d\") " Nov 01 16:25:25.259656 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:25.259628 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-credential-socket" pod "5eb560c5-c5f0-497b-a30d-fa3a44caae3d" (UID: "5eb560c5-c5f0-497b-a30d-fa3a44caae3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:25:25.259849 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:25.259828 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-istio-data" pod "5eb560c5-c5f0-497b-a30d-fa3a44caae3d" (UID: "5eb560c5-c5f0-497b-a30d-fa3a44caae3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:25:25.260357 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:25.260334 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-workload-certs" pod "5eb560c5-c5f0-497b-a30d-fa3a44caae3d" (UID: "5eb560c5-c5f0-497b-a30d-fa3a44caae3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:25:25.260508 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:25.260450 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-istiod-ca-cert" pod "5eb560c5-c5f0-497b-a30d-fa3a44caae3d" (UID: "5eb560c5-c5f0-497b-a30d-fa3a44caae3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:25:25.260741 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:25.260716 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-workload-socket" pod "5eb560c5-c5f0-497b-a30d-fa3a44caae3d" (UID: "5eb560c5-c5f0-497b-a30d-fa3a44caae3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:25:25.263203 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:25.262970 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-istio-token" pod "5eb560c5-c5f0-497b-a30d-fa3a44caae3d" (UID: "5eb560c5-c5f0-497b-a30d-fa3a44caae3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:25:25.263203 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:25.263174 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-kube-api-access-7z4ck" pod "5eb560c5-c5f0-497b-a30d-fa3a44caae3d" (UID: "5eb560c5-c5f0-497b-a30d-fa3a44caae3d"). InnerVolumeSpecName "kube-api-access-7z4ck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:25:25.263918 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:25.263897 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-istio-envoy" pod "5eb560c5-c5f0-497b-a30d-fa3a44caae3d" (UID: "5eb560c5-c5f0-497b-a30d-fa3a44caae3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:25:25.264061 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:25.263963 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-istio-podinfo" pod "5eb560c5-c5f0-497b-a30d-fa3a44caae3d" (UID: "5eb560c5-c5f0-497b-a30d-fa3a44caae3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:25:25.368173 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:25.362290 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:25:25.368173 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:25.362329 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:25:25.368173 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:25.362346 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:25:25.368173 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:25.362358 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:25:25.368173 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:25.362372 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:25:25.368173 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:25.362386 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z4ck\" (UniqueName: \"kubernetes.io/projected/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-kube-api-access-7z4ck\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:25:25.368173 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:25.362401 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:25:25.368173 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:25.362416 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:25:25.368173 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:25.362430 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb560c5-c5f0-497b-a30d-fa3a44caae3d-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:25:25.546232 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:25.546150 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" Nov 01 16:25:25.546674 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:25.546153 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" event={"ID":"5eb560c5-c5f0-497b-a30d-fa3a44caae3d","Type":"ContainerDied","Data":"bd121101c47340a0393e00cc2ca7fa796b702e6f9f1a1b643af9646f97032a3c"} Nov 01 16:25:25.546674 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:25.546294 4363 scope.go:122] "RemoveContainer" containerID="35f9a6e42cb1d896a5c0ad69e76af50c30844e172d884b5b0e44037c6e61a4f9" Nov 01 16:25:25.573759 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:25.573712 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4"] Nov 01 16:25:25.580392 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:25.580366 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4"] Nov 01 16:25:26.191170 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:26.191136 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dh7c4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:25:26.191370 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:26.191212 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dh7c4" podUID="5eb560c5-c5f0-497b-a30d-fa3a44caae3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:25:26.801640 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:26.801601 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eb560c5-c5f0-497b-a30d-fa3a44caae3d" path="/var/lib/kubelet/pods/5eb560c5-c5f0-497b-a30d-fa3a44caae3d/volumes" Nov 01 16:25:29.583198 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:29.583120 4363 generic.go:356] "Generic (PLEG): container finished" podID="9a4c04ae-eac9-46b2-b5fb-b36585723722" containerID="02be29b57b17261b3ec54fb84642abd5c33b125e81098185a462576cbf9b4c9b" exitCode=0 Nov 01 16:25:29.583719 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:29.583192 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" event={"ID":"9a4c04ae-eac9-46b2-b5fb-b36585723722","Type":"ContainerDied","Data":"02be29b57b17261b3ec54fb84642abd5c33b125e81098185a462576cbf9b4c9b"} Nov 01 16:25:29.583791 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:29.583740 4363 scope.go:122] "RemoveContainer" containerID="02be29b57b17261b3ec54fb84642abd5c33b125e81098185a462576cbf9b4c9b" Nov 01 16:25:29.585961 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:29.585927 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:25:29.586088 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:29.586034 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:25:29.586138 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:29.586087 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:25:30.641047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:30.632219 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" event={"ID":"9a4c04ae-eac9-46b2-b5fb-b36585723722","Type":"ContainerStarted","Data":"658d6ed056d3a59ace04f1b865ede340f35cf6a281e0637a358929aa22ab9038"} Nov 01 16:25:31.428067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:31.425550 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" Nov 01 16:25:31.428067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:31.425691 4363 patch_prober.go:29] interesting pod/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body= Nov 01 16:25:31.428067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:31.425742 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" podUID="9a4c04ae-eac9-46b2-b5fb-b36585723722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" Nov 01 16:25:32.421391 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:32.421356 4363 patch_prober.go:29] interesting pod/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body= Nov 01 16:25:32.421931 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:32.421411 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" podUID="9a4c04ae-eac9-46b2-b5fb-b36585723722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" Nov 01 16:25:33.421330 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:33.421296 4363 patch_prober.go:29] interesting pod/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body= Nov 01 16:25:33.421554 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:33.421357 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" podUID="9a4c04ae-eac9-46b2-b5fb-b36585723722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" Nov 01 16:25:34.421313 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:34.421277 4363 patch_prober.go:29] interesting pod/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body= Nov 01 16:25:34.421661 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:34.421631 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" podUID="9a4c04ae-eac9-46b2-b5fb-b36585723722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" Nov 01 16:25:34.835542 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:34.835453 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" Nov 01 16:25:35.422134 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:35.422091 4363 patch_prober.go:29] interesting pod/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body= Nov 01 16:25:35.422562 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:35.422161 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" podUID="9a4c04ae-eac9-46b2-b5fb-b36585723722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" Nov 01 16:25:36.445041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:36.436267 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" Nov 01 16:25:36.445041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:36.444223 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" Nov 01 16:25:38.884820 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:38.871296 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w"] Nov 01 16:25:38.884820 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:38.872560 4363 cpu_manager.go:414] "containerMap: removing container" podUID="46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e" containerName="istio-proxy" Nov 01 16:25:38.884820 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:38.872582 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e" containerName="istio-proxy" Nov 01 16:25:38.884820 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:38.872663 4363 cpu_manager.go:414] "containerMap: removing container" podUID="5eb560c5-c5f0-497b-a30d-fa3a44caae3d" containerName="istio-proxy" Nov 01 16:25:38.884820 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:38.872672 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eb560c5-c5f0-497b-a30d-fa3a44caae3d" containerName="istio-proxy" Nov 01 16:25:38.884820 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:38.872918 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eb560c5-c5f0-497b-a30d-fa3a44caae3d" containerName="istio-proxy" Nov 01 16:25:38.884820 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:38.872950 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e" containerName="istio-proxy" Nov 01 16:25:38.884820 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:38.872966 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e" containerName="istio-proxy" Nov 01 16:25:38.884820 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:38.879928 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" Nov 01 16:25:38.891099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:38.891074 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-2b94b5-openshift-default-dockercfg-fgvlz\"" Nov 01 16:25:38.896765 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:38.896736 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w"] Nov 01 16:25:38.943006 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:38.942965 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b5725f6-5b76-4300-9886-6a8c2556e6ae-workload-certs\") pod \"e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w\" (UID: \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\") " pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" Nov 01 16:25:38.943274 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:38.943074 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b5725f6-5b76-4300-9886-6a8c2556e6ae-istiod-ca-cert\") pod \"e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w\" (UID: \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\") " pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" Nov 01 16:25:38.943274 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:38.943116 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5725f6-5b76-4300-9886-6a8c2556e6ae-credential-socket\") pod \"e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w\" (UID: \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\") " pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" Nov 01 16:25:38.943274 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:38.943187 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b5725f6-5b76-4300-9886-6a8c2556e6ae-istio-envoy\") pod \"e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w\" (UID: \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\") " pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" Nov 01 16:25:38.943274 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:38.943261 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b5725f6-5b76-4300-9886-6a8c2556e6ae-istio-data\") pod \"e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w\" (UID: \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\") " pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" Nov 01 16:25:38.943495 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:38.943349 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pxgs\" (UniqueName: \"kubernetes.io/projected/3b5725f6-5b76-4300-9886-6a8c2556e6ae-kube-api-access-8pxgs\") pod \"e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w\" (UID: \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\") " pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" Nov 01 16:25:38.943553 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:38.943505 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5725f6-5b76-4300-9886-6a8c2556e6ae-workload-socket\") pod \"e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w\" (UID: \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\") " pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" Nov 01 16:25:38.943553 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:38.943546 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b5725f6-5b76-4300-9886-6a8c2556e6ae-istio-podinfo\") pod \"e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w\" (UID: \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\") " pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" Nov 01 16:25:38.943652 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:38.943586 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b5725f6-5b76-4300-9886-6a8c2556e6ae-istio-token\") pod \"e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w\" (UID: \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\") " pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" Nov 01 16:25:39.089042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:39.088562 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b5725f6-5b76-4300-9886-6a8c2556e6ae-istio-data\") pod \"e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w\" (UID: \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\") " pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" Nov 01 16:25:39.089042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:39.088705 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pxgs\" (UniqueName: \"kubernetes.io/projected/3b5725f6-5b76-4300-9886-6a8c2556e6ae-kube-api-access-8pxgs\") pod \"e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w\" (UID: \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\") " pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" Nov 01 16:25:39.089042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:39.088884 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5725f6-5b76-4300-9886-6a8c2556e6ae-workload-socket\") pod \"e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w\" (UID: \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\") " pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" Nov 01 16:25:39.089042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:39.088919 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b5725f6-5b76-4300-9886-6a8c2556e6ae-istio-podinfo\") pod \"e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w\" (UID: \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\") " pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" Nov 01 16:25:39.089042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:39.088952 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b5725f6-5b76-4300-9886-6a8c2556e6ae-istio-token\") pod \"e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w\" (UID: \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\") " pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" Nov 01 16:25:39.089042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:39.089040 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b5725f6-5b76-4300-9886-6a8c2556e6ae-workload-certs\") pod \"e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w\" (UID: \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\") " pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" Nov 01 16:25:39.089553 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:39.089104 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b5725f6-5b76-4300-9886-6a8c2556e6ae-istiod-ca-cert\") pod \"e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w\" (UID: \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\") " pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" Nov 01 16:25:39.089553 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:39.089136 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5725f6-5b76-4300-9886-6a8c2556e6ae-credential-socket\") pod \"e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w\" (UID: \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\") " pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" Nov 01 16:25:39.089553 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:39.089209 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b5725f6-5b76-4300-9886-6a8c2556e6ae-istio-envoy\") pod \"e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w\" (UID: \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\") " pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" Nov 01 16:25:39.183340 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:39.180481 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b5725f6-5b76-4300-9886-6a8c2556e6ae-istiod-ca-cert\") pod \"e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w\" (UID: \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\") " pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" Nov 01 16:25:39.183340 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:39.180975 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5725f6-5b76-4300-9886-6a8c2556e6ae-credential-socket\") pod \"e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w\" (UID: \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\") " pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" Nov 01 16:25:39.208041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:39.203358 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5725f6-5b76-4300-9886-6a8c2556e6ae-workload-socket\") pod \"e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w\" (UID: \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\") " pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" Nov 01 16:25:39.253039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:39.219428 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b5725f6-5b76-4300-9886-6a8c2556e6ae-istio-envoy\") pod \"e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w\" (UID: \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\") " pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" Nov 01 16:25:39.253039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:39.225647 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b5725f6-5b76-4300-9886-6a8c2556e6ae-workload-certs\") pod \"e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w\" (UID: \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\") " pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" Nov 01 16:25:39.253039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:39.231789 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b5725f6-5b76-4300-9886-6a8c2556e6ae-istio-data\") pod \"e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w\" (UID: \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\") " pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" Nov 01 16:25:39.303640 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:39.260866 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b5725f6-5b76-4300-9886-6a8c2556e6ae-istio-podinfo\") pod \"e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w\" (UID: \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\") " pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" Nov 01 16:25:39.364129 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:39.360731 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b5725f6-5b76-4300-9886-6a8c2556e6ae-istio-token\") pod \"e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w\" (UID: \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\") " pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" Nov 01 16:25:39.364129 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:39.361008 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pxgs\" (UniqueName: \"kubernetes.io/projected/3b5725f6-5b76-4300-9886-6a8c2556e6ae-kube-api-access-8pxgs\") pod \"e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w\" (UID: \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\") " pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" Nov 01 16:25:39.564034 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:39.561345 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" Nov 01 16:25:39.896685 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:39.896652 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w"] Nov 01 16:25:39.899205 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:25:39.899173 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b5725f6_5b76_4300_9886_6a8c2556e6ae.slice/crio-b572eaf6ae4167fb1e67c25c052812cbee10afe38edd43ee181f85d903228602 WatchSource:0}: Error finding container b572eaf6ae4167fb1e67c25c052812cbee10afe38edd43ee181f85d903228602: Status 404 returned error can't find the container with id b572eaf6ae4167fb1e67c25c052812cbee10afe38edd43ee181f85d903228602 Nov 01 16:25:39.903028 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:39.902981 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:25:39.903182 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:39.903103 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:25:39.903182 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:39.903149 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:25:40.746659 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:40.746622 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" event={"ID":"3b5725f6-5b76-4300-9886-6a8c2556e6ae","Type":"ContainerStarted","Data":"c4f9204e651009e25d691e291968fd2ba29f08a6db5f84e35abef7f4019f7a31"} Nov 01 16:25:40.746849 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:40.746688 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" event={"ID":"3b5725f6-5b76-4300-9886-6a8c2556e6ae","Type":"ContainerStarted","Data":"b572eaf6ae4167fb1e67c25c052812cbee10afe38edd43ee181f85d903228602"} Nov 01 16:25:40.768792 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:40.768739 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" podStartSLOduration=2.768723919 podStartE2EDuration="2.768723919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:25:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:25:40.765998802 +0000 UTC m=+2274.097736452" watchObservedRunningTime="2026-11-01 16:25:40.768723919 +0000 UTC m=+2274.100461570" Nov 01 16:25:41.562856 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:41.562818 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" Nov 01 16:25:41.568578 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:41.568553 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" Nov 01 16:25:41.754172 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:41.754137 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" Nov 01 16:25:41.755149 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:41.755123 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" Nov 01 16:25:46.061476 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:46.061434 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-5e2d03/auth-test-model-5e2d03-kserve-665774f749-75t2g"] Nov 01 16:25:48.311752 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:25:48.311712 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:25:53.616397 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:53.616359 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc"] Nov 01 16:25:53.617527 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:53.617499 4363 cpu_manager.go:414] "containerMap: removing container" podUID="46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e" containerName="istio-proxy" Nov 01 16:25:53.617703 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:53.617673 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46b2f8eb-f7ee-4cbc-a7db-d910ba142c9e" containerName="istio-proxy" Nov 01 16:25:53.625289 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:53.625265 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc"] Nov 01 16:25:53.625459 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:53.625360 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" Nov 01 16:25:53.629568 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:53.629541 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-104b6e-openshift-default-dockercfg-dklfp\"" Nov 01 16:25:53.709799 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:53.709750 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be90dd72-5635-4115-9826-26aaa46f70dc-workload-certs\") pod \"e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc\" (UID: \"be90dd72-5635-4115-9826-26aaa46f70dc\") " pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" Nov 01 16:25:53.710012 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:53.709903 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be90dd72-5635-4115-9826-26aaa46f70dc-istiod-ca-cert\") pod \"e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc\" (UID: \"be90dd72-5635-4115-9826-26aaa46f70dc\") " pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" Nov 01 16:25:53.710012 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:53.709984 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be90dd72-5635-4115-9826-26aaa46f70dc-istio-data\") pod \"e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc\" (UID: \"be90dd72-5635-4115-9826-26aaa46f70dc\") " pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" Nov 01 16:25:53.710183 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:53.710154 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be90dd72-5635-4115-9826-26aaa46f70dc-istio-envoy\") pod \"e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc\" (UID: \"be90dd72-5635-4115-9826-26aaa46f70dc\") " pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" Nov 01 16:25:53.710323 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:53.710297 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be90dd72-5635-4115-9826-26aaa46f70dc-istio-token\") pod \"e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc\" (UID: \"be90dd72-5635-4115-9826-26aaa46f70dc\") " pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" Nov 01 16:25:53.710405 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:53.710343 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5pnv\" (UniqueName: \"kubernetes.io/projected/be90dd72-5635-4115-9826-26aaa46f70dc-kube-api-access-d5pnv\") pod \"e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc\" (UID: \"be90dd72-5635-4115-9826-26aaa46f70dc\") " pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" Nov 01 16:25:53.710483 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:53.710461 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be90dd72-5635-4115-9826-26aaa46f70dc-workload-socket\") pod \"e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc\" (UID: \"be90dd72-5635-4115-9826-26aaa46f70dc\") " pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" Nov 01 16:25:53.710579 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:53.710565 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be90dd72-5635-4115-9826-26aaa46f70dc-istio-podinfo\") pod \"e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc\" (UID: \"be90dd72-5635-4115-9826-26aaa46f70dc\") " pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" Nov 01 16:25:53.710680 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:53.710657 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be90dd72-5635-4115-9826-26aaa46f70dc-credential-socket\") pod \"e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc\" (UID: \"be90dd72-5635-4115-9826-26aaa46f70dc\") " pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" Nov 01 16:25:53.825037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:53.816470 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be90dd72-5635-4115-9826-26aaa46f70dc-istio-token\") pod \"e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc\" (UID: \"be90dd72-5635-4115-9826-26aaa46f70dc\") " pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" Nov 01 16:25:53.825037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:53.816517 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5pnv\" (UniqueName: \"kubernetes.io/projected/be90dd72-5635-4115-9826-26aaa46f70dc-kube-api-access-d5pnv\") pod \"e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc\" (UID: \"be90dd72-5635-4115-9826-26aaa46f70dc\") " pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" Nov 01 16:25:53.825037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:53.816574 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be90dd72-5635-4115-9826-26aaa46f70dc-workload-socket\") pod \"e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc\" (UID: \"be90dd72-5635-4115-9826-26aaa46f70dc\") " pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" Nov 01 16:25:53.825037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:53.816667 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be90dd72-5635-4115-9826-26aaa46f70dc-istio-podinfo\") pod \"e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc\" (UID: \"be90dd72-5635-4115-9826-26aaa46f70dc\") " pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" Nov 01 16:25:53.825037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:53.816715 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be90dd72-5635-4115-9826-26aaa46f70dc-credential-socket\") pod \"e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc\" (UID: \"be90dd72-5635-4115-9826-26aaa46f70dc\") " pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" Nov 01 16:25:53.825037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:53.816849 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be90dd72-5635-4115-9826-26aaa46f70dc-workload-certs\") pod \"e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc\" (UID: \"be90dd72-5635-4115-9826-26aaa46f70dc\") " pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" Nov 01 16:25:53.825037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:53.816914 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be90dd72-5635-4115-9826-26aaa46f70dc-istiod-ca-cert\") pod \"e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc\" (UID: \"be90dd72-5635-4115-9826-26aaa46f70dc\") " pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" Nov 01 16:25:53.825037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:53.816969 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be90dd72-5635-4115-9826-26aaa46f70dc-istio-data\") pod \"e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc\" (UID: \"be90dd72-5635-4115-9826-26aaa46f70dc\") " pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" Nov 01 16:25:53.825037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:53.817077 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be90dd72-5635-4115-9826-26aaa46f70dc-istio-envoy\") pod \"e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc\" (UID: \"be90dd72-5635-4115-9826-26aaa46f70dc\") " pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" Nov 01 16:25:53.838729 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:53.832794 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be90dd72-5635-4115-9826-26aaa46f70dc-workload-socket\") pod \"e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc\" (UID: \"be90dd72-5635-4115-9826-26aaa46f70dc\") " pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" Nov 01 16:25:53.877920 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:53.846412 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be90dd72-5635-4115-9826-26aaa46f70dc-credential-socket\") pod \"e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc\" (UID: \"be90dd72-5635-4115-9826-26aaa46f70dc\") " pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" Nov 01 16:25:53.878155 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:53.847106 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be90dd72-5635-4115-9826-26aaa46f70dc-istiod-ca-cert\") pod \"e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc\" (UID: \"be90dd72-5635-4115-9826-26aaa46f70dc\") " pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" Nov 01 16:25:53.878276 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:53.847343 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be90dd72-5635-4115-9826-26aaa46f70dc-istio-data\") pod \"e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc\" (UID: \"be90dd72-5635-4115-9826-26aaa46f70dc\") " pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" Nov 01 16:25:53.878377 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:53.847611 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be90dd72-5635-4115-9826-26aaa46f70dc-workload-certs\") pod \"e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc\" (UID: \"be90dd72-5635-4115-9826-26aaa46f70dc\") " pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" Nov 01 16:25:53.892933 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:53.892192 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be90dd72-5635-4115-9826-26aaa46f70dc-istio-envoy\") pod \"e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc\" (UID: \"be90dd72-5635-4115-9826-26aaa46f70dc\") " pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" Nov 01 16:25:53.915947 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:53.893769 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5pnv\" (UniqueName: \"kubernetes.io/projected/be90dd72-5635-4115-9826-26aaa46f70dc-kube-api-access-d5pnv\") pod \"e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc\" (UID: \"be90dd72-5635-4115-9826-26aaa46f70dc\") " pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" Nov 01 16:25:53.915947 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:53.896411 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be90dd72-5635-4115-9826-26aaa46f70dc-istio-podinfo\") pod \"e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc\" (UID: \"be90dd72-5635-4115-9826-26aaa46f70dc\") " pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" Nov 01 16:25:53.915947 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:53.914543 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be90dd72-5635-4115-9826-26aaa46f70dc-istio-token\") pod \"e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc\" (UID: \"be90dd72-5635-4115-9826-26aaa46f70dc\") " pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" Nov 01 16:25:53.957038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:53.948198 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" Nov 01 16:25:54.439124 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:25:54.439094 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe90dd72_5635_4115_9826_26aaa46f70dc.slice/crio-cf34d5b0ecd84e5fb837332eefe63bc162e607386077d2434e4477c11feeb3b0 WatchSource:0}: Error finding container cf34d5b0ecd84e5fb837332eefe63bc162e607386077d2434e4477c11feeb3b0: Status 404 returned error can't find the container with id cf34d5b0ecd84e5fb837332eefe63bc162e607386077d2434e4477c11feeb3b0 Nov 01 16:25:54.440902 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:54.440877 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc"] Nov 01 16:25:54.442731 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:54.442698 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:25:54.442826 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:54.442762 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:25:54.442826 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:54.442792 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:25:55.111983 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:55.111934 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" event={"ID":"be90dd72-5635-4115-9826-26aaa46f70dc","Type":"ContainerStarted","Data":"61ea40fc745b3bb1015069215985f45711621cb794c7c73809efd6112fc793f0"} Nov 01 16:25:55.112434 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:55.112059 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" event={"ID":"be90dd72-5635-4115-9826-26aaa46f70dc","Type":"ContainerStarted","Data":"cf34d5b0ecd84e5fb837332eefe63bc162e607386077d2434e4477c11feeb3b0"} Nov 01 16:25:55.131935 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:55.131883 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" podStartSLOduration=2.131867565 podStartE2EDuration="2.131867565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:25:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:25:55.130757268 +0000 UTC m=+2288.462494917" watchObservedRunningTime="2026-11-01 16:25:55.131867565 +0000 UTC m=+2288.463605214" Nov 01 16:25:55.949005 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:55.948972 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" Nov 01 16:25:55.954961 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:55.954928 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" Nov 01 16:25:56.119998 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:56.119971 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" Nov 01 16:25:56.120909 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:25:56.120889 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" Nov 01 16:26:01.103847 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:01.103727 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-5e2d03/auth-test-model-5e2d03-kserve-665774f749-75t2g" podUID="e66518b6-c569-4afd-9464-ba934cccbcf7" containerName="main" containerID="cri-o://a31747fe9aba851c57ca451ab9361a562b2aa34e14e1a923aca1efa0ab59efd2" gracePeriod=45 Nov 01 16:26:01.187723 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:01.187681 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-5e2d03/auth-test-model-5e2d03-kserve-665774f749-75t2g" podUID="e66518b6-c569-4afd-9464-ba934cccbcf7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.204:8000/ready\": dial tcp 10.134.0.204:8000: connect: connection refused" Nov 01 16:26:01.511616 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:01.511585 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-5e2d03/auth-test-model-5e2d03-kserve-665774f749-75t2g" Nov 01 16:26:01.653603 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:01.653568 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e66518b6-c569-4afd-9464-ba934cccbcf7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e66518b6-c569-4afd-9464-ba934cccbcf7-model-cache\") pod \"e66518b6-c569-4afd-9464-ba934cccbcf7\" (UID: \"e66518b6-c569-4afd-9464-ba934cccbcf7\") " Nov 01 16:26:01.653808 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:01.653676 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e66518b6-c569-4afd-9464-ba934cccbcf7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e66518b6-c569-4afd-9464-ba934cccbcf7-tmp-dir\") pod \"e66518b6-c569-4afd-9464-ba934cccbcf7\" (UID: \"e66518b6-c569-4afd-9464-ba934cccbcf7\") " Nov 01 16:26:01.653808 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:01.653707 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e66518b6-c569-4afd-9464-ba934cccbcf7-home\" (UniqueName: \"kubernetes.io/empty-dir/e66518b6-c569-4afd-9464-ba934cccbcf7-home\") pod \"e66518b6-c569-4afd-9464-ba934cccbcf7\" (UID: \"e66518b6-c569-4afd-9464-ba934cccbcf7\") " Nov 01 16:26:01.653808 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:01.653778 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e66518b6-c569-4afd-9464-ba934cccbcf7-tls-certs\" (UniqueName: \"kubernetes.io/secret/e66518b6-c569-4afd-9464-ba934cccbcf7-tls-certs\") pod \"e66518b6-c569-4afd-9464-ba934cccbcf7\" (UID: \"e66518b6-c569-4afd-9464-ba934cccbcf7\") " Nov 01 16:26:01.654011 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:01.653814 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e66518b6-c569-4afd-9464-ba934cccbcf7-kube-api-access-42xz4\" (UniqueName: \"kubernetes.io/projected/e66518b6-c569-4afd-9464-ba934cccbcf7-kube-api-access-42xz4\") pod \"e66518b6-c569-4afd-9464-ba934cccbcf7\" (UID: \"e66518b6-c569-4afd-9464-ba934cccbcf7\") " Nov 01 16:26:01.654011 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:01.653916 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e66518b6-c569-4afd-9464-ba934cccbcf7-model-cache" pod "e66518b6-c569-4afd-9464-ba934cccbcf7" (UID: "e66518b6-c569-4afd-9464-ba934cccbcf7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:26:01.654011 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:01.653977 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e66518b6-c569-4afd-9464-ba934cccbcf7-tmp-dir" pod "e66518b6-c569-4afd-9464-ba934cccbcf7" (UID: "e66518b6-c569-4afd-9464-ba934cccbcf7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:26:01.654185 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:01.654044 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e66518b6-c569-4afd-9464-ba934cccbcf7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e66518b6-c569-4afd-9464-ba934cccbcf7-dshm\") pod \"e66518b6-c569-4afd-9464-ba934cccbcf7\" (UID: \"e66518b6-c569-4afd-9464-ba934cccbcf7\") " Nov 01 16:26:01.654185 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:01.654083 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e66518b6-c569-4afd-9464-ba934cccbcf7-home" pod "e66518b6-c569-4afd-9464-ba934cccbcf7" (UID: "e66518b6-c569-4afd-9464-ba934cccbcf7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:26:01.654915 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:01.654889 4363 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e66518b6-c569-4afd-9464-ba934cccbcf7-model-cache\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:26:01.654915 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:01.654913 4363 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e66518b6-c569-4afd-9464-ba934cccbcf7-tmp-dir\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:26:01.655156 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:01.654922 4363 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e66518b6-c569-4afd-9464-ba934cccbcf7-home\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:26:01.656172 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:01.656150 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e66518b6-c569-4afd-9464-ba934cccbcf7-dshm" pod "e66518b6-c569-4afd-9464-ba934cccbcf7" (UID: "e66518b6-c569-4afd-9464-ba934cccbcf7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:26:01.656296 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:01.656222 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e66518b6-c569-4afd-9464-ba934cccbcf7-kube-api-access-42xz4" pod "e66518b6-c569-4afd-9464-ba934cccbcf7" (UID: "e66518b6-c569-4afd-9464-ba934cccbcf7"). InnerVolumeSpecName "kube-api-access-42xz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:26:01.656296 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:01.656272 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e66518b6-c569-4afd-9464-ba934cccbcf7-tls-certs" pod "e66518b6-c569-4afd-9464-ba934cccbcf7" (UID: "e66518b6-c569-4afd-9464-ba934cccbcf7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:26:01.756641 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:01.756535 4363 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e66518b6-c569-4afd-9464-ba934cccbcf7-tls-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:26:01.756641 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:01.756569 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42xz4\" (UniqueName: \"kubernetes.io/projected/e66518b6-c569-4afd-9464-ba934cccbcf7-kube-api-access-42xz4\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:26:01.756641 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:01.756585 4363 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e66518b6-c569-4afd-9464-ba934cccbcf7-dshm\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:26:02.186249 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:02.186210 4363 generic.go:356] "Generic (PLEG): container finished" podID="e66518b6-c569-4afd-9464-ba934cccbcf7" containerID="a31747fe9aba851c57ca451ab9361a562b2aa34e14e1a923aca1efa0ab59efd2" exitCode=0 Nov 01 16:26:02.186722 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:02.186259 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-5e2d03/auth-test-model-5e2d03-kserve-665774f749-75t2g" event={"ID":"e66518b6-c569-4afd-9464-ba934cccbcf7","Type":"ContainerDied","Data":"a31747fe9aba851c57ca451ab9361a562b2aa34e14e1a923aca1efa0ab59efd2"} Nov 01 16:26:02.186722 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:02.186296 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-5e2d03/auth-test-model-5e2d03-kserve-665774f749-75t2g" event={"ID":"e66518b6-c569-4afd-9464-ba934cccbcf7","Type":"ContainerDied","Data":"259259e3cec6cec93fbca18b5e8255e7132c82e8ddb457f8098959ba40416aea"} Nov 01 16:26:02.186722 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:02.186319 4363 scope.go:122] "RemoveContainer" containerID="a31747fe9aba851c57ca451ab9361a562b2aa34e14e1a923aca1efa0ab59efd2" Nov 01 16:26:02.186722 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:02.186472 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-5e2d03/auth-test-model-5e2d03-kserve-665774f749-75t2g" Nov 01 16:26:02.199644 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:02.199624 4363 scope.go:122] "RemoveContainer" containerID="a31747fe9aba851c57ca451ab9361a562b2aa34e14e1a923aca1efa0ab59efd2" Nov 01 16:26:02.199924 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:26:02.199904 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a31747fe9aba851c57ca451ab9361a562b2aa34e14e1a923aca1efa0ab59efd2\": container with ID starting with a31747fe9aba851c57ca451ab9361a562b2aa34e14e1a923aca1efa0ab59efd2 not found: ID does not exist" containerID="a31747fe9aba851c57ca451ab9361a562b2aa34e14e1a923aca1efa0ab59efd2" Nov 01 16:26:02.199974 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:02.199936 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a31747fe9aba851c57ca451ab9361a562b2aa34e14e1a923aca1efa0ab59efd2"} err="failed to get container status \"a31747fe9aba851c57ca451ab9361a562b2aa34e14e1a923aca1efa0ab59efd2\": rpc error: code = NotFound desc = could not find container \"a31747fe9aba851c57ca451ab9361a562b2aa34e14e1a923aca1efa0ab59efd2\": container with ID starting with a31747fe9aba851c57ca451ab9361a562b2aa34e14e1a923aca1efa0ab59efd2 not found: ID does not exist" Nov 01 16:26:02.211768 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:02.211743 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-5e2d03/auth-test-model-5e2d03-kserve-665774f749-75t2g"] Nov 01 16:26:02.215763 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:02.215744 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-5e2d03/auth-test-model-5e2d03-kserve-665774f749-75t2g"] Nov 01 16:26:02.800897 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:02.800861 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e66518b6-c569-4afd-9464-ba934cccbcf7" path="/var/lib/kubelet/pods/e66518b6-c569-4afd-9464-ba934cccbcf7/volumes" Nov 01 16:26:18.469512 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:26:18.469470 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:26:43.247223 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:43.247190 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-383fde/auth-test-model-383fde-kserve-57c55ccb8f-rqp6x"] Nov 01 16:26:46.352985 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:46.352950 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf"] Nov 01 16:26:46.353535 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:46.353223 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" podUID="1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12" containerName="istio-proxy" containerID="cri-o://7c5e178d9f75b1517a8dd2d804127ff4a9ac088a5337cb37341ca8a3c29446f1" gracePeriod=30 Nov 01 16:26:47.884458 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:47.884407 4363 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:26:47.884970 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:47.884461 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" podUID="1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:26:48.312998 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:26:48.312902 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:26:49.647111 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:49.647067 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-675cc9cb59-vmxp4"] Nov 01 16:26:49.676040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:49.663546 4363 cpu_manager.go:414] "containerMap: removing container" podUID="e66518b6-c569-4afd-9464-ba934cccbcf7" containerName="main" Nov 01 16:26:49.690118 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:49.690089 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e66518b6-c569-4afd-9464-ba934cccbcf7" containerName="main" Nov 01 16:26:49.756388 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:49.690769 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="e66518b6-c569-4afd-9464-ba934cccbcf7" containerName="main" Nov 01 16:26:49.867164 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:49.867123 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-675cc9cb59-vmxp4"] Nov 01 16:26:49.867478 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:49.867458 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-675cc9cb59-vmxp4" Nov 01 16:26:49.872881 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:49.872111 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-104b6e-dockercfg-b7rf5\"" Nov 01 16:26:49.872881 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:49.872178 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-104b6e\"" Nov 01 16:26:50.010665 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:50.010579 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flsp4\" (UniqueName: \"kubernetes.io/projected/2d97c2a5-f8a5-4b3f-aae1-2ba5295a7c90-kube-api-access-flsp4\") pod \"payload-pre-processing-e2e-ipp-a-104b6e-675cc9cb59-vmxp4\" (UID: \"2d97c2a5-f8a5-4b3f-aae1-2ba5295a7c90\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-675cc9cb59-vmxp4" Nov 01 16:26:50.010665 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:50.010622 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d97c2a5-f8a5-4b3f-aae1-2ba5295a7c90-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-104b6e-675cc9cb59-vmxp4\" (UID: \"2d97c2a5-f8a5-4b3f-aae1-2ba5295a7c90\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-675cc9cb59-vmxp4" Nov 01 16:26:50.112099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:50.112060 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flsp4\" (UniqueName: \"kubernetes.io/projected/2d97c2a5-f8a5-4b3f-aae1-2ba5295a7c90-kube-api-access-flsp4\") pod \"payload-pre-processing-e2e-ipp-a-104b6e-675cc9cb59-vmxp4\" (UID: \"2d97c2a5-f8a5-4b3f-aae1-2ba5295a7c90\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-675cc9cb59-vmxp4" Nov 01 16:26:50.112283 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:50.112126 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d97c2a5-f8a5-4b3f-aae1-2ba5295a7c90-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-104b6e-675cc9cb59-vmxp4\" (UID: \"2d97c2a5-f8a5-4b3f-aae1-2ba5295a7c90\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-675cc9cb59-vmxp4" Nov 01 16:26:50.112912 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:50.112884 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d97c2a5-f8a5-4b3f-aae1-2ba5295a7c90-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-104b6e-675cc9cb59-vmxp4\" (UID: \"2d97c2a5-f8a5-4b3f-aae1-2ba5295a7c90\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-675cc9cb59-vmxp4" Nov 01 16:26:50.120144 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:50.120095 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flsp4\" (UniqueName: \"kubernetes.io/projected/2d97c2a5-f8a5-4b3f-aae1-2ba5295a7c90-kube-api-access-flsp4\") pod \"payload-pre-processing-e2e-ipp-a-104b6e-675cc9cb59-vmxp4\" (UID: \"2d97c2a5-f8a5-4b3f-aae1-2ba5295a7c90\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-675cc9cb59-vmxp4" Nov 01 16:26:50.191128 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:50.191097 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-675cc9cb59-vmxp4" Nov 01 16:26:50.489074 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:50.489042 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-675cc9cb59-vmxp4"] Nov 01 16:26:50.494538 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:50.494512 4363 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 16:26:51.069632 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:51.069594 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-675cc9cb59-vmxp4" event={"ID":"2d97c2a5-f8a5-4b3f-aae1-2ba5295a7c90","Type":"ContainerStarted","Data":"fe287acd573f50d78e998021dff089f24e85189967403f698338466284a1ce7c"} Nov 01 16:26:51.835011 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:51.834978 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" Nov 01 16:26:51.861835 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:51.861799 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-istio-token\" (UniqueName: \"kubernetes.io/projected/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-istio-token\") pod \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\" (UID: \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\") " Nov 01 16:26:51.862058 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:51.861867 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-istio-data\") pod \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\" (UID: \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\") " Nov 01 16:26:51.862058 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:51.861970 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-kube-api-access-twwp8\" (UniqueName: \"kubernetes.io/projected/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-kube-api-access-twwp8\") pod \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\" (UID: \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\") " Nov 01 16:26:51.862058 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:51.862046 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-workload-socket\") pod \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\" (UID: \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\") " Nov 01 16:26:51.862248 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:51.862126 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-istio-podinfo\") pod \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\" (UID: \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\") " Nov 01 16:26:51.862324 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:51.862254 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-istiod-ca-cert\") pod \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\" (UID: \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\") " Nov 01 16:26:51.862384 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:51.862351 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-credential-socket\") pod \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\" (UID: \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\") " Nov 01 16:26:51.862438 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:51.862385 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-istio-envoy\") pod \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\" (UID: \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\") " Nov 01 16:26:51.862504 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:51.862474 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-workload-certs\") pod \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\" (UID: \"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12\") " Nov 01 16:26:51.863729 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:51.862931 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-credential-socket" pod "1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12" (UID: "1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:26:51.863729 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:51.863197 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-workload-socket" pod "1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12" (UID: "1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:26:51.863729 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:51.863267 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-istiod-ca-cert" pod "1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12" (UID: "1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:26:51.863729 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:51.863571 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:26:51.863729 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:51.863593 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:26:51.863729 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:51.863609 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:26:51.863729 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:51.863707 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-workload-certs" pod "1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12" (UID: "1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:26:51.864411 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:51.864387 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-istio-data" pod "1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12" (UID: "1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:26:51.865527 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:51.865494 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-istio-envoy" pod "1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12" (UID: "1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:26:51.865814 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:51.865780 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-kube-api-access-twwp8" pod "1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12" (UID: "1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12"). InnerVolumeSpecName "kube-api-access-twwp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:26:51.910635 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:51.910603 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-istio-podinfo" pod "1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12" (UID: "1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:26:51.910779 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:51.910744 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-istio-token" pod "1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12" (UID: "1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:26:51.967222 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:51.967094 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:26:51.967222 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:51.967132 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:26:51.967222 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:51.967147 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:26:51.967222 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:51.967162 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:26:51.967222 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:51.967178 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twwp8\" (UniqueName: \"kubernetes.io/projected/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-kube-api-access-twwp8\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:26:51.967222 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:51.967193 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:26:52.080797 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:52.080704 4363 generic.go:356] "Generic (PLEG): container finished" podID="1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12" containerID="7c5e178d9f75b1517a8dd2d804127ff4a9ac088a5337cb37341ca8a3c29446f1" exitCode=0 Nov 01 16:26:52.080797 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:52.080757 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" Nov 01 16:26:52.080797 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:52.080755 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" event={"ID":"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12","Type":"ContainerDied","Data":"7c5e178d9f75b1517a8dd2d804127ff4a9ac088a5337cb37341ca8a3c29446f1"} Nov 01 16:26:52.081482 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:52.080817 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf" event={"ID":"1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12","Type":"ContainerDied","Data":"ee0ad12683ed464798009188d69966759e24d3ddf75244201b01515070c310c7"} Nov 01 16:26:52.081482 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:52.080846 4363 scope.go:122] "RemoveContainer" containerID="7c5e178d9f75b1517a8dd2d804127ff4a9ac088a5337cb37341ca8a3c29446f1" Nov 01 16:26:52.097807 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:52.097746 4363 scope.go:122] "RemoveContainer" containerID="b39a2cceb2e7f80ace969b2f8ef8e83f046c29be469d309a114e4c8f25098b49" Nov 01 16:26:52.113044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:52.112985 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf"] Nov 01 16:26:52.115252 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:52.115189 4363 scope.go:122] "RemoveContainer" containerID="7c5e178d9f75b1517a8dd2d804127ff4a9ac088a5337cb37341ca8a3c29446f1" Nov 01 16:26:52.115578 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:26:52.115547 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c5e178d9f75b1517a8dd2d804127ff4a9ac088a5337cb37341ca8a3c29446f1\": container with ID starting with 7c5e178d9f75b1517a8dd2d804127ff4a9ac088a5337cb37341ca8a3c29446f1 not found: ID does not exist" containerID="7c5e178d9f75b1517a8dd2d804127ff4a9ac088a5337cb37341ca8a3c29446f1" Nov 01 16:26:52.115677 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:52.115589 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c5e178d9f75b1517a8dd2d804127ff4a9ac088a5337cb37341ca8a3c29446f1"} err="failed to get container status \"7c5e178d9f75b1517a8dd2d804127ff4a9ac088a5337cb37341ca8a3c29446f1\": rpc error: code = NotFound desc = could not find container \"7c5e178d9f75b1517a8dd2d804127ff4a9ac088a5337cb37341ca8a3c29446f1\": container with ID starting with 7c5e178d9f75b1517a8dd2d804127ff4a9ac088a5337cb37341ca8a3c29446f1 not found: ID does not exist" Nov 01 16:26:52.115677 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:52.115613 4363 scope.go:122] "RemoveContainer" containerID="b39a2cceb2e7f80ace969b2f8ef8e83f046c29be469d309a114e4c8f25098b49" Nov 01 16:26:52.116046 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:26:52.115966 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b39a2cceb2e7f80ace969b2f8ef8e83f046c29be469d309a114e4c8f25098b49\": container with ID starting with b39a2cceb2e7f80ace969b2f8ef8e83f046c29be469d309a114e4c8f25098b49 not found: ID does not exist" containerID="b39a2cceb2e7f80ace969b2f8ef8e83f046c29be469d309a114e4c8f25098b49" Nov 01 16:26:52.116046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:52.116004 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b39a2cceb2e7f80ace969b2f8ef8e83f046c29be469d309a114e4c8f25098b49"} err="failed to get container status \"b39a2cceb2e7f80ace969b2f8ef8e83f046c29be469d309a114e4c8f25098b49\": rpc error: code = NotFound desc = could not find container \"b39a2cceb2e7f80ace969b2f8ef8e83f046c29be469d309a114e4c8f25098b49\": container with ID starting with b39a2cceb2e7f80ace969b2f8ef8e83f046c29be469d309a114e4c8f25098b49 not found: ID does not exist" Nov 01 16:26:52.120307 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:52.120284 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5hnrf"] Nov 01 16:26:52.805323 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:52.805288 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12" path="/var/lib/kubelet/pods/1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12/volumes" Nov 01 16:26:53.128271 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:53.127897 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-675cc9cb59-vmxp4" event={"ID":"2d97c2a5-f8a5-4b3f-aae1-2ba5295a7c90","Type":"ContainerStarted","Data":"31bdf252913009e0f8bcc87794e4ad72968c4837dad4f661857658a042ea41b7"} Nov 01 16:26:53.128271 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:53.128261 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-675cc9cb59-vmxp4" Nov 01 16:26:53.145315 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:53.145273 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-675cc9cb59-vmxp4" podStartSLOduration=1.95233927 podStartE2EDuration="4.145260308s" totalImagesPullingTime="2.192921038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:26:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:26:53.142705775 +0000 UTC m=+2346.474443416" watchObservedRunningTime="2026-11-01 16:26:53.145260308 +0000 UTC m=+2346.476997958" Nov 01 16:26:58.282442 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:58.282397 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w5-383fde/auth-test-model-383fde-kserve-57c55ccb8f-rqp6x" podUID="36f441c7-2672-4581-9a0e-7142326e172b" containerName="main" containerID="cri-o://b87713ac319cb123e6302ca79ca8ebd682c13e22482ead2dc659452186320cdc" gracePeriod=45 Nov 01 16:26:58.427200 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:58.427162 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w5-383fde/auth-test-model-383fde-kserve-57c55ccb8f-rqp6x" podUID="36f441c7-2672-4581-9a0e-7142326e172b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.209:8000/ready\": dial tcp 10.134.0.209:8000: connect: connection refused" Nov 01 16:26:58.682917 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:58.682892 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-383fde/auth-test-model-383fde-kserve-57c55ccb8f-rqp6x" Nov 01 16:26:58.809687 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:58.809658 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-e1e630/ipp-route-e1e630-kserve-85dbb46dd5-qqv5q"] Nov 01 16:26:58.810594 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:58.810564 4363 cpu_manager.go:414] "containerMap: removing container" podUID="36f441c7-2672-4581-9a0e-7142326e172b" containerName="main" Nov 01 16:26:58.810594 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:58.810589 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36f441c7-2672-4581-9a0e-7142326e172b" containerName="main" Nov 01 16:26:58.810811 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:58.810603 4363 cpu_manager.go:414] "containerMap: removing container" podUID="1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12" containerName="istio-proxy" Nov 01 16:26:58.810811 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:58.810608 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12" containerName="istio-proxy" Nov 01 16:26:58.810811 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:58.810621 4363 cpu_manager.go:414] "containerMap: removing container" podUID="1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12" containerName="istio-proxy" Nov 01 16:26:58.810811 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:58.810626 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12" containerName="istio-proxy" Nov 01 16:26:58.810811 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:58.810739 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="36f441c7-2672-4581-9a0e-7142326e172b" containerName="main" Nov 01 16:26:58.810811 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:58.810746 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12" containerName="istio-proxy" Nov 01 16:26:58.810811 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:58.810752 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e7a1c82-7d0f-4304-b4dc-5c36b0f6ab12" containerName="istio-proxy" Nov 01 16:26:58.814695 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:58.814672 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e1e630/ipp-route-e1e630-kserve-85dbb46dd5-qqv5q" Nov 01 16:26:58.826430 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:58.826388 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-e1e630\"/\"openshift-service-ca.crt\"" Nov 01 16:26:58.826588 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:58.826449 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-e1e630\"/\"ipp-route-e1e630-kserve-self-signed-certs\"" Nov 01 16:26:58.826588 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:58.826457 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-e1e630\"/\"default-dockercfg-h9r8f\"" Nov 01 16:26:58.826588 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:58.826542 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-e1e630\"/\"kube-root-ca.crt\"" Nov 01 16:26:58.829771 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:58.829660 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-e1e630/ipp-route-e1e630-kserve-85dbb46dd5-qqv5q"] Nov 01 16:26:58.836552 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:58.836526 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36f441c7-2672-4581-9a0e-7142326e172b-kube-api-access-spgll\" (UniqueName: \"kubernetes.io/projected/36f441c7-2672-4581-9a0e-7142326e172b-kube-api-access-spgll\") pod \"36f441c7-2672-4581-9a0e-7142326e172b\" (UID: \"36f441c7-2672-4581-9a0e-7142326e172b\") " Nov 01 16:26:58.836688 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:58.836650 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36f441c7-2672-4581-9a0e-7142326e172b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36f441c7-2672-4581-9a0e-7142326e172b-model-cache\") pod \"36f441c7-2672-4581-9a0e-7142326e172b\" (UID: \"36f441c7-2672-4581-9a0e-7142326e172b\") " Nov 01 16:26:58.836765 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:58.836736 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36f441c7-2672-4581-9a0e-7142326e172b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/36f441c7-2672-4581-9a0e-7142326e172b-dshm\") pod \"36f441c7-2672-4581-9a0e-7142326e172b\" (UID: \"36f441c7-2672-4581-9a0e-7142326e172b\") " Nov 01 16:26:58.837109 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:58.837083 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36f441c7-2672-4581-9a0e-7142326e172b-home\" (UniqueName: \"kubernetes.io/empty-dir/36f441c7-2672-4581-9a0e-7142326e172b-home\") pod \"36f441c7-2672-4581-9a0e-7142326e172b\" (UID: \"36f441c7-2672-4581-9a0e-7142326e172b\") " Nov 01 16:26:58.837201 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:58.837124 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36f441c7-2672-4581-9a0e-7142326e172b-tls-certs\" (UniqueName: \"kubernetes.io/secret/36f441c7-2672-4581-9a0e-7142326e172b-tls-certs\") pod \"36f441c7-2672-4581-9a0e-7142326e172b\" (UID: \"36f441c7-2672-4581-9a0e-7142326e172b\") " Nov 01 16:26:58.837298 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:58.837281 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36f441c7-2672-4581-9a0e-7142326e172b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36f441c7-2672-4581-9a0e-7142326e172b-tmp-dir\") pod \"36f441c7-2672-4581-9a0e-7142326e172b\" (UID: \"36f441c7-2672-4581-9a0e-7142326e172b\") " Nov 01 16:26:58.838245 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:58.838163 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36f441c7-2672-4581-9a0e-7142326e172b-model-cache" pod "36f441c7-2672-4581-9a0e-7142326e172b" (UID: "36f441c7-2672-4581-9a0e-7142326e172b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:26:58.838362 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:58.838271 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36f441c7-2672-4581-9a0e-7142326e172b-tmp-dir" pod "36f441c7-2672-4581-9a0e-7142326e172b" (UID: "36f441c7-2672-4581-9a0e-7142326e172b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:26:58.839225 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:58.839206 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36f441c7-2672-4581-9a0e-7142326e172b-home" pod "36f441c7-2672-4581-9a0e-7142326e172b" (UID: "36f441c7-2672-4581-9a0e-7142326e172b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:26:58.840274 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:58.840255 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36f441c7-2672-4581-9a0e-7142326e172b-dshm" pod "36f441c7-2672-4581-9a0e-7142326e172b" (UID: "36f441c7-2672-4581-9a0e-7142326e172b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:26:58.846401 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:58.846377 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36f441c7-2672-4581-9a0e-7142326e172b-kube-api-access-spgll" pod "36f441c7-2672-4581-9a0e-7142326e172b" (UID: "36f441c7-2672-4581-9a0e-7142326e172b"). InnerVolumeSpecName "kube-api-access-spgll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:26:58.855520 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:58.855497 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36f441c7-2672-4581-9a0e-7142326e172b-tls-certs" pod "36f441c7-2672-4581-9a0e-7142326e172b" (UID: "36f441c7-2672-4581-9a0e-7142326e172b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:26:58.939948 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:58.939861 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/474f6a6d-0ab2-4f46-b9b8-f29f577f7787-home\") pod \"ipp-route-e1e630-kserve-85dbb46dd5-qqv5q\" (UID: \"474f6a6d-0ab2-4f46-b9b8-f29f577f7787\") " pod="ai-tenant-e2e-ipp-a-e1e630/ipp-route-e1e630-kserve-85dbb46dd5-qqv5q" Nov 01 16:26:58.940158 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:58.940012 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/474f6a6d-0ab2-4f46-b9b8-f29f577f7787-dshm\") pod \"ipp-route-e1e630-kserve-85dbb46dd5-qqv5q\" (UID: \"474f6a6d-0ab2-4f46-b9b8-f29f577f7787\") " pod="ai-tenant-e2e-ipp-a-e1e630/ipp-route-e1e630-kserve-85dbb46dd5-qqv5q" Nov 01 16:26:58.940223 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:58.940156 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg6sf\" (UniqueName: \"kubernetes.io/projected/474f6a6d-0ab2-4f46-b9b8-f29f577f7787-kube-api-access-wg6sf\") pod \"ipp-route-e1e630-kserve-85dbb46dd5-qqv5q\" (UID: \"474f6a6d-0ab2-4f46-b9b8-f29f577f7787\") " pod="ai-tenant-e2e-ipp-a-e1e630/ipp-route-e1e630-kserve-85dbb46dd5-qqv5q" Nov 01 16:26:58.940282 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:58.940235 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/474f6a6d-0ab2-4f46-b9b8-f29f577f7787-model-cache\") pod \"ipp-route-e1e630-kserve-85dbb46dd5-qqv5q\" (UID: \"474f6a6d-0ab2-4f46-b9b8-f29f577f7787\") " pod="ai-tenant-e2e-ipp-a-e1e630/ipp-route-e1e630-kserve-85dbb46dd5-qqv5q" Nov 01 16:26:58.940398 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:58.940380 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/474f6a6d-0ab2-4f46-b9b8-f29f577f7787-tmp-dir\") pod \"ipp-route-e1e630-kserve-85dbb46dd5-qqv5q\" (UID: \"474f6a6d-0ab2-4f46-b9b8-f29f577f7787\") " pod="ai-tenant-e2e-ipp-a-e1e630/ipp-route-e1e630-kserve-85dbb46dd5-qqv5q" Nov 01 16:26:58.940475 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:58.940454 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/474f6a6d-0ab2-4f46-b9b8-f29f577f7787-tls-certs\") pod \"ipp-route-e1e630-kserve-85dbb46dd5-qqv5q\" (UID: \"474f6a6d-0ab2-4f46-b9b8-f29f577f7787\") " pod="ai-tenant-e2e-ipp-a-e1e630/ipp-route-e1e630-kserve-85dbb46dd5-qqv5q" Nov 01 16:26:58.940588 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:58.940566 4363 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36f441c7-2672-4581-9a0e-7142326e172b-home\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:26:58.940643 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:58.940592 4363 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36f441c7-2672-4581-9a0e-7142326e172b-tls-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:26:58.940643 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:58.940609 4363 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36f441c7-2672-4581-9a0e-7142326e172b-tmp-dir\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:26:58.940643 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:58.940619 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spgll\" (UniqueName: \"kubernetes.io/projected/36f441c7-2672-4581-9a0e-7142326e172b-kube-api-access-spgll\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:26:58.940643 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:58.940629 4363 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36f441c7-2672-4581-9a0e-7142326e172b-model-cache\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:26:58.940643 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:58.940641 4363 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36f441c7-2672-4581-9a0e-7142326e172b-dshm\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:26:59.053041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:59.046327 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/474f6a6d-0ab2-4f46-b9b8-f29f577f7787-model-cache\") pod \"ipp-route-e1e630-kserve-85dbb46dd5-qqv5q\" (UID: \"474f6a6d-0ab2-4f46-b9b8-f29f577f7787\") " pod="ai-tenant-e2e-ipp-a-e1e630/ipp-route-e1e630-kserve-85dbb46dd5-qqv5q" Nov 01 16:26:59.053041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:59.046490 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/474f6a6d-0ab2-4f46-b9b8-f29f577f7787-tmp-dir\") pod \"ipp-route-e1e630-kserve-85dbb46dd5-qqv5q\" (UID: \"474f6a6d-0ab2-4f46-b9b8-f29f577f7787\") " pod="ai-tenant-e2e-ipp-a-e1e630/ipp-route-e1e630-kserve-85dbb46dd5-qqv5q" Nov 01 16:26:59.053041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:59.046551 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/474f6a6d-0ab2-4f46-b9b8-f29f577f7787-tls-certs\") pod \"ipp-route-e1e630-kserve-85dbb46dd5-qqv5q\" (UID: \"474f6a6d-0ab2-4f46-b9b8-f29f577f7787\") " pod="ai-tenant-e2e-ipp-a-e1e630/ipp-route-e1e630-kserve-85dbb46dd5-qqv5q" Nov 01 16:26:59.053041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:59.046639 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/474f6a6d-0ab2-4f46-b9b8-f29f577f7787-home\") pod \"ipp-route-e1e630-kserve-85dbb46dd5-qqv5q\" (UID: \"474f6a6d-0ab2-4f46-b9b8-f29f577f7787\") " pod="ai-tenant-e2e-ipp-a-e1e630/ipp-route-e1e630-kserve-85dbb46dd5-qqv5q" Nov 01 16:26:59.053041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:59.046755 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/474f6a6d-0ab2-4f46-b9b8-f29f577f7787-dshm\") pod \"ipp-route-e1e630-kserve-85dbb46dd5-qqv5q\" (UID: \"474f6a6d-0ab2-4f46-b9b8-f29f577f7787\") " pod="ai-tenant-e2e-ipp-a-e1e630/ipp-route-e1e630-kserve-85dbb46dd5-qqv5q" Nov 01 16:26:59.053041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:59.046833 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg6sf\" (UniqueName: \"kubernetes.io/projected/474f6a6d-0ab2-4f46-b9b8-f29f577f7787-kube-api-access-wg6sf\") pod \"ipp-route-e1e630-kserve-85dbb46dd5-qqv5q\" (UID: \"474f6a6d-0ab2-4f46-b9b8-f29f577f7787\") " pod="ai-tenant-e2e-ipp-a-e1e630/ipp-route-e1e630-kserve-85dbb46dd5-qqv5q" Nov 01 16:26:59.053041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:59.047658 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/474f6a6d-0ab2-4f46-b9b8-f29f577f7787-model-cache\") pod \"ipp-route-e1e630-kserve-85dbb46dd5-qqv5q\" (UID: \"474f6a6d-0ab2-4f46-b9b8-f29f577f7787\") " pod="ai-tenant-e2e-ipp-a-e1e630/ipp-route-e1e630-kserve-85dbb46dd5-qqv5q" Nov 01 16:26:59.053041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:59.047923 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/474f6a6d-0ab2-4f46-b9b8-f29f577f7787-tmp-dir\") pod \"ipp-route-e1e630-kserve-85dbb46dd5-qqv5q\" (UID: \"474f6a6d-0ab2-4f46-b9b8-f29f577f7787\") " pod="ai-tenant-e2e-ipp-a-e1e630/ipp-route-e1e630-kserve-85dbb46dd5-qqv5q" Nov 01 16:26:59.053041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:59.048563 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/474f6a6d-0ab2-4f46-b9b8-f29f577f7787-home\") pod \"ipp-route-e1e630-kserve-85dbb46dd5-qqv5q\" (UID: \"474f6a6d-0ab2-4f46-b9b8-f29f577f7787\") " pod="ai-tenant-e2e-ipp-a-e1e630/ipp-route-e1e630-kserve-85dbb46dd5-qqv5q" Nov 01 16:26:59.062036 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:59.058301 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/474f6a6d-0ab2-4f46-b9b8-f29f577f7787-dshm\") pod \"ipp-route-e1e630-kserve-85dbb46dd5-qqv5q\" (UID: \"474f6a6d-0ab2-4f46-b9b8-f29f577f7787\") " pod="ai-tenant-e2e-ipp-a-e1e630/ipp-route-e1e630-kserve-85dbb46dd5-qqv5q" Nov 01 16:26:59.070862 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:59.070446 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg6sf\" (UniqueName: \"kubernetes.io/projected/474f6a6d-0ab2-4f46-b9b8-f29f577f7787-kube-api-access-wg6sf\") pod \"ipp-route-e1e630-kserve-85dbb46dd5-qqv5q\" (UID: \"474f6a6d-0ab2-4f46-b9b8-f29f577f7787\") " pod="ai-tenant-e2e-ipp-a-e1e630/ipp-route-e1e630-kserve-85dbb46dd5-qqv5q" Nov 01 16:26:59.079037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:59.072707 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/474f6a6d-0ab2-4f46-b9b8-f29f577f7787-tls-certs\") pod \"ipp-route-e1e630-kserve-85dbb46dd5-qqv5q\" (UID: \"474f6a6d-0ab2-4f46-b9b8-f29f577f7787\") " pod="ai-tenant-e2e-ipp-a-e1e630/ipp-route-e1e630-kserve-85dbb46dd5-qqv5q" Nov 01 16:26:59.154521 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:59.152244 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e1e630/ipp-route-e1e630-kserve-85dbb46dd5-qqv5q" Nov 01 16:26:59.196758 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:59.196545 4363 generic.go:356] "Generic (PLEG): container finished" podID="36f441c7-2672-4581-9a0e-7142326e172b" containerID="b87713ac319cb123e6302ca79ca8ebd682c13e22482ead2dc659452186320cdc" exitCode=0 Nov 01 16:26:59.196758 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:59.196645 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-383fde/auth-test-model-383fde-kserve-57c55ccb8f-rqp6x" Nov 01 16:26:59.196758 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:59.196672 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-383fde/auth-test-model-383fde-kserve-57c55ccb8f-rqp6x" event={"ID":"36f441c7-2672-4581-9a0e-7142326e172b","Type":"ContainerDied","Data":"b87713ac319cb123e6302ca79ca8ebd682c13e22482ead2dc659452186320cdc"} Nov 01 16:26:59.196758 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:59.196715 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-383fde/auth-test-model-383fde-kserve-57c55ccb8f-rqp6x" event={"ID":"36f441c7-2672-4581-9a0e-7142326e172b","Type":"ContainerDied","Data":"5b8646d5d4def7609887b59fca56c3bb8c9a3a22803e6cf7194df280eb11cc16"} Nov 01 16:26:59.196758 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:59.196737 4363 scope.go:122] "RemoveContainer" containerID="b87713ac319cb123e6302ca79ca8ebd682c13e22482ead2dc659452186320cdc" Nov 01 16:26:59.213749 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:59.213723 4363 scope.go:122] "RemoveContainer" containerID="b87713ac319cb123e6302ca79ca8ebd682c13e22482ead2dc659452186320cdc" Nov 01 16:26:59.214988 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:26:59.214961 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b87713ac319cb123e6302ca79ca8ebd682c13e22482ead2dc659452186320cdc\": container with ID starting with b87713ac319cb123e6302ca79ca8ebd682c13e22482ead2dc659452186320cdc not found: ID does not exist" containerID="b87713ac319cb123e6302ca79ca8ebd682c13e22482ead2dc659452186320cdc" Nov 01 16:26:59.215499 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:59.215001 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b87713ac319cb123e6302ca79ca8ebd682c13e22482ead2dc659452186320cdc"} err="failed to get container status \"b87713ac319cb123e6302ca79ca8ebd682c13e22482ead2dc659452186320cdc\": rpc error: code = NotFound desc = could not find container \"b87713ac319cb123e6302ca79ca8ebd682c13e22482ead2dc659452186320cdc\": container with ID starting with b87713ac319cb123e6302ca79ca8ebd682c13e22482ead2dc659452186320cdc not found: ID does not exist" Nov 01 16:26:59.227984 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:59.227964 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-383fde/auth-test-model-383fde-kserve-57c55ccb8f-rqp6x"] Nov 01 16:26:59.233203 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:59.233179 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w5-383fde/auth-test-model-383fde-kserve-57c55ccb8f-rqp6x"] Nov 01 16:26:59.477505 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:26:59.477470 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-e1e630/ipp-route-e1e630-kserve-85dbb46dd5-qqv5q"] Nov 01 16:26:59.478578 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:26:59.478548 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod474f6a6d_0ab2_4f46_b9b8_f29f577f7787.slice/crio-d46c8c3ec19d6852059e405167e5134e1a7885010468ef6516a38a59502ec8ad WatchSource:0}: Error finding container d46c8c3ec19d6852059e405167e5134e1a7885010468ef6516a38a59502ec8ad: Status 404 returned error can't find the container with id d46c8c3ec19d6852059e405167e5134e1a7885010468ef6516a38a59502ec8ad Nov 01 16:27:00.209392 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:00.209352 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e1e630/ipp-route-e1e630-kserve-85dbb46dd5-qqv5q" event={"ID":"474f6a6d-0ab2-4f46-b9b8-f29f577f7787","Type":"ContainerStarted","Data":"6ba8a563e37b7d8b4e8f0a5869b2f43fed73a62556405e59548bfb3ffa3b15ef"} Nov 01 16:27:00.209594 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:00.209408 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e1e630/ipp-route-e1e630-kserve-85dbb46dd5-qqv5q" event={"ID":"474f6a6d-0ab2-4f46-b9b8-f29f577f7787","Type":"ContainerStarted","Data":"d46c8c3ec19d6852059e405167e5134e1a7885010468ef6516a38a59502ec8ad"} Nov 01 16:27:00.254412 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:00.253196 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-e1e630/ipp-route-e1e630-kserve-85dbb46dd5-qqv5q" podStartSLOduration=2.253157448 podStartE2EDuration="2.253157448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:26:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:27:00.225448442 +0000 UTC m=+2353.557186093" watchObservedRunningTime="2026-11-01 16:27:00.253157448 +0000 UTC m=+2353.584895141" Nov 01 16:27:00.803127 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:00.803093 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36f441c7-2672-4581-9a0e-7142326e172b" path="/var/lib/kubelet/pods/36f441c7-2672-4581-9a0e-7142326e172b/volumes" Nov 01 16:27:01.153975 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:01.153945 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-e1e630/ipp-route-e1e630-kserve-85dbb46dd5-qqv5q" Nov 01 16:27:01.430804 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:01.430726 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j"] Nov 01 16:27:01.431088 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:01.431053 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" podUID="9427e75b-2800-4bee-8711-b1e1306d8f09" containerName="istio-proxy" containerID="cri-o://89fc859a7cce033af5e69edd157263ee4c5548312cf3b3998a6b2c01d2e6bb2d" gracePeriod=30 Nov 01 16:27:04.141003 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:04.140974 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-675cc9cb59-vmxp4" Nov 01 16:27:05.228667 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:05.228621 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq"] Nov 01 16:27:05.236711 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:05.236681 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" Nov 01 16:27:05.238054 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:05.237991 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq"] Nov 01 16:27:05.240001 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:05.239976 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-7321d6-openshift-default-dockercfg-tk2gq\"" Nov 01 16:27:05.278257 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:05.278222 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-istio-podinfo\") pod \"e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq\" (UID: \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\") " pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" Nov 01 16:27:05.278644 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:05.278615 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-workload-socket\") pod \"e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq\" (UID: \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\") " pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" Nov 01 16:27:05.278923 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:05.278896 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-istio-token\") pod \"e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq\" (UID: \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\") " pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" Nov 01 16:27:05.279053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:05.278953 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-credential-socket\") pod \"e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq\" (UID: \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\") " pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" Nov 01 16:27:05.279053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:05.279035 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-istio-data\") pod \"e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq\" (UID: \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\") " pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" Nov 01 16:27:05.279163 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:05.279091 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-istiod-ca-cert\") pod \"e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq\" (UID: \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\") " pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" Nov 01 16:27:05.279275 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:05.279255 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf5dk\" (UniqueName: \"kubernetes.io/projected/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-kube-api-access-wf5dk\") pod \"e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq\" (UID: \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\") " pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" Nov 01 16:27:05.279331 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:05.279295 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-istio-envoy\") pod \"e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq\" (UID: \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\") " pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" Nov 01 16:27:05.279394 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:05.279379 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-workload-certs\") pod \"e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq\" (UID: \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\") " pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" Nov 01 16:27:05.471043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:05.466724 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf5dk\" (UniqueName: \"kubernetes.io/projected/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-kube-api-access-wf5dk\") pod \"e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq\" (UID: \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\") " pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" Nov 01 16:27:05.471043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:05.466786 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-istio-envoy\") pod \"e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq\" (UID: \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\") " pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" Nov 01 16:27:05.471043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:05.466839 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-workload-certs\") pod \"e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq\" (UID: \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\") " pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" Nov 01 16:27:05.471043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:05.466962 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-istio-podinfo\") pod \"e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq\" (UID: \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\") " pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" Nov 01 16:27:05.471043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:05.467153 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-workload-socket\") pod \"e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq\" (UID: \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\") " pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" Nov 01 16:27:05.471043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:05.467295 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-istio-token\") pod \"e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq\" (UID: \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\") " pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" Nov 01 16:27:05.471043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:05.467331 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-credential-socket\") pod \"e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq\" (UID: \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\") " pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" Nov 01 16:27:05.471043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:05.467379 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-istio-data\") pod \"e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq\" (UID: \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\") " pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" Nov 01 16:27:05.471043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:05.467424 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-istiod-ca-cert\") pod \"e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq\" (UID: \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\") " pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" Nov 01 16:27:05.471043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:05.468339 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-istiod-ca-cert\") pod \"e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq\" (UID: \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\") " pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" Nov 01 16:27:05.525089 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:05.521562 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-workload-certs\") pod \"e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq\" (UID: \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\") " pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" Nov 01 16:27:05.557619 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:05.532243 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-credential-socket\") pod \"e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq\" (UID: \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\") " pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" Nov 01 16:27:05.557619 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:05.532566 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-workload-socket\") pod \"e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq\" (UID: \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\") " pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" Nov 01 16:27:05.557619 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:05.533482 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-istio-data\") pod \"e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq\" (UID: \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\") " pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" Nov 01 16:27:05.625042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:05.618516 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-istio-token\") pod \"e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq\" (UID: \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\") " pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" Nov 01 16:27:05.687041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:05.683274 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-istio-envoy\") pod \"e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq\" (UID: \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\") " pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" Nov 01 16:27:05.700694 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:05.687384 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf5dk\" (UniqueName: \"kubernetes.io/projected/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-kube-api-access-wf5dk\") pod \"e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq\" (UID: \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\") " pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" Nov 01 16:27:05.721587 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:05.720511 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-istio-podinfo\") pod \"e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq\" (UID: \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\") " pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" Nov 01 16:27:05.855116 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:05.855083 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" Nov 01 16:27:06.153845 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:27:06.153815 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod742547a4_fbd7_40d3_a7a3_86fc91fd7cc3.slice/crio-3259ffae98371937cf65c77d2581fdbf86aeb4dc1310cac0c6e090ffbeafb5c4 WatchSource:0}: Error finding container 3259ffae98371937cf65c77d2581fdbf86aeb4dc1310cac0c6e090ffbeafb5c4: Status 404 returned error can't find the container with id 3259ffae98371937cf65c77d2581fdbf86aeb4dc1310cac0c6e090ffbeafb5c4 Nov 01 16:27:06.153845 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:06.153833 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq"] Nov 01 16:27:06.156829 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:06.156798 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:27:06.156933 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:06.156867 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:27:06.156933 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:06.156898 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:27:06.262804 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:06.262766 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" event={"ID":"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3","Type":"ContainerStarted","Data":"2d271a9c18d6ff4575a67d700a8b05bbd8356e297a3de9b6e16758d5ff5242e2"} Nov 01 16:27:06.263261 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:06.262824 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" event={"ID":"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3","Type":"ContainerStarted","Data":"3259ffae98371937cf65c77d2581fdbf86aeb4dc1310cac0c6e090ffbeafb5c4"} Nov 01 16:27:06.288065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:06.287104 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" podStartSLOduration=1.287084271 podStartE2EDuration="1.287084271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:27:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:27:06.281054134 +0000 UTC m=+2359.612791784" watchObservedRunningTime="2026-11-01 16:27:06.287084271 +0000 UTC m=+2359.618821921" Nov 01 16:27:06.840227 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:06.840202 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" Nov 01 16:27:06.855352 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:06.855329 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" Nov 01 16:27:06.942669 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:06.942635 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9427e75b-2800-4bee-8711-b1e1306d8f09-istio-token\" (UniqueName: \"kubernetes.io/projected/9427e75b-2800-4bee-8711-b1e1306d8f09-istio-token\") pod \"9427e75b-2800-4bee-8711-b1e1306d8f09\" (UID: \"9427e75b-2800-4bee-8711-b1e1306d8f09\") " Nov 01 16:27:06.942857 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:06.942719 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9427e75b-2800-4bee-8711-b1e1306d8f09-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9427e75b-2800-4bee-8711-b1e1306d8f09-istio-podinfo\") pod \"9427e75b-2800-4bee-8711-b1e1306d8f09\" (UID: \"9427e75b-2800-4bee-8711-b1e1306d8f09\") " Nov 01 16:27:06.942857 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:06.942750 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9427e75b-2800-4bee-8711-b1e1306d8f09-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9427e75b-2800-4bee-8711-b1e1306d8f09-istiod-ca-cert\") pod \"9427e75b-2800-4bee-8711-b1e1306d8f09\" (UID: \"9427e75b-2800-4bee-8711-b1e1306d8f09\") " Nov 01 16:27:06.942857 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:06.942811 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9427e75b-2800-4bee-8711-b1e1306d8f09-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9427e75b-2800-4bee-8711-b1e1306d8f09-workload-certs\") pod \"9427e75b-2800-4bee-8711-b1e1306d8f09\" (UID: \"9427e75b-2800-4bee-8711-b1e1306d8f09\") " Nov 01 16:27:06.942857 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:06.942853 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9427e75b-2800-4bee-8711-b1e1306d8f09-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9427e75b-2800-4bee-8711-b1e1306d8f09-istio-data\") pod \"9427e75b-2800-4bee-8711-b1e1306d8f09\" (UID: \"9427e75b-2800-4bee-8711-b1e1306d8f09\") " Nov 01 16:27:06.943167 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:06.942906 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9427e75b-2800-4bee-8711-b1e1306d8f09-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9427e75b-2800-4bee-8711-b1e1306d8f09-istio-envoy\") pod \"9427e75b-2800-4bee-8711-b1e1306d8f09\" (UID: \"9427e75b-2800-4bee-8711-b1e1306d8f09\") " Nov 01 16:27:06.943761 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:06.943290 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9427e75b-2800-4bee-8711-b1e1306d8f09-workload-certs" pod "9427e75b-2800-4bee-8711-b1e1306d8f09" (UID: "9427e75b-2800-4bee-8711-b1e1306d8f09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:27:06.943761 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:06.943298 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9427e75b-2800-4bee-8711-b1e1306d8f09-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9427e75b-2800-4bee-8711-b1e1306d8f09-credential-socket\") pod \"9427e75b-2800-4bee-8711-b1e1306d8f09\" (UID: \"9427e75b-2800-4bee-8711-b1e1306d8f09\") " Nov 01 16:27:06.943761 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:06.943479 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9427e75b-2800-4bee-8711-b1e1306d8f09-istio-data" pod "9427e75b-2800-4bee-8711-b1e1306d8f09" (UID: "9427e75b-2800-4bee-8711-b1e1306d8f09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:27:06.943761 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:06.943502 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9427e75b-2800-4bee-8711-b1e1306d8f09-kube-api-access-pnxjl\" (UniqueName: \"kubernetes.io/projected/9427e75b-2800-4bee-8711-b1e1306d8f09-kube-api-access-pnxjl\") pod \"9427e75b-2800-4bee-8711-b1e1306d8f09\" (UID: \"9427e75b-2800-4bee-8711-b1e1306d8f09\") " Nov 01 16:27:06.943761 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:06.943642 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9427e75b-2800-4bee-8711-b1e1306d8f09-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9427e75b-2800-4bee-8711-b1e1306d8f09-workload-socket\") pod \"9427e75b-2800-4bee-8711-b1e1306d8f09\" (UID: \"9427e75b-2800-4bee-8711-b1e1306d8f09\") " Nov 01 16:27:06.943761 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:06.943666 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9427e75b-2800-4bee-8711-b1e1306d8f09-istiod-ca-cert" pod "9427e75b-2800-4bee-8711-b1e1306d8f09" (UID: "9427e75b-2800-4bee-8711-b1e1306d8f09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:27:06.943761 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:06.943744 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9427e75b-2800-4bee-8711-b1e1306d8f09-credential-socket" pod "9427e75b-2800-4bee-8711-b1e1306d8f09" (UID: "9427e75b-2800-4bee-8711-b1e1306d8f09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:27:06.944234 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:06.944036 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9427e75b-2800-4bee-8711-b1e1306d8f09-workload-socket" pod "9427e75b-2800-4bee-8711-b1e1306d8f09" (UID: "9427e75b-2800-4bee-8711-b1e1306d8f09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:27:06.945149 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:06.945128 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9427e75b-2800-4bee-8711-b1e1306d8f09-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:06.945149 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:06.945151 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9427e75b-2800-4bee-8711-b1e1306d8f09-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:06.945297 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:06.945165 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9427e75b-2800-4bee-8711-b1e1306d8f09-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:06.945297 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:06.945178 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9427e75b-2800-4bee-8711-b1e1306d8f09-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:06.945297 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:06.945190 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9427e75b-2800-4bee-8711-b1e1306d8f09-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:06.947252 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:06.947227 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9427e75b-2800-4bee-8711-b1e1306d8f09-istio-podinfo" pod "9427e75b-2800-4bee-8711-b1e1306d8f09" (UID: "9427e75b-2800-4bee-8711-b1e1306d8f09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:27:06.949538 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:06.949492 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9427e75b-2800-4bee-8711-b1e1306d8f09-istio-token" pod "9427e75b-2800-4bee-8711-b1e1306d8f09" (UID: "9427e75b-2800-4bee-8711-b1e1306d8f09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:27:06.950795 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:06.950772 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9427e75b-2800-4bee-8711-b1e1306d8f09-kube-api-access-pnxjl" pod "9427e75b-2800-4bee-8711-b1e1306d8f09" (UID: "9427e75b-2800-4bee-8711-b1e1306d8f09"). InnerVolumeSpecName "kube-api-access-pnxjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:27:06.951089 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:06.951061 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9427e75b-2800-4bee-8711-b1e1306d8f09-istio-envoy" pod "9427e75b-2800-4bee-8711-b1e1306d8f09" (UID: "9427e75b-2800-4bee-8711-b1e1306d8f09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:27:07.046503 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:07.046471 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9427e75b-2800-4bee-8711-b1e1306d8f09-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:07.046503 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:07.046504 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9427e75b-2800-4bee-8711-b1e1306d8f09-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:07.046747 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:07.046517 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9427e75b-2800-4bee-8711-b1e1306d8f09-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:07.046747 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:07.046530 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnxjl\" (UniqueName: \"kubernetes.io/projected/9427e75b-2800-4bee-8711-b1e1306d8f09-kube-api-access-pnxjl\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:07.272666 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:07.272633 4363 generic.go:356] "Generic (PLEG): container finished" podID="9427e75b-2800-4bee-8711-b1e1306d8f09" containerID="89fc859a7cce033af5e69edd157263ee4c5548312cf3b3998a6b2c01d2e6bb2d" exitCode=0 Nov 01 16:27:07.273108 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:07.272680 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" Nov 01 16:27:07.273108 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:07.272712 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" event={"ID":"9427e75b-2800-4bee-8711-b1e1306d8f09","Type":"ContainerDied","Data":"89fc859a7cce033af5e69edd157263ee4c5548312cf3b3998a6b2c01d2e6bb2d"} Nov 01 16:27:07.273108 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:07.272766 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j" event={"ID":"9427e75b-2800-4bee-8711-b1e1306d8f09","Type":"ContainerDied","Data":"d4f1854c1dafda38865c6222182fb180267ff10077b433d0cc52b51669d6b24e"} Nov 01 16:27:07.273108 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:07.272787 4363 scope.go:122] "RemoveContainer" containerID="89fc859a7cce033af5e69edd157263ee4c5548312cf3b3998a6b2c01d2e6bb2d" Nov 01 16:27:07.285239 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:07.284324 4363 scope.go:122] "RemoveContainer" containerID="89fc859a7cce033af5e69edd157263ee4c5548312cf3b3998a6b2c01d2e6bb2d" Nov 01 16:27:07.285239 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:27:07.284639 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89fc859a7cce033af5e69edd157263ee4c5548312cf3b3998a6b2c01d2e6bb2d\": container with ID starting with 89fc859a7cce033af5e69edd157263ee4c5548312cf3b3998a6b2c01d2e6bb2d not found: ID does not exist" containerID="89fc859a7cce033af5e69edd157263ee4c5548312cf3b3998a6b2c01d2e6bb2d" Nov 01 16:27:07.285239 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:07.284672 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89fc859a7cce033af5e69edd157263ee4c5548312cf3b3998a6b2c01d2e6bb2d"} err="failed to get container status \"89fc859a7cce033af5e69edd157263ee4c5548312cf3b3998a6b2c01d2e6bb2d\": rpc error: code = NotFound desc = could not find container \"89fc859a7cce033af5e69edd157263ee4c5548312cf3b3998a6b2c01d2e6bb2d\": container with ID starting with 89fc859a7cce033af5e69edd157263ee4c5548312cf3b3998a6b2c01d2e6bb2d not found: ID does not exist" Nov 01 16:27:07.301860 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:07.301809 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j"] Nov 01 16:27:07.308678 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:07.308650 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-tnj4j"] Nov 01 16:27:07.862333 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:07.862303 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" Nov 01 16:27:08.283874 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:08.283778 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" Nov 01 16:27:08.284968 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:08.284943 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" Nov 01 16:27:08.802232 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:08.802194 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9427e75b-2800-4bee-8711-b1e1306d8f09" path="/var/lib/kubelet/pods/9427e75b-2800-4bee-8711-b1e1306d8f09/volumes" Nov 01 16:27:09.153928 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:09.153885 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-e1e630/ipp-route-e1e630-kserve-85dbb46dd5-qqv5q" Nov 01 16:27:09.167174 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:09.167142 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-e1e630/ipp-route-e1e630-kserve-85dbb46dd5-qqv5q" Nov 01 16:27:09.303918 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:09.303882 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-e1e630/ipp-route-e1e630-kserve-85dbb46dd5-qqv5q" Nov 01 16:27:09.996515 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:09.996480 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-64bfb77c47-v7d22"] Nov 01 16:27:09.997333 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:09.997316 4363 cpu_manager.go:414] "containerMap: removing container" podUID="9427e75b-2800-4bee-8711-b1e1306d8f09" containerName="istio-proxy" Nov 01 16:27:09.997427 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:09.997334 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9427e75b-2800-4bee-8711-b1e1306d8f09" containerName="istio-proxy" Nov 01 16:27:09.997464 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:09.997457 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="9427e75b-2800-4bee-8711-b1e1306d8f09" containerName="istio-proxy" Nov 01 16:27:10.002698 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:10.002677 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-64bfb77c47-v7d22" Nov 01 16:27:10.012570 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:10.012539 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-64bfb77c47-v7d22"] Nov 01 16:27:10.014417 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:10.013721 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-7321d6\"" Nov 01 16:27:10.014417 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:10.013848 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-7321d6-dockercfg-4b9cp\"" Nov 01 16:27:10.037244 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:10.037215 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7321d6-55df99b949-vs6jc"] Nov 01 16:27:10.043710 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:10.043679 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7321d6-55df99b949-vs6jc" Nov 01 16:27:10.050304 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:10.050230 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7321d6-55df99b949-vs6jc"] Nov 01 16:27:10.291094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:10.275379 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58jxc\" (UniqueName: \"kubernetes.io/projected/8b90f058-e42e-4b11-974b-2e22a08d7515-kube-api-access-58jxc\") pod \"payload-pre-processing-e2e-ipp-b-7321d6-64bfb77c47-v7d22\" (UID: \"8b90f058-e42e-4b11-974b-2e22a08d7515\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-64bfb77c47-v7d22" Nov 01 16:27:10.291094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:10.275443 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b90f058-e42e-4b11-974b-2e22a08d7515-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7321d6-64bfb77c47-v7d22\" (UID: \"8b90f058-e42e-4b11-974b-2e22a08d7515\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-64bfb77c47-v7d22" Nov 01 16:27:10.291094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:10.275617 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b288c39-87b8-441d-a28f-0a47c01e7d2c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7321d6-55df99b949-vs6jc\" (UID: \"8b288c39-87b8-441d-a28f-0a47c01e7d2c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7321d6-55df99b949-vs6jc" Nov 01 16:27:10.291094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:10.275745 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djn2q\" (UniqueName: \"kubernetes.io/projected/8b288c39-87b8-441d-a28f-0a47c01e7d2c-kube-api-access-djn2q\") pod \"payload-processing-e2e-ipp-b-7321d6-55df99b949-vs6jc\" (UID: \"8b288c39-87b8-441d-a28f-0a47c01e7d2c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7321d6-55df99b949-vs6jc" Nov 01 16:27:10.377749 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:10.377713 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djn2q\" (UniqueName: \"kubernetes.io/projected/8b288c39-87b8-441d-a28f-0a47c01e7d2c-kube-api-access-djn2q\") pod \"payload-processing-e2e-ipp-b-7321d6-55df99b949-vs6jc\" (UID: \"8b288c39-87b8-441d-a28f-0a47c01e7d2c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7321d6-55df99b949-vs6jc" Nov 01 16:27:10.378534 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:10.378506 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58jxc\" (UniqueName: \"kubernetes.io/projected/8b90f058-e42e-4b11-974b-2e22a08d7515-kube-api-access-58jxc\") pod \"payload-pre-processing-e2e-ipp-b-7321d6-64bfb77c47-v7d22\" (UID: \"8b90f058-e42e-4b11-974b-2e22a08d7515\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-64bfb77c47-v7d22" Nov 01 16:27:10.378670 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:10.378657 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b90f058-e42e-4b11-974b-2e22a08d7515-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7321d6-64bfb77c47-v7d22\" (UID: \"8b90f058-e42e-4b11-974b-2e22a08d7515\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-64bfb77c47-v7d22" Nov 01 16:27:10.387041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:10.378886 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b288c39-87b8-441d-a28f-0a47c01e7d2c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7321d6-55df99b949-vs6jc\" (UID: \"8b288c39-87b8-441d-a28f-0a47c01e7d2c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7321d6-55df99b949-vs6jc" Nov 01 16:27:10.387454 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:10.384526 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b90f058-e42e-4b11-974b-2e22a08d7515-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7321d6-64bfb77c47-v7d22\" (UID: \"8b90f058-e42e-4b11-974b-2e22a08d7515\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-64bfb77c47-v7d22" Nov 01 16:27:10.389072 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:10.389045 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b288c39-87b8-441d-a28f-0a47c01e7d2c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7321d6-55df99b949-vs6jc\" (UID: \"8b288c39-87b8-441d-a28f-0a47c01e7d2c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7321d6-55df99b949-vs6jc" Nov 01 16:27:10.431255 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:10.430808 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djn2q\" (UniqueName: \"kubernetes.io/projected/8b288c39-87b8-441d-a28f-0a47c01e7d2c-kube-api-access-djn2q\") pod \"payload-processing-e2e-ipp-b-7321d6-55df99b949-vs6jc\" (UID: \"8b288c39-87b8-441d-a28f-0a47c01e7d2c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7321d6-55df99b949-vs6jc" Nov 01 16:27:10.435181 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:10.434983 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58jxc\" (UniqueName: \"kubernetes.io/projected/8b90f058-e42e-4b11-974b-2e22a08d7515-kube-api-access-58jxc\") pod \"payload-pre-processing-e2e-ipp-b-7321d6-64bfb77c47-v7d22\" (UID: \"8b90f058-e42e-4b11-974b-2e22a08d7515\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-64bfb77c47-v7d22" Nov 01 16:27:10.636271 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:10.636221 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-64bfb77c47-v7d22" Nov 01 16:27:10.662287 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:10.662255 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7321d6-55df99b949-vs6jc" Nov 01 16:27:11.133616 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:11.133523 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-64bfb77c47-v7d22"] Nov 01 16:27:11.236384 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:11.234940 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7321d6-55df99b949-vs6jc"] Nov 01 16:27:11.639593 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:11.639549 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-64bfb77c47-v7d22" event={"ID":"8b90f058-e42e-4b11-974b-2e22a08d7515","Type":"ContainerStarted","Data":"694c40351522f1ec664a9fe04dc8b04032fe4fca939dddbd5116c04299add602"} Nov 01 16:27:11.645623 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:11.645594 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-64bfb77c47-v7d22" Nov 01 16:27:11.658040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:11.653994 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7321d6-55df99b949-vs6jc" event={"ID":"8b288c39-87b8-441d-a28f-0a47c01e7d2c","Type":"ContainerStarted","Data":"0174a06b9403c7d549572fba5c5eebcf783103df16edec0dd98a40d3a490583c"} Nov 01 16:27:11.658292 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:11.658266 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7321d6-55df99b949-vs6jc" event={"ID":"8b288c39-87b8-441d-a28f-0a47c01e7d2c","Type":"ContainerStarted","Data":"765a03a6d78d574ad64c39878dd7a62a1af3452652ae6ce9df82f0b720546a16"} Nov 01 16:27:11.658684 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:11.658666 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7321d6-55df99b949-vs6jc" Nov 01 16:27:11.769041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:11.749530 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-7321d6-55df99b949-vs6jc" podStartSLOduration=1.7495122570000001 podStartE2EDuration="1.749512257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:27:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:27:11.748432169 +0000 UTC m=+2365.080169819" watchObservedRunningTime="2026-11-01 16:27:11.749512257 +0000 UTC m=+2365.081249908" Nov 01 16:27:11.769041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:11.750789 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-64bfb77c47-v7d22" podStartSLOduration=2.750778841 podStartE2EDuration="2.750778841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:27:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:27:11.688255326 +0000 UTC m=+2365.019992978" watchObservedRunningTime="2026-11-01 16:27:11.750778841 +0000 UTC m=+2365.082516492" Nov 01 16:27:12.708053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:12.706448 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:27:12.708053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:12.706871 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:27:12.728040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:12.718362 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:27:12.768048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:12.760621 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-64bfb77c47-v7d22" event={"ID":"8b90f058-e42e-4b11-974b-2e22a08d7515","Type":"ContainerStarted","Data":"a291a45b04901ff68526a67d754d2c16eb2eeb11a9776c3191a70d864c066445"} Nov 01 16:27:13.148035 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:13.147779 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:27:13.383046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:13.377292 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp" podUID="f8ca3021-8cbd-4cfc-aab7-33bb656f4c01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.225:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:27:13.945953 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:13.945897 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:27:15.572080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:15.571960 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6960-f06d82"] Nov 01 16:27:15.581040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:15.577820 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6960-f06d82" Nov 01 16:27:15.602041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:15.594370 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hqwjk\"" Nov 01 16:27:15.661104 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:15.661072 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6960-f06d82"] Nov 01 16:27:15.734160 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:15.687854 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cg6d\" (UniqueName: \"kubernetes.io/projected/1a90298b-67d5-43aa-942d-01bbb1d7f953-kube-api-access-9cg6d\") pod \"test-curl-6960-f06d82\" (UID: \"1a90298b-67d5-43aa-942d-01bbb1d7f953\") " pod="opendatahub/test-curl-6960-f06d82" Nov 01 16:27:15.840094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:15.839805 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cg6d\" (UniqueName: \"kubernetes.io/projected/1a90298b-67d5-43aa-942d-01bbb1d7f953-kube-api-access-9cg6d\") pod \"test-curl-6960-f06d82\" (UID: \"1a90298b-67d5-43aa-942d-01bbb1d7f953\") " pod="opendatahub/test-curl-6960-f06d82" Nov 01 16:27:15.848262 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:15.848230 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cg6d\" (UniqueName: \"kubernetes.io/projected/1a90298b-67d5-43aa-942d-01bbb1d7f953-kube-api-access-9cg6d\") pod \"test-curl-6960-f06d82\" (UID: \"1a90298b-67d5-43aa-942d-01bbb1d7f953\") " pod="opendatahub/test-curl-6960-f06d82" Nov 01 16:27:15.950545 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:15.950514 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6960-f06d82" Nov 01 16:27:16.962954 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:16.962913 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6960-f06d82"] Nov 01 16:27:16.964705 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:27:16.964681 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a90298b_67d5_43aa_942d_01bbb1d7f953.slice/crio-33f8cd14af4623fce0255c866a9a339d5b904b0a485c804a2f84f271aa129c56 WatchSource:0}: Error finding container 33f8cd14af4623fce0255c866a9a339d5b904b0a485c804a2f84f271aa129c56: Status 404 returned error can't find the container with id 33f8cd14af4623fce0255c866a9a339d5b904b0a485c804a2f84f271aa129c56 Nov 01 16:27:17.951826 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:17.950318 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6960-f06d82" event={"ID":"1a90298b-67d5-43aa-942d-01bbb1d7f953","Type":"ContainerStarted","Data":"7c0f20ae0d6a1a22a12f8f7a460b51e64f2f0c5656372ff7f22b238bfe7449d2"} Nov 01 16:27:17.951826 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:17.950376 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6960-f06d82" event={"ID":"1a90298b-67d5-43aa-942d-01bbb1d7f953","Type":"ContainerStarted","Data":"33f8cd14af4623fce0255c866a9a339d5b904b0a485c804a2f84f271aa129c56"} Nov 01 16:27:17.973812 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:17.973747 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6960-f06d82" podStartSLOduration=2.973727941 podStartE2EDuration="2.973727941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:27:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:27:17.972465081 +0000 UTC m=+2371.304202735" watchObservedRunningTime="2026-11-01 16:27:17.973727941 +0000 UTC m=+2371.305465591" Nov 01 16:27:18.369736 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:27:18.369697 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:27:18.870363 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:18.870333 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6960-f06d82"] Nov 01 16:27:18.870577 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:18.870560 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6960-f06d82"] Nov 01 16:27:18.976825 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:18.976798 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6960-1dc305"] Nov 01 16:27:19.031042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:19.005915 4363 cpu_manager.go:414] "containerMap: removing container" podUID="1a90298b-67d5-43aa-942d-01bbb1d7f953" containerName="test-curl-6960-f06d82" Nov 01 16:27:19.031042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:19.005953 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a90298b-67d5-43aa-942d-01bbb1d7f953" containerName="test-curl-6960-f06d82" Nov 01 16:27:19.031042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:19.006170 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a90298b-67d5-43aa-942d-01bbb1d7f953" containerName="test-curl-6960-f06d82" Nov 01 16:27:19.062395 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:19.062362 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6960-1dc305"] Nov 01 16:27:19.062766 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:19.062746 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6960-1dc305" Nov 01 16:27:19.247041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:19.245530 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd2gb\" (UniqueName: \"kubernetes.io/projected/7dcafa96-b4da-487e-b2c5-da5f29b3c70d-kube-api-access-rd2gb\") pod \"test-curl-6960-1dc305\" (UID: \"7dcafa96-b4da-487e-b2c5-da5f29b3c70d\") " pod="opendatahub/test-curl-6960-1dc305" Nov 01 16:27:19.347255 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:19.347215 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd2gb\" (UniqueName: \"kubernetes.io/projected/7dcafa96-b4da-487e-b2c5-da5f29b3c70d-kube-api-access-rd2gb\") pod \"test-curl-6960-1dc305\" (UID: \"7dcafa96-b4da-487e-b2c5-da5f29b3c70d\") " pod="opendatahub/test-curl-6960-1dc305" Nov 01 16:27:19.367733 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:19.367695 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd2gb\" (UniqueName: \"kubernetes.io/projected/7dcafa96-b4da-487e-b2c5-da5f29b3c70d-kube-api-access-rd2gb\") pod \"test-curl-6960-1dc305\" (UID: \"7dcafa96-b4da-487e-b2c5-da5f29b3c70d\") " pod="opendatahub/test-curl-6960-1dc305" Nov 01 16:27:19.495741 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:19.495644 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6960-1dc305" Nov 01 16:27:19.795671 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:19.795646 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6960-1dc305"] Nov 01 16:27:19.799845 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:27:19.799791 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7dcafa96_b4da_487e_b2c5_da5f29b3c70d.slice/crio-a8c99a81af07dc1e8fbd355345088df9c82817e1dbd28ad07cc681b678efb3a3 WatchSource:0}: Error finding container a8c99a81af07dc1e8fbd355345088df9c82817e1dbd28ad07cc681b678efb3a3: Status 404 returned error can't find the container with id a8c99a81af07dc1e8fbd355345088df9c82817e1dbd28ad07cc681b678efb3a3 Nov 01 16:27:19.972823 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:19.972790 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6960-1dc305" event={"ID":"7dcafa96-b4da-487e-b2c5-da5f29b3c70d","Type":"ContainerStarted","Data":"016182e8b167e21ff1cb92cd35fbe110807b897cd8e5eeea50cf8d85c0f37d90"} Nov 01 16:27:19.972974 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:19.972831 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6960-1dc305" event={"ID":"7dcafa96-b4da-487e-b2c5-da5f29b3c70d","Type":"ContainerStarted","Data":"a8c99a81af07dc1e8fbd355345088df9c82817e1dbd28ad07cc681b678efb3a3"} Nov 01 16:27:19.973096 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:19.973067 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6960-f06d82" podUID="1a90298b-67d5-43aa-942d-01bbb1d7f953" containerName="test-curl-6960-f06d82" containerID="cri-o://7c0f20ae0d6a1a22a12f8f7a460b51e64f2f0c5656372ff7f22b238bfe7449d2" gracePeriod=2 Nov 01 16:27:19.986644 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:27:19.986614 4363 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6960-f06d82\" is forbidden: User \"system:node:ip-10-0-128-238.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-238.us-east-2.compute.internal' and this object" podUID="1a90298b-67d5-43aa-942d-01bbb1d7f953" pod="opendatahub/test-curl-6960-f06d82" Nov 01 16:27:19.988871 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:19.988819 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6960-1dc305" podStartSLOduration=1.988802568 podStartE2EDuration="1.988802568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:27:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:27:19.984742504 +0000 UTC m=+2373.316480155" watchObservedRunningTime="2026-11-01 16:27:19.988802568 +0000 UTC m=+2373.320540219" Nov 01 16:27:20.670037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:20.659071 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6960-1dc305"] Nov 01 16:27:20.689035 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:20.678300 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6960-1dc305"] Nov 01 16:27:22.016314 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:22.016280 4363 generic.go:356] "Generic (PLEG): container finished" podID="1a90298b-67d5-43aa-942d-01bbb1d7f953" containerID="7c0f20ae0d6a1a22a12f8f7a460b51e64f2f0c5656372ff7f22b238bfe7449d2" exitCode=137 Nov 01 16:27:22.016871 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:22.016831 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6960-1dc305" podUID="7dcafa96-b4da-487e-b2c5-da5f29b3c70d" containerName="test-curl-6960-1dc305" containerID="cri-o://016182e8b167e21ff1cb92cd35fbe110807b897cd8e5eeea50cf8d85c0f37d90" gracePeriod=2 Nov 01 16:27:22.215104 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:22.215072 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6960-f06d82" Nov 01 16:27:22.330260 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:22.330177 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a90298b-67d5-43aa-942d-01bbb1d7f953-kube-api-access-9cg6d\" (UniqueName: \"kubernetes.io/projected/1a90298b-67d5-43aa-942d-01bbb1d7f953-kube-api-access-9cg6d\") pod \"1a90298b-67d5-43aa-942d-01bbb1d7f953\" (UID: \"1a90298b-67d5-43aa-942d-01bbb1d7f953\") " Nov 01 16:27:22.332384 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:22.332348 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a90298b-67d5-43aa-942d-01bbb1d7f953-kube-api-access-9cg6d" pod "1a90298b-67d5-43aa-942d-01bbb1d7f953" (UID: "1a90298b-67d5-43aa-942d-01bbb1d7f953"). InnerVolumeSpecName "kube-api-access-9cg6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:27:22.431979 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:22.431941 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cg6d\" (UniqueName: \"kubernetes.io/projected/1a90298b-67d5-43aa-942d-01bbb1d7f953-kube-api-access-9cg6d\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:22.763146 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:22.763111 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7321d6-55df99b949-vs6jc" Nov 01 16:27:22.802437 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:22.802399 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a90298b-67d5-43aa-942d-01bbb1d7f953" path="/var/lib/kubelet/pods/1a90298b-67d5-43aa-942d-01bbb1d7f953/volumes" Nov 01 16:27:23.028554 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:23.028467 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6960-f06d82" Nov 01 16:27:23.028554 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:23.028511 4363 scope.go:122] "RemoveContainer" containerID="7c0f20ae0d6a1a22a12f8f7a460b51e64f2f0c5656372ff7f22b238bfe7449d2" Nov 01 16:27:24.988550 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:24.354186 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6960-1dc305" Nov 01 16:27:24.988550 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:24.372995 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dcafa96-b4da-487e-b2c5-da5f29b3c70d-kube-api-access-rd2gb\" (UniqueName: \"kubernetes.io/projected/7dcafa96-b4da-487e-b2c5-da5f29b3c70d-kube-api-access-rd2gb\") pod \"7dcafa96-b4da-487e-b2c5-da5f29b3c70d\" (UID: \"7dcafa96-b4da-487e-b2c5-da5f29b3c70d\") " Nov 01 16:27:24.988550 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:24.375225 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dcafa96-b4da-487e-b2c5-da5f29b3c70d-kube-api-access-rd2gb" pod "7dcafa96-b4da-487e-b2c5-da5f29b3c70d" (UID: "7dcafa96-b4da-487e-b2c5-da5f29b3c70d"). InnerVolumeSpecName "kube-api-access-rd2gb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:27:24.988550 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:24.475123 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd2gb\" (UniqueName: \"kubernetes.io/projected/7dcafa96-b4da-487e-b2c5-da5f29b3c70d-kube-api-access-rd2gb\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:24.988550 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:24.804845 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dcafa96-b4da-487e-b2c5-da5f29b3c70d" path="/var/lib/kubelet/pods/7dcafa96-b4da-487e-b2c5-da5f29b3c70d/volumes" Nov 01 16:27:24.988550 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:24.866693 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-64bfb77c47-v7d22" Nov 01 16:27:25.078464 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:25.078423 4363 generic.go:356] "Generic (PLEG): container finished" podID="7dcafa96-b4da-487e-b2c5-da5f29b3c70d" containerID="016182e8b167e21ff1cb92cd35fbe110807b897cd8e5eeea50cf8d85c0f37d90" exitCode=137 Nov 01 16:27:25.078671 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:25.078606 4363 scope.go:122] "RemoveContainer" containerID="016182e8b167e21ff1cb92cd35fbe110807b897cd8e5eeea50cf8d85c0f37d90" Nov 01 16:27:25.078734 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:25.078724 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6960-1dc305" Nov 01 16:27:25.092722 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:25.092698 4363 scope.go:122] "RemoveContainer" containerID="016182e8b167e21ff1cb92cd35fbe110807b897cd8e5eeea50cf8d85c0f37d90" Nov 01 16:27:25.093066 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:27:25.093005 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"016182e8b167e21ff1cb92cd35fbe110807b897cd8e5eeea50cf8d85c0f37d90\": container with ID starting with 016182e8b167e21ff1cb92cd35fbe110807b897cd8e5eeea50cf8d85c0f37d90 not found: ID does not exist" containerID="016182e8b167e21ff1cb92cd35fbe110807b897cd8e5eeea50cf8d85c0f37d90" Nov 01 16:27:25.093120 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:25.093081 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"016182e8b167e21ff1cb92cd35fbe110807b897cd8e5eeea50cf8d85c0f37d90"} err="failed to get container status \"016182e8b167e21ff1cb92cd35fbe110807b897cd8e5eeea50cf8d85c0f37d90\": rpc error: code = NotFound desc = could not find container \"016182e8b167e21ff1cb92cd35fbe110807b897cd8e5eeea50cf8d85c0f37d90\": container with ID starting with 016182e8b167e21ff1cb92cd35fbe110807b897cd8e5eeea50cf8d85c0f37d90 not found: ID does not exist" Nov 01 16:27:26.090731 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:26.090691 4363 generic.go:356] "Generic (PLEG): container finished" podID="be90dd72-5635-4115-9826-26aaa46f70dc" containerID="61ea40fc745b3bb1015069215985f45711621cb794c7c73809efd6112fc793f0" exitCode=0 Nov 01 16:27:26.091290 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:26.090732 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" event={"ID":"be90dd72-5635-4115-9826-26aaa46f70dc","Type":"ContainerDied","Data":"61ea40fc745b3bb1015069215985f45711621cb794c7c73809efd6112fc793f0"} Nov 01 16:27:26.091290 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:26.091251 4363 scope.go:122] "RemoveContainer" containerID="61ea40fc745b3bb1015069215985f45711621cb794c7c73809efd6112fc793f0" Nov 01 16:27:26.093060 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:26.093007 4363 generic.go:356] "Generic (PLEG): container finished" podID="3b5725f6-5b76-4300-9886-6a8c2556e6ae" containerID="c4f9204e651009e25d691e291968fd2ba29f08a6db5f84e35abef7f4019f7a31" exitCode=0 Nov 01 16:27:26.093192 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:26.093075 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" event={"ID":"3b5725f6-5b76-4300-9886-6a8c2556e6ae","Type":"ContainerDied","Data":"c4f9204e651009e25d691e291968fd2ba29f08a6db5f84e35abef7f4019f7a31"} Nov 01 16:27:26.093479 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:26.093451 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:27:26.093569 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:26.093500 4363 scope.go:122] "RemoveContainer" containerID="c4f9204e651009e25d691e291968fd2ba29f08a6db5f84e35abef7f4019f7a31" Nov 01 16:27:26.093569 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:26.093541 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:27:26.093700 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:26.093593 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:27:26.095794 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:26.095759 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:27:26.095898 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:26.095844 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:27:26.095898 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:26.095891 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:27:26.121418 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:26.121357 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" Nov 01 16:27:26.799083 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:26.778334 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" Nov 01 16:27:27.149261 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:27.149220 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" event={"ID":"be90dd72-5635-4115-9826-26aaa46f70dc","Type":"ContainerStarted","Data":"d1453bb6d55de12f27f4019e8f62d38775bd38d3192056af021f06cb55ab74a3"} Nov 01 16:27:27.156420 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:27.156388 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" event={"ID":"3b5725f6-5b76-4300-9886-6a8c2556e6ae","Type":"ContainerStarted","Data":"67be541f031c228b189f751595a8c551d35203f778f2bad6a1856122069c90e5"} Nov 01 16:27:27.595135 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:27.563350 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" Nov 01 16:27:27.595135 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:27.594789 4363 patch_prober.go:29] interesting pod/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Nov 01 16:27:27.595135 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:27.594842 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" podUID="3b5725f6-5b76-4300-9886-6a8c2556e6ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Nov 01 16:27:27.998247 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:27.998093 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" Nov 01 16:27:28.071041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:28.063142 4363 patch_prober.go:29] interesting pod/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body= Nov 01 16:27:28.071041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:28.063200 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" podUID="be90dd72-5635-4115-9826-26aaa46f70dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" Nov 01 16:27:28.563057 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:28.563007 4363 patch_prober.go:29] interesting pod/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Nov 01 16:27:28.563547 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:28.563081 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" podUID="3b5725f6-5b76-4300-9886-6a8c2556e6ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Nov 01 16:27:28.957050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:28.956762 4363 patch_prober.go:29] interesting pod/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body= Nov 01 16:27:28.957050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:28.956862 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" podUID="be90dd72-5635-4115-9826-26aaa46f70dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" Nov 01 16:27:29.700237 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:29.700195 4363 patch_prober.go:29] interesting pod/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Nov 01 16:27:29.700872 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:29.700839 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" podUID="3b5725f6-5b76-4300-9886-6a8c2556e6ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Nov 01 16:27:29.955837 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:29.954776 4363 patch_prober.go:29] interesting pod/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body= Nov 01 16:27:29.955837 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:29.954859 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" podUID="be90dd72-5635-4115-9826-26aaa46f70dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" Nov 01 16:27:30.563853 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:30.563817 4363 patch_prober.go:29] interesting pod/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Nov 01 16:27:30.564278 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:30.564247 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" podUID="3b5725f6-5b76-4300-9886-6a8c2556e6ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Nov 01 16:27:30.959958 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:30.959921 4363 patch_prober.go:29] interesting pod/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body= Nov 01 16:27:30.960547 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:30.960516 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" podUID="be90dd72-5635-4115-9826-26aaa46f70dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" Nov 01 16:27:31.336806 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:31.336710 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a101f7e1-7r2hq"] Nov 01 16:27:31.359486 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:31.358670 4363 cpu_manager.go:414] "containerMap: removing container" podUID="7dcafa96-b4da-487e-b2c5-da5f29b3c70d" containerName="test-curl-6960-1dc305" Nov 01 16:27:31.359486 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:31.358704 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dcafa96-b4da-487e-b2c5-da5f29b3c70d" containerName="test-curl-6960-1dc305" Nov 01 16:27:31.359486 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:31.358917 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dcafa96-b4da-487e-b2c5-da5f29b3c70d" containerName="test-curl-6960-1dc305" Nov 01 16:27:31.364036 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:31.363996 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a101f7e1-7r2hq"] Nov 01 16:27:31.364201 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:31.364128 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a101f7e1-7r2hq" Nov 01 16:27:31.367686 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:31.367666 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:27:31.368036 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:31.367779 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:27:31.570046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:31.557349 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slm49\" (UniqueName: \"kubernetes.io/projected/1ace069d-b1a6-4ca9-b630-2828fdd25be2-kube-api-access-slm49\") pod \"maas-api-revoke-sub-a101f7e1-7r2hq\" (UID: \"1ace069d-b1a6-4ca9-b630-2828fdd25be2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a101f7e1-7r2hq" Nov 01 16:27:31.570046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:31.557669 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ace069d-b1a6-4ca9-b630-2828fdd25be2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a101f7e1-7r2hq\" (UID: \"1ace069d-b1a6-4ca9-b630-2828fdd25be2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a101f7e1-7r2hq" Nov 01 16:27:31.570046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:31.558039 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ace069d-b1a6-4ca9-b630-2828fdd25be2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a101f7e1-7r2hq\" (UID: \"1ace069d-b1a6-4ca9-b630-2828fdd25be2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a101f7e1-7r2hq" Nov 01 16:27:31.587096 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:31.585888 4363 patch_prober.go:29] interesting pod/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Nov 01 16:27:31.587096 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:31.585952 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" podUID="3b5725f6-5b76-4300-9886-6a8c2556e6ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Nov 01 16:27:31.661041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:31.660864 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ace069d-b1a6-4ca9-b630-2828fdd25be2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a101f7e1-7r2hq\" (UID: \"1ace069d-b1a6-4ca9-b630-2828fdd25be2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a101f7e1-7r2hq" Nov 01 16:27:31.661255 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:31.661087 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slm49\" (UniqueName: \"kubernetes.io/projected/1ace069d-b1a6-4ca9-b630-2828fdd25be2-kube-api-access-slm49\") pod \"maas-api-revoke-sub-a101f7e1-7r2hq\" (UID: \"1ace069d-b1a6-4ca9-b630-2828fdd25be2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a101f7e1-7r2hq" Nov 01 16:27:31.661325 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:31.661273 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ace069d-b1a6-4ca9-b630-2828fdd25be2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a101f7e1-7r2hq\" (UID: \"1ace069d-b1a6-4ca9-b630-2828fdd25be2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a101f7e1-7r2hq" Nov 01 16:27:31.674358 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:31.674325 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ace069d-b1a6-4ca9-b630-2828fdd25be2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a101f7e1-7r2hq\" (UID: \"1ace069d-b1a6-4ca9-b630-2828fdd25be2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a101f7e1-7r2hq" Nov 01 16:27:31.682498 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:31.682466 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slm49\" (UniqueName: \"kubernetes.io/projected/1ace069d-b1a6-4ca9-b630-2828fdd25be2-kube-api-access-slm49\") pod \"maas-api-revoke-sub-a101f7e1-7r2hq\" (UID: \"1ace069d-b1a6-4ca9-b630-2828fdd25be2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a101f7e1-7r2hq" Nov 01 16:27:31.711573 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:31.711540 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ace069d-b1a6-4ca9-b630-2828fdd25be2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a101f7e1-7r2hq\" (UID: \"1ace069d-b1a6-4ca9-b630-2828fdd25be2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a101f7e1-7r2hq" Nov 01 16:27:31.751188 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:31.751157 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a101f7e1-7r2hq" Nov 01 16:27:32.018631 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:32.018145 4363 patch_prober.go:29] interesting pod/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body= Nov 01 16:27:32.018631 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:32.018219 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" podUID="be90dd72-5635-4115-9826-26aaa46f70dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" Nov 01 16:27:32.549203 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:32.549110 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a101f7e1-7r2hq"] Nov 01 16:27:32.562788 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:32.562760 4363 patch_prober.go:29] interesting pod/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Nov 01 16:27:32.562896 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:32.562818 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" podUID="3b5725f6-5b76-4300-9886-6a8c2556e6ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Nov 01 16:27:32.951042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:32.950604 4363 patch_prober.go:29] interesting pod/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body= Nov 01 16:27:32.951042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:32.950664 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" podUID="be90dd72-5635-4115-9826-26aaa46f70dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" Nov 01 16:27:33.562496 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:33.562431 4363 patch_prober.go:29] interesting pod/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Nov 01 16:27:33.562496 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:33.562488 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" podUID="3b5725f6-5b76-4300-9886-6a8c2556e6ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Nov 01 16:27:33.563276 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:33.563240 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a101f7e1-7r2hq" event={"ID":"1ace069d-b1a6-4ca9-b630-2828fdd25be2","Type":"ContainerStarted","Data":"c9f912e8dab655c9f6bfa3c9a5d582105b5df6bdffcd50c4c234320aa606a9f3"} Nov 01 16:27:33.563421 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:33.563304 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a101f7e1-7r2hq" event={"ID":"1ace069d-b1a6-4ca9-b630-2828fdd25be2","Type":"ContainerStarted","Data":"e6564a5612cfaf49460356531f1552b9e98b542ce0a897bfb6eb346501e7601a"} Nov 01 16:27:33.591921 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:33.591848 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a101f7e1-7r2hq" podStartSLOduration=2.5918236820000002 podStartE2EDuration="2.591823682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:27:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:27:33.578428769 +0000 UTC m=+2386.910166469" watchObservedRunningTime="2026-11-01 16:27:33.591823682 +0000 UTC m=+2386.923561332" Nov 01 16:27:34.008335 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:34.007053 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" Nov 01 16:27:34.008335 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:34.008176 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" Nov 01 16:27:34.008993 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:34.008966 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" Nov 01 16:27:34.568069 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:34.568037 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" Nov 01 16:27:34.568566 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:34.568546 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" Nov 01 16:27:34.569291 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:34.569267 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" Nov 01 16:27:34.573877 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:34.573854 4363 generic.go:356] "Generic (PLEG): container finished" podID="1ace069d-b1a6-4ca9-b630-2828fdd25be2" containerID="c9f912e8dab655c9f6bfa3c9a5d582105b5df6bdffcd50c4c234320aa606a9f3" exitCode=0 Nov 01 16:27:34.573996 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:34.573929 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a101f7e1-7r2hq" event={"ID":"1ace069d-b1a6-4ca9-b630-2828fdd25be2","Type":"ContainerDied","Data":"c9f912e8dab655c9f6bfa3c9a5d582105b5df6bdffcd50c4c234320aa606a9f3"} Nov 01 16:27:35.857276 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:35.857251 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a101f7e1-7r2hq" Nov 01 16:27:36.001425 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:36.001394 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ace069d-b1a6-4ca9-b630-2828fdd25be2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ace069d-b1a6-4ca9-b630-2828fdd25be2-maas-api-xks-ca\") pod \"1ace069d-b1a6-4ca9-b630-2828fdd25be2\" (UID: \"1ace069d-b1a6-4ca9-b630-2828fdd25be2\") " Nov 01 16:27:36.001651 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:36.001460 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ace069d-b1a6-4ca9-b630-2828fdd25be2-kube-api-access-slm49\" (UniqueName: \"kubernetes.io/projected/1ace069d-b1a6-4ca9-b630-2828fdd25be2-kube-api-access-slm49\") pod \"1ace069d-b1a6-4ca9-b630-2828fdd25be2\" (UID: \"1ace069d-b1a6-4ca9-b630-2828fdd25be2\") " Nov 01 16:27:36.001651 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:36.001604 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ace069d-b1a6-4ca9-b630-2828fdd25be2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ace069d-b1a6-4ca9-b630-2828fdd25be2-maas-api-openshift-ca\") pod \"1ace069d-b1a6-4ca9-b630-2828fdd25be2\" (UID: \"1ace069d-b1a6-4ca9-b630-2828fdd25be2\") " Nov 01 16:27:36.001964 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:36.001939 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ace069d-b1a6-4ca9-b630-2828fdd25be2-maas-api-openshift-ca" pod "1ace069d-b1a6-4ca9-b630-2828fdd25be2" (UID: "1ace069d-b1a6-4ca9-b630-2828fdd25be2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:27:36.002669 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:36.002644 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ace069d-b1a6-4ca9-b630-2828fdd25be2-maas-api-openshift-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:36.003576 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:36.003556 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ace069d-b1a6-4ca9-b630-2828fdd25be2-maas-api-xks-ca" pod "1ace069d-b1a6-4ca9-b630-2828fdd25be2" (UID: "1ace069d-b1a6-4ca9-b630-2828fdd25be2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:27:36.003657 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:36.003641 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ace069d-b1a6-4ca9-b630-2828fdd25be2-kube-api-access-slm49" pod "1ace069d-b1a6-4ca9-b630-2828fdd25be2" (UID: "1ace069d-b1a6-4ca9-b630-2828fdd25be2"). InnerVolumeSpecName "kube-api-access-slm49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:27:36.104185 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:36.104152 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ace069d-b1a6-4ca9-b630-2828fdd25be2-maas-api-xks-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:36.104185 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:36.104184 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slm49\" (UniqueName: \"kubernetes.io/projected/1ace069d-b1a6-4ca9-b630-2828fdd25be2-kube-api-access-slm49\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:36.593830 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:36.593802 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a101f7e1-7r2hq" Nov 01 16:27:36.594050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:36.593849 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a101f7e1-7r2hq" event={"ID":"1ace069d-b1a6-4ca9-b630-2828fdd25be2","Type":"ContainerDied","Data":"e6564a5612cfaf49460356531f1552b9e98b542ce0a897bfb6eb346501e7601a"} Nov 01 16:27:36.594050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:36.593941 4363 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6564a5612cfaf49460356531f1552b9e98b542ce0a897bfb6eb346501e7601a" Nov 01 16:27:37.374534 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:37.374493 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-7225e5/auth-test-model-7225e5-kserve-84f65d66df-jd8qc"] Nov 01 16:27:42.177047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:42.173790 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d864b18-rdvnh"] Nov 01 16:27:42.177047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:42.175436 4363 cpu_manager.go:414] "containerMap: removing container" podUID="1ace069d-b1a6-4ca9-b630-2828fdd25be2" containerName="revoke-keys" Nov 01 16:27:42.177047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:42.175462 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ace069d-b1a6-4ca9-b630-2828fdd25be2" containerName="revoke-keys" Nov 01 16:27:42.177047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:42.175658 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ace069d-b1a6-4ca9-b630-2828fdd25be2" containerName="revoke-keys" Nov 01 16:27:42.181716 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:42.181163 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d864b18-rdvnh" Nov 01 16:27:42.186733 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:42.186697 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d864b18-rdvnh"] Nov 01 16:27:42.191741 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:42.191704 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:27:42.191873 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:42.191776 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:27:42.222081 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:42.221207 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b11d52ce-758e-42f5-bf7b-88adcb1cf938-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d864b18-rdvnh\" (UID: \"b11d52ce-758e-42f5-bf7b-88adcb1cf938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d864b18-rdvnh" Nov 01 16:27:42.222081 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:42.221344 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsfl7\" (UniqueName: \"kubernetes.io/projected/b11d52ce-758e-42f5-bf7b-88adcb1cf938-kube-api-access-xsfl7\") pod \"maas-api-revoke-sub-9d864b18-rdvnh\" (UID: \"b11d52ce-758e-42f5-bf7b-88adcb1cf938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d864b18-rdvnh" Nov 01 16:27:42.222081 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:42.221561 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b11d52ce-758e-42f5-bf7b-88adcb1cf938-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d864b18-rdvnh\" (UID: \"b11d52ce-758e-42f5-bf7b-88adcb1cf938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d864b18-rdvnh" Nov 01 16:27:42.323358 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:42.323317 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b11d52ce-758e-42f5-bf7b-88adcb1cf938-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d864b18-rdvnh\" (UID: \"b11d52ce-758e-42f5-bf7b-88adcb1cf938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d864b18-rdvnh" Nov 01 16:27:42.323565 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:42.323458 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsfl7\" (UniqueName: \"kubernetes.io/projected/b11d52ce-758e-42f5-bf7b-88adcb1cf938-kube-api-access-xsfl7\") pod \"maas-api-revoke-sub-9d864b18-rdvnh\" (UID: \"b11d52ce-758e-42f5-bf7b-88adcb1cf938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d864b18-rdvnh" Nov 01 16:27:42.323646 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:42.323622 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b11d52ce-758e-42f5-bf7b-88adcb1cf938-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d864b18-rdvnh\" (UID: \"b11d52ce-758e-42f5-bf7b-88adcb1cf938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d864b18-rdvnh" Nov 01 16:27:42.324321 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:42.324289 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b11d52ce-758e-42f5-bf7b-88adcb1cf938-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d864b18-rdvnh\" (UID: \"b11d52ce-758e-42f5-bf7b-88adcb1cf938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d864b18-rdvnh" Nov 01 16:27:42.326080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:42.326053 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b11d52ce-758e-42f5-bf7b-88adcb1cf938-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d864b18-rdvnh\" (UID: \"b11d52ce-758e-42f5-bf7b-88adcb1cf938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d864b18-rdvnh" Nov 01 16:27:42.331061 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:42.331006 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsfl7\" (UniqueName: \"kubernetes.io/projected/b11d52ce-758e-42f5-bf7b-88adcb1cf938-kube-api-access-xsfl7\") pod \"maas-api-revoke-sub-9d864b18-rdvnh\" (UID: \"b11d52ce-758e-42f5-bf7b-88adcb1cf938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d864b18-rdvnh" Nov 01 16:27:42.510085 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:42.509984 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d864b18-rdvnh" Nov 01 16:27:42.868193 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:42.868074 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d864b18-rdvnh"] Nov 01 16:27:43.684305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:43.684260 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d864b18-rdvnh" event={"ID":"b11d52ce-758e-42f5-bf7b-88adcb1cf938","Type":"ContainerStarted","Data":"e2f7aa1ed9e33848458123834c55d962487b1f1e737a316ce54325ee41b4f244"} Nov 01 16:27:43.684815 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:43.684322 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d864b18-rdvnh" event={"ID":"b11d52ce-758e-42f5-bf7b-88adcb1cf938","Type":"ContainerStarted","Data":"452e1bbbbe979c65668815fb4079a91aaac4d4a5514d6a8d554187541c7ca622"} Nov 01 16:27:44.694128 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:44.694095 4363 generic.go:356] "Generic (PLEG): container finished" podID="b11d52ce-758e-42f5-bf7b-88adcb1cf938" containerID="e2f7aa1ed9e33848458123834c55d962487b1f1e737a316ce54325ee41b4f244" exitCode=0 Nov 01 16:27:44.694541 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:44.694175 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d864b18-rdvnh" event={"ID":"b11d52ce-758e-42f5-bf7b-88adcb1cf938","Type":"ContainerDied","Data":"e2f7aa1ed9e33848458123834c55d962487b1f1e737a316ce54325ee41b4f244"} Nov 01 16:27:45.894175 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:45.894147 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d864b18-rdvnh" Nov 01 16:27:45.994715 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:45.994623 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b11d52ce-758e-42f5-bf7b-88adcb1cf938-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b11d52ce-758e-42f5-bf7b-88adcb1cf938-maas-api-xks-ca\") pod \"b11d52ce-758e-42f5-bf7b-88adcb1cf938\" (UID: \"b11d52ce-758e-42f5-bf7b-88adcb1cf938\") " Nov 01 16:27:45.994914 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:45.994896 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b11d52ce-758e-42f5-bf7b-88adcb1cf938-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b11d52ce-758e-42f5-bf7b-88adcb1cf938-maas-api-openshift-ca\") pod \"b11d52ce-758e-42f5-bf7b-88adcb1cf938\" (UID: \"b11d52ce-758e-42f5-bf7b-88adcb1cf938\") " Nov 01 16:27:45.995043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:45.995005 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b11d52ce-758e-42f5-bf7b-88adcb1cf938-kube-api-access-xsfl7\" (UniqueName: \"kubernetes.io/projected/b11d52ce-758e-42f5-bf7b-88adcb1cf938-kube-api-access-xsfl7\") pod \"b11d52ce-758e-42f5-bf7b-88adcb1cf938\" (UID: \"b11d52ce-758e-42f5-bf7b-88adcb1cf938\") " Nov 01 16:27:45.995363 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:45.995334 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b11d52ce-758e-42f5-bf7b-88adcb1cf938-maas-api-openshift-ca" pod "b11d52ce-758e-42f5-bf7b-88adcb1cf938" (UID: "b11d52ce-758e-42f5-bf7b-88adcb1cf938"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:27:45.995955 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:45.995931 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b11d52ce-758e-42f5-bf7b-88adcb1cf938-maas-api-openshift-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:45.996890 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:45.996871 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b11d52ce-758e-42f5-bf7b-88adcb1cf938-maas-api-xks-ca" pod "b11d52ce-758e-42f5-bf7b-88adcb1cf938" (UID: "b11d52ce-758e-42f5-bf7b-88adcb1cf938"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:27:45.996991 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:45.996975 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b11d52ce-758e-42f5-bf7b-88adcb1cf938-kube-api-access-xsfl7" pod "b11d52ce-758e-42f5-bf7b-88adcb1cf938" (UID: "b11d52ce-758e-42f5-bf7b-88adcb1cf938"). InnerVolumeSpecName "kube-api-access-xsfl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:27:46.097456 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:46.097419 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b11d52ce-758e-42f5-bf7b-88adcb1cf938-maas-api-xks-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:46.097456 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:46.097452 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsfl7\" (UniqueName: \"kubernetes.io/projected/b11d52ce-758e-42f5-bf7b-88adcb1cf938-kube-api-access-xsfl7\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:46.717481 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:46.717449 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d864b18-rdvnh" Nov 01 16:27:46.717481 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:46.717464 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d864b18-rdvnh" event={"ID":"b11d52ce-758e-42f5-bf7b-88adcb1cf938","Type":"ContainerDied","Data":"452e1bbbbe979c65668815fb4079a91aaac4d4a5514d6a8d554187541c7ca622"} Nov 01 16:27:46.717755 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:46.717558 4363 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="452e1bbbbe979c65668815fb4079a91aaac4d4a5514d6a8d554187541c7ca622" Nov 01 16:27:47.898152 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:47.898122 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtwhw_8254aafa-dd5b-4150-87e4-d165dab6bcb5/ovn-acl-logging/0.log" Nov 01 16:27:47.916487 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:47.916465 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtwhw_8254aafa-dd5b-4150-87e4-d165dab6bcb5/ovn-acl-logging/0.log" Nov 01 16:27:48.421645 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:27:48.421263 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:27:49.041591 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:49.041556 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7321d6-55df99b949-vs6jc"] Nov 01 16:27:49.064830 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:49.064796 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-64bfb77c47-v7d22"] Nov 01 16:27:49.786435 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:49.786399 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7225e5-8ljwn"] Nov 01 16:27:49.789111 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:49.789085 4363 cpu_manager.go:414] "containerMap: removing container" podUID="b11d52ce-758e-42f5-bf7b-88adcb1cf938" containerName="revoke-keys" Nov 01 16:27:49.789306 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:49.789291 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b11d52ce-758e-42f5-bf7b-88adcb1cf938" containerName="revoke-keys" Nov 01 16:27:49.789816 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:49.789791 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="b11d52ce-758e-42f5-bf7b-88adcb1cf938" containerName="revoke-keys" Nov 01 16:27:49.822643 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:49.821999 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7225e5-8ljwn"] Nov 01 16:27:49.831323 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:49.830586 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7225e5-8ljwn" Nov 01 16:27:49.839454 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:49.836861 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:27:49.839454 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:49.836931 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:27:50.014244 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:50.014205 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0b81184-16f4-4349-a387-88e14db5c1f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-7225e5-8ljwn\" (UID: \"f0b81184-16f4-4349-a387-88e14db5c1f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7225e5-8ljwn" Nov 01 16:27:50.014652 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:50.014628 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv9gj\" (UniqueName: \"kubernetes.io/projected/f0b81184-16f4-4349-a387-88e14db5c1f4-kube-api-access-hv9gj\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-7225e5-8ljwn\" (UID: \"f0b81184-16f4-4349-a387-88e14db5c1f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7225e5-8ljwn" Nov 01 16:27:50.014832 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:50.014816 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0b81184-16f4-4349-a387-88e14db5c1f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-7225e5-8ljwn\" (UID: \"f0b81184-16f4-4349-a387-88e14db5c1f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7225e5-8ljwn" Nov 01 16:27:50.116744 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:50.116707 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv9gj\" (UniqueName: \"kubernetes.io/projected/f0b81184-16f4-4349-a387-88e14db5c1f4-kube-api-access-hv9gj\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-7225e5-8ljwn\" (UID: \"f0b81184-16f4-4349-a387-88e14db5c1f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7225e5-8ljwn" Nov 01 16:27:50.117270 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:50.116754 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0b81184-16f4-4349-a387-88e14db5c1f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-7225e5-8ljwn\" (UID: \"f0b81184-16f4-4349-a387-88e14db5c1f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7225e5-8ljwn" Nov 01 16:27:50.117270 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:50.117049 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0b81184-16f4-4349-a387-88e14db5c1f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-7225e5-8ljwn\" (UID: \"f0b81184-16f4-4349-a387-88e14db5c1f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7225e5-8ljwn" Nov 01 16:27:50.117660 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:50.117633 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0b81184-16f4-4349-a387-88e14db5c1f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-7225e5-8ljwn\" (UID: \"f0b81184-16f4-4349-a387-88e14db5c1f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7225e5-8ljwn" Nov 01 16:27:50.119517 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:50.119490 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0b81184-16f4-4349-a387-88e14db5c1f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-7225e5-8ljwn\" (UID: \"f0b81184-16f4-4349-a387-88e14db5c1f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7225e5-8ljwn" Nov 01 16:27:50.127522 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:50.127496 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv9gj\" (UniqueName: \"kubernetes.io/projected/f0b81184-16f4-4349-a387-88e14db5c1f4-kube-api-access-hv9gj\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-7225e5-8ljwn\" (UID: \"f0b81184-16f4-4349-a387-88e14db5c1f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7225e5-8ljwn" Nov 01 16:27:50.168450 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:50.168426 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7225e5-8ljwn" Nov 01 16:27:50.471462 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:50.471431 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7225e5-8ljwn"] Nov 01 16:27:50.472328 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:27:50.472301 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0b81184_16f4_4349_a387_88e14db5c1f4.slice/crio-6ad294fbe2c25a184ca472183bd20d2fa639262657fb9e75db1821037885f1a5 WatchSource:0}: Error finding container 6ad294fbe2c25a184ca472183bd20d2fa639262657fb9e75db1821037885f1a5: Status 404 returned error can't find the container with id 6ad294fbe2c25a184ca472183bd20d2fa639262657fb9e75db1821037885f1a5 Nov 01 16:27:50.773065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:50.772972 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7225e5-8ljwn" event={"ID":"f0b81184-16f4-4349-a387-88e14db5c1f4","Type":"ContainerStarted","Data":"6ad294fbe2c25a184ca472183bd20d2fa639262657fb9e75db1821037885f1a5"} Nov 01 16:27:51.781666 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:51.781626 4363 generic.go:356] "Generic (PLEG): container finished" podID="f0b81184-16f4-4349-a387-88e14db5c1f4" containerID="3565b660bbb4998cdc35a591ec6e4775275d5dede96d5fafd564b06b72f81908" exitCode=0 Nov 01 16:27:51.782083 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:51.781660 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7225e5-8ljwn" event={"ID":"f0b81184-16f4-4349-a387-88e14db5c1f4","Type":"ContainerDied","Data":"3565b660bbb4998cdc35a591ec6e4775275d5dede96d5fafd564b06b72f81908"} Nov 01 16:27:52.403348 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:52.403307 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-7225e5/auth-test-model-7225e5-kserve-84f65d66df-jd8qc" podUID="901ec925-24b0-4f15-a356-b7a5da163158" containerName="main" containerID="cri-o://db4bec0bf45dc84723c13ba4883f1bb50308d69649e5ea75aaf0c355084fd540" gracePeriod=45 Nov 01 16:27:52.708936 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:52.708904 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-7225e5/auth-test-model-7225e5-kserve-84f65d66df-jd8qc" Nov 01 16:27:52.793766 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:52.793719 4363 generic.go:356] "Generic (PLEG): container finished" podID="901ec925-24b0-4f15-a356-b7a5da163158" containerID="db4bec0bf45dc84723c13ba4883f1bb50308d69649e5ea75aaf0c355084fd540" exitCode=0 Nov 01 16:27:52.794292 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:52.794011 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-7225e5/auth-test-model-7225e5-kserve-84f65d66df-jd8qc" Nov 01 16:27:52.805831 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:52.805805 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-7225e5/auth-test-model-7225e5-kserve-84f65d66df-jd8qc" event={"ID":"901ec925-24b0-4f15-a356-b7a5da163158","Type":"ContainerDied","Data":"db4bec0bf45dc84723c13ba4883f1bb50308d69649e5ea75aaf0c355084fd540"} Nov 01 16:27:52.805981 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:52.805847 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-7225e5/auth-test-model-7225e5-kserve-84f65d66df-jd8qc" event={"ID":"901ec925-24b0-4f15-a356-b7a5da163158","Type":"ContainerDied","Data":"a8940ffb361429ed76c4791e0636a22971cb9362c843e0d1a33d501d0022edae"} Nov 01 16:27:52.805981 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:52.805867 4363 scope.go:122] "RemoveContainer" containerID="db4bec0bf45dc84723c13ba4883f1bb50308d69649e5ea75aaf0c355084fd540" Nov 01 16:27:52.821436 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:52.820851 4363 scope.go:122] "RemoveContainer" containerID="db4bec0bf45dc84723c13ba4883f1bb50308d69649e5ea75aaf0c355084fd540" Nov 01 16:27:52.821436 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:27:52.821317 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db4bec0bf45dc84723c13ba4883f1bb50308d69649e5ea75aaf0c355084fd540\": container with ID starting with db4bec0bf45dc84723c13ba4883f1bb50308d69649e5ea75aaf0c355084fd540 not found: ID does not exist" containerID="db4bec0bf45dc84723c13ba4883f1bb50308d69649e5ea75aaf0c355084fd540" Nov 01 16:27:52.821436 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:52.821353 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db4bec0bf45dc84723c13ba4883f1bb50308d69649e5ea75aaf0c355084fd540"} err="failed to get container status \"db4bec0bf45dc84723c13ba4883f1bb50308d69649e5ea75aaf0c355084fd540\": rpc error: code = NotFound desc = could not find container \"db4bec0bf45dc84723c13ba4883f1bb50308d69649e5ea75aaf0c355084fd540\": container with ID starting with db4bec0bf45dc84723c13ba4883f1bb50308d69649e5ea75aaf0c355084fd540 not found: ID does not exist" Nov 01 16:27:52.846329 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:52.841725 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/901ec925-24b0-4f15-a356-b7a5da163158-kube-api-access-89jxw\" (UniqueName: \"kubernetes.io/projected/901ec925-24b0-4f15-a356-b7a5da163158-kube-api-access-89jxw\") pod \"901ec925-24b0-4f15-a356-b7a5da163158\" (UID: \"901ec925-24b0-4f15-a356-b7a5da163158\") " Nov 01 16:27:52.850819 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:52.850768 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/901ec925-24b0-4f15-a356-b7a5da163158-tmp-dir" pod "901ec925-24b0-4f15-a356-b7a5da163158" (UID: "901ec925-24b0-4f15-a356-b7a5da163158"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:27:52.851034 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:52.850993 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/901ec925-24b0-4f15-a356-b7a5da163158-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/901ec925-24b0-4f15-a356-b7a5da163158-tmp-dir\") pod \"901ec925-24b0-4f15-a356-b7a5da163158\" (UID: \"901ec925-24b0-4f15-a356-b7a5da163158\") " Nov 01 16:27:52.856371 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:52.856270 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/901ec925-24b0-4f15-a356-b7a5da163158-home" pod "901ec925-24b0-4f15-a356-b7a5da163158" (UID: "901ec925-24b0-4f15-a356-b7a5da163158"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:27:52.875150 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:52.861154 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/901ec925-24b0-4f15-a356-b7a5da163158-home\" (UniqueName: \"kubernetes.io/empty-dir/901ec925-24b0-4f15-a356-b7a5da163158-home\") pod \"901ec925-24b0-4f15-a356-b7a5da163158\" (UID: \"901ec925-24b0-4f15-a356-b7a5da163158\") " Nov 01 16:27:52.875150 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:52.861357 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/901ec925-24b0-4f15-a356-b7a5da163158-dshm\" (UniqueName: \"kubernetes.io/empty-dir/901ec925-24b0-4f15-a356-b7a5da163158-dshm\") pod \"901ec925-24b0-4f15-a356-b7a5da163158\" (UID: \"901ec925-24b0-4f15-a356-b7a5da163158\") " Nov 01 16:27:52.875150 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:52.861430 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/901ec925-24b0-4f15-a356-b7a5da163158-tls-certs\" (UniqueName: \"kubernetes.io/secret/901ec925-24b0-4f15-a356-b7a5da163158-tls-certs\") pod \"901ec925-24b0-4f15-a356-b7a5da163158\" (UID: \"901ec925-24b0-4f15-a356-b7a5da163158\") " Nov 01 16:27:52.875150 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:52.861616 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/901ec925-24b0-4f15-a356-b7a5da163158-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/901ec925-24b0-4f15-a356-b7a5da163158-model-cache\") pod \"901ec925-24b0-4f15-a356-b7a5da163158\" (UID: \"901ec925-24b0-4f15-a356-b7a5da163158\") " Nov 01 16:27:52.875150 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:52.862539 4363 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/901ec925-24b0-4f15-a356-b7a5da163158-tmp-dir\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:52.875150 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:52.862555 4363 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/901ec925-24b0-4f15-a356-b7a5da163158-home\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:52.875150 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:52.865723 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/901ec925-24b0-4f15-a356-b7a5da163158-model-cache" pod "901ec925-24b0-4f15-a356-b7a5da163158" (UID: "901ec925-24b0-4f15-a356-b7a5da163158"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:27:52.884039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:52.876115 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/901ec925-24b0-4f15-a356-b7a5da163158-dshm" pod "901ec925-24b0-4f15-a356-b7a5da163158" (UID: "901ec925-24b0-4f15-a356-b7a5da163158"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:27:52.884376 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:52.884330 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/901ec925-24b0-4f15-a356-b7a5da163158-tls-certs" pod "901ec925-24b0-4f15-a356-b7a5da163158" (UID: "901ec925-24b0-4f15-a356-b7a5da163158"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:27:52.923285 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:52.905550 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/901ec925-24b0-4f15-a356-b7a5da163158-kube-api-access-89jxw" pod "901ec925-24b0-4f15-a356-b7a5da163158" (UID: "901ec925-24b0-4f15-a356-b7a5da163158"). InnerVolumeSpecName "kube-api-access-89jxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:27:52.964984 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:52.964765 4363 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/901ec925-24b0-4f15-a356-b7a5da163158-dshm\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:52.964984 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:52.964797 4363 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/901ec925-24b0-4f15-a356-b7a5da163158-tls-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:52.964984 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:52.964811 4363 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/901ec925-24b0-4f15-a356-b7a5da163158-model-cache\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:52.964984 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:52.964824 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89jxw\" (UniqueName: \"kubernetes.io/projected/901ec925-24b0-4f15-a356-b7a5da163158-kube-api-access-89jxw\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:53.056507 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:53.056480 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7225e5-8ljwn" Nov 01 16:27:53.129323 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:53.129282 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-7225e5/auth-test-model-7225e5-kserve-84f65d66df-jd8qc"] Nov 01 16:27:53.132876 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:53.132204 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-7225e5/auth-test-model-7225e5-kserve-84f65d66df-jd8qc"] Nov 01 16:27:53.168411 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:53.168385 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0b81184-16f4-4349-a387-88e14db5c1f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0b81184-16f4-4349-a387-88e14db5c1f4-maas-api-openshift-ca\") pod \"f0b81184-16f4-4349-a387-88e14db5c1f4\" (UID: \"f0b81184-16f4-4349-a387-88e14db5c1f4\") " Nov 01 16:27:53.168601 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:53.168578 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0b81184-16f4-4349-a387-88e14db5c1f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0b81184-16f4-4349-a387-88e14db5c1f4-maas-api-xks-ca\") pod \"f0b81184-16f4-4349-a387-88e14db5c1f4\" (UID: \"f0b81184-16f4-4349-a387-88e14db5c1f4\") " Nov 01 16:27:53.168713 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:53.168661 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0b81184-16f4-4349-a387-88e14db5c1f4-kube-api-access-hv9gj\" (UniqueName: \"kubernetes.io/projected/f0b81184-16f4-4349-a387-88e14db5c1f4-kube-api-access-hv9gj\") pod \"f0b81184-16f4-4349-a387-88e14db5c1f4\" (UID: \"f0b81184-16f4-4349-a387-88e14db5c1f4\") " Nov 01 16:27:53.168757 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:53.168731 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0b81184-16f4-4349-a387-88e14db5c1f4-maas-api-openshift-ca" pod "f0b81184-16f4-4349-a387-88e14db5c1f4" (UID: "f0b81184-16f4-4349-a387-88e14db5c1f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:27:53.169338 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:53.169316 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0b81184-16f4-4349-a387-88e14db5c1f4-maas-api-openshift-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:53.170698 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:53.170671 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0b81184-16f4-4349-a387-88e14db5c1f4-maas-api-xks-ca" pod "f0b81184-16f4-4349-a387-88e14db5c1f4" (UID: "f0b81184-16f4-4349-a387-88e14db5c1f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:27:53.170777 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:53.170747 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0b81184-16f4-4349-a387-88e14db5c1f4-kube-api-access-hv9gj" pod "f0b81184-16f4-4349-a387-88e14db5c1f4" (UID: "f0b81184-16f4-4349-a387-88e14db5c1f4"). InnerVolumeSpecName "kube-api-access-hv9gj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:27:53.271107 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:53.270999 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0b81184-16f4-4349-a387-88e14db5c1f4-maas-api-xks-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:53.271107 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:53.271057 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv9gj\" (UniqueName: \"kubernetes.io/projected/f0b81184-16f4-4349-a387-88e14db5c1f4-kube-api-access-hv9gj\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:53.803447 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:53.803421 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7225e5-8ljwn" Nov 01 16:27:53.803956 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:53.803459 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7225e5-8ljwn" event={"ID":"f0b81184-16f4-4349-a387-88e14db5c1f4","Type":"ContainerDied","Data":"6ad294fbe2c25a184ca472183bd20d2fa639262657fb9e75db1821037885f1a5"} Nov 01 16:27:53.803956 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:53.803513 4363 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ad294fbe2c25a184ca472183bd20d2fa639262657fb9e75db1821037885f1a5" Nov 01 16:27:54.042688 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:54.042632 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-7321d6-55df99b949-vs6jc" podUID="8b288c39-87b8-441d-a28f-0a47c01e7d2c" containerName="payload-processing" containerID="cri-o://0174a06b9403c7d549572fba5c5eebcf783103df16edec0dd98a40d3a490583c" gracePeriod=25 Nov 01 16:27:54.065522 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:54.065493 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-64bfb77c47-v7d22" podUID="8b90f058-e42e-4b11-974b-2e22a08d7515" containerName="payload-pre-processing" containerID="cri-o://a291a45b04901ff68526a67d754d2c16eb2eeb11a9776c3191a70d864c066445" gracePeriod=25 Nov 01 16:27:54.482327 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:54.482304 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7321d6-55df99b949-vs6jc" Nov 01 16:27:54.486163 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:54.486143 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-64bfb77c47-v7d22" Nov 01 16:27:54.607165 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:54.607064 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b288c39-87b8-441d-a28f-0a47c01e7d2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b288c39-87b8-441d-a28f-0a47c01e7d2c-plugins-config-volume\") pod \"8b288c39-87b8-441d-a28f-0a47c01e7d2c\" (UID: \"8b288c39-87b8-441d-a28f-0a47c01e7d2c\") " Nov 01 16:27:54.607358 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:54.607184 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b90f058-e42e-4b11-974b-2e22a08d7515-kube-api-access-58jxc\" (UniqueName: \"kubernetes.io/projected/8b90f058-e42e-4b11-974b-2e22a08d7515-kube-api-access-58jxc\") pod \"8b90f058-e42e-4b11-974b-2e22a08d7515\" (UID: \"8b90f058-e42e-4b11-974b-2e22a08d7515\") " Nov 01 16:27:54.607424 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:54.607353 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b288c39-87b8-441d-a28f-0a47c01e7d2c-kube-api-access-djn2q\" (UniqueName: \"kubernetes.io/projected/8b288c39-87b8-441d-a28f-0a47c01e7d2c-kube-api-access-djn2q\") pod \"8b288c39-87b8-441d-a28f-0a47c01e7d2c\" (UID: \"8b288c39-87b8-441d-a28f-0a47c01e7d2c\") " Nov 01 16:27:54.607424 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:54.607405 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b90f058-e42e-4b11-974b-2e22a08d7515-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b90f058-e42e-4b11-974b-2e22a08d7515-plugins-config-volume\") pod \"8b90f058-e42e-4b11-974b-2e22a08d7515\" (UID: \"8b90f058-e42e-4b11-974b-2e22a08d7515\") " Nov 01 16:27:54.607534 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:54.607462 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b288c39-87b8-441d-a28f-0a47c01e7d2c-plugins-config-volume" pod "8b288c39-87b8-441d-a28f-0a47c01e7d2c" (UID: "8b288c39-87b8-441d-a28f-0a47c01e7d2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:27:54.607827 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:54.607801 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b90f058-e42e-4b11-974b-2e22a08d7515-plugins-config-volume" pod "8b90f058-e42e-4b11-974b-2e22a08d7515" (UID: "8b90f058-e42e-4b11-974b-2e22a08d7515"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:27:54.608416 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:54.608359 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b288c39-87b8-441d-a28f-0a47c01e7d2c-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:54.608416 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:54.608390 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b90f058-e42e-4b11-974b-2e22a08d7515-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:54.609852 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:54.609827 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b90f058-e42e-4b11-974b-2e22a08d7515-kube-api-access-58jxc" pod "8b90f058-e42e-4b11-974b-2e22a08d7515" (UID: "8b90f058-e42e-4b11-974b-2e22a08d7515"). InnerVolumeSpecName "kube-api-access-58jxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:27:54.609941 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:54.609929 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b288c39-87b8-441d-a28f-0a47c01e7d2c-kube-api-access-djn2q" pod "8b288c39-87b8-441d-a28f-0a47c01e7d2c" (UID: "8b288c39-87b8-441d-a28f-0a47c01e7d2c"). InnerVolumeSpecName "kube-api-access-djn2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:27:54.710530 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:54.710493 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58jxc\" (UniqueName: \"kubernetes.io/projected/8b90f058-e42e-4b11-974b-2e22a08d7515-kube-api-access-58jxc\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:54.710530 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:54.710526 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djn2q\" (UniqueName: \"kubernetes.io/projected/8b288c39-87b8-441d-a28f-0a47c01e7d2c-kube-api-access-djn2q\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:54.800949 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:54.800912 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="901ec925-24b0-4f15-a356-b7a5da163158" path="/var/lib/kubelet/pods/901ec925-24b0-4f15-a356-b7a5da163158/volumes" Nov 01 16:27:54.818666 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:54.818643 4363 generic.go:356] "Generic (PLEG): container finished" podID="8b90f058-e42e-4b11-974b-2e22a08d7515" containerID="a291a45b04901ff68526a67d754d2c16eb2eeb11a9776c3191a70d864c066445" exitCode=0 Nov 01 16:27:54.820097 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:54.818700 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-64bfb77c47-v7d22" Nov 01 16:27:54.820097 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:54.818732 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-64bfb77c47-v7d22" event={"ID":"8b90f058-e42e-4b11-974b-2e22a08d7515","Type":"ContainerDied","Data":"a291a45b04901ff68526a67d754d2c16eb2eeb11a9776c3191a70d864c066445"} Nov 01 16:27:54.820097 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:54.818801 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-64bfb77c47-v7d22" event={"ID":"8b90f058-e42e-4b11-974b-2e22a08d7515","Type":"ContainerDied","Data":"694c40351522f1ec664a9fe04dc8b04032fe4fca939dddbd5116c04299add602"} Nov 01 16:27:54.820097 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:54.818847 4363 scope.go:122] "RemoveContainer" containerID="a291a45b04901ff68526a67d754d2c16eb2eeb11a9776c3191a70d864c066445" Nov 01 16:27:54.820914 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:54.820753 4363 generic.go:356] "Generic (PLEG): container finished" podID="8b288c39-87b8-441d-a28f-0a47c01e7d2c" containerID="0174a06b9403c7d549572fba5c5eebcf783103df16edec0dd98a40d3a490583c" exitCode=0 Nov 01 16:27:54.820914 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:54.820838 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7321d6-55df99b949-vs6jc" Nov 01 16:27:54.820914 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:54.820851 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7321d6-55df99b949-vs6jc" event={"ID":"8b288c39-87b8-441d-a28f-0a47c01e7d2c","Type":"ContainerDied","Data":"0174a06b9403c7d549572fba5c5eebcf783103df16edec0dd98a40d3a490583c"} Nov 01 16:27:54.820914 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:54.820887 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7321d6-55df99b949-vs6jc" event={"ID":"8b288c39-87b8-441d-a28f-0a47c01e7d2c","Type":"ContainerDied","Data":"765a03a6d78d574ad64c39878dd7a62a1af3452652ae6ce9df82f0b720546a16"} Nov 01 16:27:54.832672 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:54.832655 4363 scope.go:122] "RemoveContainer" containerID="a291a45b04901ff68526a67d754d2c16eb2eeb11a9776c3191a70d864c066445" Nov 01 16:27:54.832984 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:27:54.832964 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a291a45b04901ff68526a67d754d2c16eb2eeb11a9776c3191a70d864c066445\": container with ID starting with a291a45b04901ff68526a67d754d2c16eb2eeb11a9776c3191a70d864c066445 not found: ID does not exist" containerID="a291a45b04901ff68526a67d754d2c16eb2eeb11a9776c3191a70d864c066445" Nov 01 16:27:54.833070 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:54.832990 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a291a45b04901ff68526a67d754d2c16eb2eeb11a9776c3191a70d864c066445"} err="failed to get container status \"a291a45b04901ff68526a67d754d2c16eb2eeb11a9776c3191a70d864c066445\": rpc error: code = NotFound desc = could not find container \"a291a45b04901ff68526a67d754d2c16eb2eeb11a9776c3191a70d864c066445\": container with ID starting with a291a45b04901ff68526a67d754d2c16eb2eeb11a9776c3191a70d864c066445 not found: ID does not exist" Nov 01 16:27:54.833070 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:54.833007 4363 scope.go:122] "RemoveContainer" containerID="0174a06b9403c7d549572fba5c5eebcf783103df16edec0dd98a40d3a490583c" Nov 01 16:27:54.840649 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:54.840630 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-64bfb77c47-v7d22"] Nov 01 16:27:54.845734 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:54.845715 4363 scope.go:122] "RemoveContainer" containerID="0174a06b9403c7d549572fba5c5eebcf783103df16edec0dd98a40d3a490583c" Nov 01 16:27:54.846025 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:27:54.845995 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0174a06b9403c7d549572fba5c5eebcf783103df16edec0dd98a40d3a490583c\": container with ID starting with 0174a06b9403c7d549572fba5c5eebcf783103df16edec0dd98a40d3a490583c not found: ID does not exist" containerID="0174a06b9403c7d549572fba5c5eebcf783103df16edec0dd98a40d3a490583c" Nov 01 16:27:54.846124 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:54.846037 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0174a06b9403c7d549572fba5c5eebcf783103df16edec0dd98a40d3a490583c"} err="failed to get container status \"0174a06b9403c7d549572fba5c5eebcf783103df16edec0dd98a40d3a490583c\": rpc error: code = NotFound desc = could not find container \"0174a06b9403c7d549572fba5c5eebcf783103df16edec0dd98a40d3a490583c\": container with ID starting with 0174a06b9403c7d549572fba5c5eebcf783103df16edec0dd98a40d3a490583c not found: ID does not exist" Nov 01 16:27:54.846327 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:54.846308 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-64bfb77c47-v7d22"] Nov 01 16:27:54.851513 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:54.851494 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7321d6-55df99b949-vs6jc"] Nov 01 16:27:54.855263 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:54.855239 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7321d6-55df99b949-vs6jc"] Nov 01 16:27:56.801438 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:56.801401 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b288c39-87b8-441d-a28f-0a47c01e7d2c" path="/var/lib/kubelet/pods/8b288c39-87b8-441d-a28f-0a47c01e7d2c/volumes" Nov 01 16:27:56.801970 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:56.801892 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b90f058-e42e-4b11-974b-2e22a08d7515" path="/var/lib/kubelet/pods/8b90f058-e42e-4b11-974b-2e22a08d7515/volumes" Nov 01 16:27:59.856085 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:59.856046 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7225e5-d8c8dcb65-ts6np"] Nov 01 16:27:59.874369 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:27:59.874336 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7225e5-db455946c-psv2h"] Nov 01 16:28:03.032513 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:03.032482 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9a482b/sub-test-model-9a482b-kserve-d54b499b5-4svj8"] Nov 01 16:28:04.856482 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:04.856439 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7225e5-d8c8dcb65-ts6np" podUID="f3899829-a9d3-470c-8d39-befbe630cb96" containerName="payload-processing" containerID="cri-o://a18883ebb081f95e165615cf0f1830fd5fb5d86d36a3d8b9cdccb209921558d0" gracePeriod=25 Nov 01 16:28:04.870615 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:04.870581 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7225e5-d8c8dcb65-ts6np" podUID="f3899829-a9d3-470c-8d39-befbe630cb96" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:28:04.874671 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:04.874636 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7225e5-db455946c-psv2h" podUID="bfe20137-e20c-49c2-a99a-8cc8d517003a" containerName="payload-pre-processing" containerID="cri-o://0f8fdaa78b6a45c03910823d799944cc8ee1aaaecbee652ec1443ff4a06291fc" gracePeriod=25 Nov 01 16:28:08.199091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:08.172507 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 01 16:28:08.199091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:08.181374 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-9a482b/sub-test-model-9a482b-kserve-d54b499b5-4svj8" podUID="b70b20e2-9bf8-49b5-b68b-6b45ef65f737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.216:8000/ready\": context deadline exceeded" Nov 01 16:28:08.274754 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:08.274687 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:28:08.313869 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:08.305421 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-e1e630/ipp-route-e1e630-kserve-85dbb46dd5-qqv5q" podUID="474f6a6d-0ab2-4f46-b9b8-f29f577f7787" containerName="main" probeResult="failure" output="Get \"https://10.134.0.233:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:28:08.386954 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:08.386909 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp" podUID="f8ca3021-8cbd-4cfc-aab7-33bb656f4c01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.225:8000/ready\": context deadline exceeded" Nov 01 16:28:08.540539 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:08.539976 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7321d6-5c5749c946-tmb8z"] Nov 01 16:28:08.559579 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:08.542078 4363 cpu_manager.go:414] "containerMap: removing container" podUID="8b90f058-e42e-4b11-974b-2e22a08d7515" containerName="payload-pre-processing" Nov 01 16:28:08.559916 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:08.559873 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b90f058-e42e-4b11-974b-2e22a08d7515" containerName="payload-pre-processing" Nov 01 16:28:08.560106 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:08.560075 4363 cpu_manager.go:414] "containerMap: removing container" podUID="8b288c39-87b8-441d-a28f-0a47c01e7d2c" containerName="payload-processing" Nov 01 16:28:08.560217 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:08.560205 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b288c39-87b8-441d-a28f-0a47c01e7d2c" containerName="payload-processing" Nov 01 16:28:08.560397 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:08.560371 4363 cpu_manager.go:414] "containerMap: removing container" podUID="f0b81184-16f4-4349-a387-88e14db5c1f4" containerName="revoke-keys" Nov 01 16:28:08.560498 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:08.560487 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0b81184-16f4-4349-a387-88e14db5c1f4" containerName="revoke-keys" Nov 01 16:28:08.560666 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:08.560639 4363 cpu_manager.go:414] "containerMap: removing container" podUID="901ec925-24b0-4f15-a356-b7a5da163158" containerName="main" Nov 01 16:28:08.560762 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:08.560752 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="901ec925-24b0-4f15-a356-b7a5da163158" containerName="main" Nov 01 16:28:08.561234 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:08.561205 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="901ec925-24b0-4f15-a356-b7a5da163158" containerName="main" Nov 01 16:28:08.561385 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:08.561371 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b90f058-e42e-4b11-974b-2e22a08d7515" containerName="payload-pre-processing" Nov 01 16:28:08.561504 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:08.561493 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b288c39-87b8-441d-a28f-0a47c01e7d2c" containerName="payload-processing" Nov 01 16:28:08.561619 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:08.561608 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0b81184-16f4-4349-a387-88e14db5c1f4" containerName="revoke-keys" Nov 01 16:28:08.608275 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:08.606238 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7321d6-5c5749c946-tmb8z"] Nov 01 16:28:08.608275 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:08.606390 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7321d6-5c5749c946-tmb8z" Nov 01 16:28:08.621345 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:08.621312 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-7321d6\"" Nov 01 16:28:08.622396 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:08.622365 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-7321d6-dockercfg-nc4z7\"" Nov 01 16:28:08.674289 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:08.661260 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-6f78d745c9-xgnqg"] Nov 01 16:28:08.674289 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:08.670248 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:28:08.711049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:08.684954 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:28:08.720162 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:08.720115 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:28:08.779946 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:08.753887 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngkzg\" (UniqueName: \"kubernetes.io/projected/184a13e5-38cc-42b9-a792-dda5dfdc4336-kube-api-access-ngkzg\") pod \"payload-processing-e2e-ipp-b-7321d6-5c5749c946-tmb8z\" (UID: \"184a13e5-38cc-42b9-a792-dda5dfdc4336\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7321d6-5c5749c946-tmb8z" Nov 01 16:28:08.779946 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:08.754076 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/184a13e5-38cc-42b9-a792-dda5dfdc4336-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7321d6-5c5749c946-tmb8z\" (UID: \"184a13e5-38cc-42b9-a792-dda5dfdc4336\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7321d6-5c5749c946-tmb8z" Nov 01 16:28:08.804242 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:08.804158 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-6f78d745c9-xgnqg" Nov 01 16:28:08.877078 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:08.870881 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/184a13e5-38cc-42b9-a792-dda5dfdc4336-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7321d6-5c5749c946-tmb8z\" (UID: \"184a13e5-38cc-42b9-a792-dda5dfdc4336\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7321d6-5c5749c946-tmb8z" Nov 01 16:28:08.877078 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:08.871388 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngkzg\" (UniqueName: \"kubernetes.io/projected/184a13e5-38cc-42b9-a792-dda5dfdc4336-kube-api-access-ngkzg\") pod \"payload-processing-e2e-ipp-b-7321d6-5c5749c946-tmb8z\" (UID: \"184a13e5-38cc-42b9-a792-dda5dfdc4336\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7321d6-5c5749c946-tmb8z" Nov 01 16:28:08.890549 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:08.887903 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/184a13e5-38cc-42b9-a792-dda5dfdc4336-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7321d6-5c5749c946-tmb8z\" (UID: \"184a13e5-38cc-42b9-a792-dda5dfdc4336\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7321d6-5c5749c946-tmb8z" Nov 01 16:28:08.990039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:08.979716 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c57093ac-f9f4-4739-bb9c-6b0aef206ba3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7321d6-6f78d745c9-xgnqg\" (UID: \"c57093ac-f9f4-4739-bb9c-6b0aef206ba3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-6f78d745c9-xgnqg" Nov 01 16:28:09.014494 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:09.014459 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn8vv\" (UniqueName: \"kubernetes.io/projected/c57093ac-f9f4-4739-bb9c-6b0aef206ba3-kube-api-access-tn8vv\") pod \"payload-pre-processing-e2e-ipp-b-7321d6-6f78d745c9-xgnqg\" (UID: \"c57093ac-f9f4-4739-bb9c-6b0aef206ba3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-6f78d745c9-xgnqg" Nov 01 16:28:09.078491 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:09.078406 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngkzg\" (UniqueName: \"kubernetes.io/projected/184a13e5-38cc-42b9-a792-dda5dfdc4336-kube-api-access-ngkzg\") pod \"payload-processing-e2e-ipp-b-7321d6-5c5749c946-tmb8z\" (UID: \"184a13e5-38cc-42b9-a792-dda5dfdc4336\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7321d6-5c5749c946-tmb8z" Nov 01 16:28:09.119974 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:09.119939 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-6f78d745c9-xgnqg"] Nov 01 16:28:09.121464 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:09.121429 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c57093ac-f9f4-4739-bb9c-6b0aef206ba3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7321d6-6f78d745c9-xgnqg\" (UID: \"c57093ac-f9f4-4739-bb9c-6b0aef206ba3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-6f78d745c9-xgnqg" Nov 01 16:28:09.121696 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:09.121677 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn8vv\" (UniqueName: \"kubernetes.io/projected/c57093ac-f9f4-4739-bb9c-6b0aef206ba3-kube-api-access-tn8vv\") pod \"payload-pre-processing-e2e-ipp-b-7321d6-6f78d745c9-xgnqg\" (UID: \"c57093ac-f9f4-4739-bb9c-6b0aef206ba3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-6f78d745c9-xgnqg" Nov 01 16:28:09.171047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:09.122717 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c57093ac-f9f4-4739-bb9c-6b0aef206ba3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7321d6-6f78d745c9-xgnqg\" (UID: \"c57093ac-f9f4-4739-bb9c-6b0aef206ba3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-6f78d745c9-xgnqg" Nov 01 16:28:09.199174 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:09.150595 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7321d6-5c5749c946-tmb8z" Nov 01 16:28:09.200971 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:09.156427 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:28:09.218619 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:09.198421 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-9a482b/sub-test-model-9a482b-kserve-d54b499b5-4svj8" podUID="b70b20e2-9bf8-49b5-b68b-6b45ef65f737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.216:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:28:09.235043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:09.198470 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 01 16:28:09.235043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:09.222349 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-9a482b/sub-test-model-9a482b-kserve-d54b499b5-4svj8" Nov 01 16:28:09.235043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:09.222411 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 16:28:09.290548 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:09.290497 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:28:09.799951 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:09.344134 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-e1e630/ipp-route-e1e630-kserve-85dbb46dd5-qqv5q" podUID="474f6a6d-0ab2-4f46-b9b8-f29f577f7787" containerName="main" probeResult="failure" output="Get \"https://10.134.0.233:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:28:09.801441 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:09.801412 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 16:28:09.801654 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:09.801637 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-e1e630/ipp-route-e1e630-kserve-85dbb46dd5-qqv5q" Nov 01 16:28:09.802609 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:09.389099 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp" podUID="f8ca3021-8cbd-4cfc-aab7-33bb656f4c01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.225:8000/ready\": context deadline exceeded" Nov 01 16:28:09.823225 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:09.823189 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp" Nov 01 16:28:09.824607 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:09.685276 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 16:28:09.826292 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:09.721644 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:28:09.846265 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:09.531868 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp" podUID="f8ca3021-8cbd-4cfc-aab7-33bb656f4c01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.225:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:28:09.846265 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:09.829282 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:28:09.846265 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:09.829359 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:28:09.879046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:09.679324 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 01 16:28:09.879046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:09.877694 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:28:09.952096 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:09.947954 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn8vv\" (UniqueName: \"kubernetes.io/projected/c57093ac-f9f4-4739-bb9c-6b0aef206ba3-kube-api-access-tn8vv\") pod \"payload-pre-processing-e2e-ipp-b-7321d6-6f78d745c9-xgnqg\" (UID: \"c57093ac-f9f4-4739-bb9c-6b0aef206ba3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-6f78d745c9-xgnqg" Nov 01 16:28:10.092101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:10.051771 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-6f78d745c9-xgnqg" Nov 01 16:28:11.319938 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:10.237109 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-9a482b/sub-test-model-9a482b-kserve-d54b499b5-4svj8" podUID="b70b20e2-9bf8-49b5-b68b-6b45ef65f737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.216:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:28:11.322067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:10.237146 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 01 16:28:11.333148 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:10.238323 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-ipp-a-e1e630/ipp-route-e1e630-kserve-85dbb46dd5-qqv5q" podUID="474f6a6d-0ab2-4f46-b9b8-f29f577f7787" containerName="main" probeResult="failure" output="Get \"https://10.134.0.233:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:28:11.334764 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:10.238363 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:28:11.375370 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:10.738469 4363 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": context deadline exceeded" Nov 01 16:28:11.409058 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:10.836099 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp" podUID="f8ca3021-8cbd-4cfc-aab7-33bb656f4c01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.225:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:28:11.409058 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:10.900818 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-e1e630/ipp-route-e1e630-kserve-85dbb46dd5-qqv5q" podUID="474f6a6d-0ab2-4f46-b9b8-f29f577f7787" containerName="main" probeResult="failure" output="Get \"https://10.134.0.233:8000/ready\": context deadline exceeded" Nov 01 16:28:11.409058 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:10.900953 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 01 16:28:11.409058 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:10.901144 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:28:11.409058 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:10.900417 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:28:11.409058 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:11.275330 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:28:11.432308 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:11.432275 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:28:11.472348 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:11.472310 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-9a482b/sub-test-model-9a482b-kserve-d54b499b5-4svj8" Nov 01 16:28:11.549124 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:11.546402 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:28:11.549124 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:11.547618 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 16:28:11.549124 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:11.549012 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-e1e630/ipp-route-e1e630-kserve-85dbb46dd5-qqv5q" Nov 01 16:28:11.550759 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:11.550368 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 16:28:11.558718 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:11.558697 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:28:11.559469 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:11.559443 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:28:11.561006 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:11.560990 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp" Nov 01 16:28:11.651180 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:11.651152 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7321d6-5c5749c946-tmb8z"] Nov 01 16:28:11.657420 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:28:11.657385 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod184a13e5_38cc_42b9_a792_dda5dfdc4336.slice/crio-700185a256f4ef4cfa619c5c7bd6d09c8f218716cc995369f5a53ef25a5f34c3 WatchSource:0}: Error finding container 700185a256f4ef4cfa619c5c7bd6d09c8f218716cc995369f5a53ef25a5f34c3: Status 404 returned error can't find the container with id 700185a256f4ef4cfa619c5c7bd6d09c8f218716cc995369f5a53ef25a5f34c3 Nov 01 16:28:12.070291 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:12.070259 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-6f78d745c9-xgnqg"] Nov 01 16:28:12.072555 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:28:12.072522 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc57093ac_f9f4_4739_bb9c_6b0aef206ba3.slice/crio-14a2ced2981c427e01041d811108bfe774d75b158de47583e3d0718532b49abd WatchSource:0}: Error finding container 14a2ced2981c427e01041d811108bfe774d75b158de47583e3d0718532b49abd: Status 404 returned error can't find the container with id 14a2ced2981c427e01041d811108bfe774d75b158de47583e3d0718532b49abd Nov 01 16:28:12.583869 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:12.583835 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7321d6-5c5749c946-tmb8z" event={"ID":"184a13e5-38cc-42b9-a792-dda5dfdc4336","Type":"ContainerStarted","Data":"82eb4761154ee60637c8aaf5ef00f46839316b3a354b29469c05f3a572966cb5"} Nov 01 16:28:12.584367 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:12.583893 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7321d6-5c5749c946-tmb8z" event={"ID":"184a13e5-38cc-42b9-a792-dda5dfdc4336","Type":"ContainerStarted","Data":"700185a256f4ef4cfa619c5c7bd6d09c8f218716cc995369f5a53ef25a5f34c3"} Nov 01 16:28:12.584878 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:12.584853 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7321d6-5c5749c946-tmb8z" Nov 01 16:28:12.600071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:12.588713 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-6f78d745c9-xgnqg" event={"ID":"c57093ac-f9f4-4739-bb9c-6b0aef206ba3","Type":"ContainerStarted","Data":"b5a32dc03572943a772673d52e31506791e03db87a179dfbcc8d053e17cb931f"} Nov 01 16:28:12.600071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:12.588754 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-6f78d745c9-xgnqg" event={"ID":"c57093ac-f9f4-4739-bb9c-6b0aef206ba3","Type":"ContainerStarted","Data":"14a2ced2981c427e01041d811108bfe774d75b158de47583e3d0718532b49abd"} Nov 01 16:28:12.600071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:12.588782 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-6f78d745c9-xgnqg" Nov 01 16:28:12.612218 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:12.605800 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-7321d6-5c5749c946-tmb8z" podStartSLOduration=4.605782131 podStartE2EDuration="4.605782131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:28:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:28:12.602336711 +0000 UTC m=+2425.934074364" watchObservedRunningTime="2026-11-01 16:28:12.605782131 +0000 UTC m=+2425.937519784" Nov 01 16:28:13.539037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:13.538226 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-6f78d745c9-xgnqg" podStartSLOduration=5.538209867 podStartE2EDuration="5.538209867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:28:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:28:12.632715649 +0000 UTC m=+2425.964453300" watchObservedRunningTime="2026-11-01 16:28:13.538209867 +0000 UTC m=+2426.869947516" Nov 01 16:28:13.543571 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:13.543537 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7225e5-646b876456-9wqwp"] Nov 01 16:28:13.543797 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:13.543771 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7225e5-646b876456-9wqwp" podUID="033f03c6-c73b-45e7-8567-fdb94c4e5b8d" containerName="maas-api" containerID="cri-o://c822381d2995d721c3155ff3dd17bd4fa1c36a9234e073fe7042b14394e1d57a" gracePeriod=30 Nov 01 16:28:14.221536 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:14.221500 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7225e5-8ljwn"] Nov 01 16:28:14.226522 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:14.226486 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7225e5-8ljwn"] Nov 01 16:28:14.741814 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:14.741781 4363 generic.go:356] "Generic (PLEG): container finished" podID="033f03c6-c73b-45e7-8567-fdb94c4e5b8d" containerID="c822381d2995d721c3155ff3dd17bd4fa1c36a9234e073fe7042b14394e1d57a" exitCode=0 Nov 01 16:28:14.742011 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:14.741861 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7225e5-646b876456-9wqwp" event={"ID":"033f03c6-c73b-45e7-8567-fdb94c4e5b8d","Type":"ContainerDied","Data":"c822381d2995d721c3155ff3dd17bd4fa1c36a9234e073fe7042b14394e1d57a"} Nov 01 16:28:14.831228 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:14.831195 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0b81184-16f4-4349-a387-88e14db5c1f4" path="/var/lib/kubelet/pods/f0b81184-16f4-4349-a387-88e14db5c1f4/volumes" Nov 01 16:28:14.871397 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:14.871358 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7225e5-d8c8dcb65-ts6np" podUID="f3899829-a9d3-470c-8d39-befbe630cb96" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:28:14.919263 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:14.917850 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7225e5-db455946c-psv2h" podUID="bfe20137-e20c-49c2-a99a-8cc8d517003a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:28:14.951848 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:14.951822 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7225e5-646b876456-9wqwp" Nov 01 16:28:15.031471 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:15.030416 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/033f03c6-c73b-45e7-8567-fdb94c4e5b8d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/033f03c6-c73b-45e7-8567-fdb94c4e5b8d-openshift-service-ca\") pod \"033f03c6-c73b-45e7-8567-fdb94c4e5b8d\" (UID: \"033f03c6-c73b-45e7-8567-fdb94c4e5b8d\") " Nov 01 16:28:15.031471 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:15.030653 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/033f03c6-c73b-45e7-8567-fdb94c4e5b8d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/033f03c6-c73b-45e7-8567-fdb94c4e5b8d-metrics-certs\") pod \"033f03c6-c73b-45e7-8567-fdb94c4e5b8d\" (UID: \"033f03c6-c73b-45e7-8567-fdb94c4e5b8d\") " Nov 01 16:28:15.031471 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:15.030816 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/033f03c6-c73b-45e7-8567-fdb94c4e5b8d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/033f03c6-c73b-45e7-8567-fdb94c4e5b8d-maas-api-tls\") pod \"033f03c6-c73b-45e7-8567-fdb94c4e5b8d\" (UID: \"033f03c6-c73b-45e7-8567-fdb94c4e5b8d\") " Nov 01 16:28:15.031471 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:15.030945 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/033f03c6-c73b-45e7-8567-fdb94c4e5b8d-kube-api-access-q29nn\" (UniqueName: \"kubernetes.io/projected/033f03c6-c73b-45e7-8567-fdb94c4e5b8d-kube-api-access-q29nn\") pod \"033f03c6-c73b-45e7-8567-fdb94c4e5b8d\" (UID: \"033f03c6-c73b-45e7-8567-fdb94c4e5b8d\") " Nov 01 16:28:15.032856 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:15.032456 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/033f03c6-c73b-45e7-8567-fdb94c4e5b8d-openshift-service-ca" pod "033f03c6-c73b-45e7-8567-fdb94c4e5b8d" (UID: "033f03c6-c73b-45e7-8567-fdb94c4e5b8d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:28:15.034809 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:15.034783 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/033f03c6-c73b-45e7-8567-fdb94c4e5b8d-kube-api-access-q29nn" pod "033f03c6-c73b-45e7-8567-fdb94c4e5b8d" (UID: "033f03c6-c73b-45e7-8567-fdb94c4e5b8d"). InnerVolumeSpecName "kube-api-access-q29nn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:15.035117 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:15.035086 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/033f03c6-c73b-45e7-8567-fdb94c4e5b8d-maas-api-tls" pod "033f03c6-c73b-45e7-8567-fdb94c4e5b8d" (UID: "033f03c6-c73b-45e7-8567-fdb94c4e5b8d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:28:15.035706 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:15.035687 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/033f03c6-c73b-45e7-8567-fdb94c4e5b8d-metrics-certs" pod "033f03c6-c73b-45e7-8567-fdb94c4e5b8d" (UID: "033f03c6-c73b-45e7-8567-fdb94c4e5b8d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:28:15.140038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:15.135444 4363 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/033f03c6-c73b-45e7-8567-fdb94c4e5b8d-openshift-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:15.140038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:15.135495 4363 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/033f03c6-c73b-45e7-8567-fdb94c4e5b8d-metrics-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:15.140038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:15.135513 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/033f03c6-c73b-45e7-8567-fdb94c4e5b8d-maas-api-tls\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:15.140038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:15.135527 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q29nn\" (UniqueName: \"kubernetes.io/projected/033f03c6-c73b-45e7-8567-fdb94c4e5b8d-kube-api-access-q29nn\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:15.750513 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:15.750483 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7225e5-646b876456-9wqwp" Nov 01 16:28:15.750938 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:15.750522 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7225e5-646b876456-9wqwp" event={"ID":"033f03c6-c73b-45e7-8567-fdb94c4e5b8d","Type":"ContainerDied","Data":"91c26d3e03f7bee7741266bae6b5c75674a7bef0af85c17243f93410187e75e2"} Nov 01 16:28:15.750938 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:15.750590 4363 scope.go:122] "RemoveContainer" containerID="c822381d2995d721c3155ff3dd17bd4fa1c36a9234e073fe7042b14394e1d57a" Nov 01 16:28:15.774389 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:15.774362 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7225e5-646b876456-9wqwp"] Nov 01 16:28:15.807040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:15.803098 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7225e5-646b876456-9wqwp"] Nov 01 16:28:16.374446 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:16.374417 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6992258b-cx54c"] Nov 01 16:28:16.375318 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:16.375298 4363 cpu_manager.go:414] "containerMap: removing container" podUID="033f03c6-c73b-45e7-8567-fdb94c4e5b8d" containerName="maas-api" Nov 01 16:28:16.375407 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:16.375319 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="033f03c6-c73b-45e7-8567-fdb94c4e5b8d" containerName="maas-api" Nov 01 16:28:16.375466 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:16.375457 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="033f03c6-c73b-45e7-8567-fdb94c4e5b8d" containerName="maas-api" Nov 01 16:28:16.389057 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:16.389006 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6992258b-cx54c"] Nov 01 16:28:16.389213 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:16.389150 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6992258b-cx54c" Nov 01 16:28:16.394078 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:16.392526 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:28:16.394224 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:16.394210 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:28:16.467128 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:16.467092 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b034b9f6-e3e0-4682-9323-f99ce63ddaf1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6992258b-cx54c\" (UID: \"b034b9f6-e3e0-4682-9323-f99ce63ddaf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6992258b-cx54c" Nov 01 16:28:16.467334 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:16.467139 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd8x9\" (UniqueName: \"kubernetes.io/projected/b034b9f6-e3e0-4682-9323-f99ce63ddaf1-kube-api-access-hd8x9\") pod \"maas-api-revoke-sub-6992258b-cx54c\" (UID: \"b034b9f6-e3e0-4682-9323-f99ce63ddaf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6992258b-cx54c" Nov 01 16:28:16.467334 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:16.467243 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b034b9f6-e3e0-4682-9323-f99ce63ddaf1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6992258b-cx54c\" (UID: \"b034b9f6-e3e0-4682-9323-f99ce63ddaf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6992258b-cx54c" Nov 01 16:28:16.568736 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:16.568690 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b034b9f6-e3e0-4682-9323-f99ce63ddaf1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6992258b-cx54c\" (UID: \"b034b9f6-e3e0-4682-9323-f99ce63ddaf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6992258b-cx54c" Nov 01 16:28:16.568895 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:16.568749 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd8x9\" (UniqueName: \"kubernetes.io/projected/b034b9f6-e3e0-4682-9323-f99ce63ddaf1-kube-api-access-hd8x9\") pod \"maas-api-revoke-sub-6992258b-cx54c\" (UID: \"b034b9f6-e3e0-4682-9323-f99ce63ddaf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6992258b-cx54c" Nov 01 16:28:16.568895 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:16.568857 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b034b9f6-e3e0-4682-9323-f99ce63ddaf1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6992258b-cx54c\" (UID: \"b034b9f6-e3e0-4682-9323-f99ce63ddaf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6992258b-cx54c" Nov 01 16:28:16.569540 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:16.569515 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b034b9f6-e3e0-4682-9323-f99ce63ddaf1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6992258b-cx54c\" (UID: \"b034b9f6-e3e0-4682-9323-f99ce63ddaf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6992258b-cx54c" Nov 01 16:28:16.571158 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:16.571133 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b034b9f6-e3e0-4682-9323-f99ce63ddaf1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6992258b-cx54c\" (UID: \"b034b9f6-e3e0-4682-9323-f99ce63ddaf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6992258b-cx54c" Nov 01 16:28:16.576346 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:16.576321 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd8x9\" (UniqueName: \"kubernetes.io/projected/b034b9f6-e3e0-4682-9323-f99ce63ddaf1-kube-api-access-hd8x9\") pod \"maas-api-revoke-sub-6992258b-cx54c\" (UID: \"b034b9f6-e3e0-4682-9323-f99ce63ddaf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6992258b-cx54c" Nov 01 16:28:16.730107 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:16.727350 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6992258b-cx54c" Nov 01 16:28:16.825933 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:16.824363 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="033f03c6-c73b-45e7-8567-fdb94c4e5b8d" path="/var/lib/kubelet/pods/033f03c6-c73b-45e7-8567-fdb94c4e5b8d/volumes" Nov 01 16:28:17.117603 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:17.117569 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6992258b-cx54c"] Nov 01 16:28:17.118816 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:28:17.118784 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb034b9f6_e3e0_4682_9323_f99ce63ddaf1.slice/crio-ff7fed4496c4bddc3ae98f43a2068be566d855e60f4d2179107095ea211dd696 WatchSource:0}: Error finding container ff7fed4496c4bddc3ae98f43a2068be566d855e60f4d2179107095ea211dd696: Status 404 returned error can't find the container with id ff7fed4496c4bddc3ae98f43a2068be566d855e60f4d2179107095ea211dd696 Nov 01 16:28:17.795159 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:17.794980 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6992258b-cx54c" event={"ID":"b034b9f6-e3e0-4682-9323-f99ce63ddaf1","Type":"ContainerStarted","Data":"5ebf85af4e623dbba635f82d4d41892bcab41e222c59e9abe96186dbe29d9217"} Nov 01 16:28:17.795159 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:17.795056 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6992258b-cx54c" event={"ID":"b034b9f6-e3e0-4682-9323-f99ce63ddaf1","Type":"ContainerStarted","Data":"ff7fed4496c4bddc3ae98f43a2068be566d855e60f4d2179107095ea211dd696"} Nov 01 16:28:18.061475 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:18.061349 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-9a482b/sub-test-model-9a482b-kserve-d54b499b5-4svj8" podUID="b70b20e2-9bf8-49b5-b68b-6b45ef65f737" containerName="main" containerID="cri-o://2019bd5e04a1dee1227ac5eca1ca2fa5e54083ae36e220eb275f05af100ad450" gracePeriod=45 Nov 01 16:28:18.066804 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:18.066765 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-9a482b/sub-test-model-9a482b-kserve-d54b499b5-4svj8" podUID="b70b20e2-9bf8-49b5-b68b-6b45ef65f737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.216:8000/ready\": EOF" Nov 01 16:28:18.337567 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:28:18.337423 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:28:18.430281 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:18.430249 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9a482b/sub-test-model-9a482b-kserve-d54b499b5-4svj8" Nov 01 16:28:18.450321 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:18.450257 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6992258b-cx54c" podStartSLOduration=2.450237578 podStartE2EDuration="2.450237578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:28:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:28:17.813769871 +0000 UTC m=+2431.145507522" watchObservedRunningTime="2026-11-01 16:28:18.450237578 +0000 UTC m=+2431.781975231" Nov 01 16:28:18.530982 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:18.530943 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b70b20e2-9bf8-49b5-b68b-6b45ef65f737-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b70b20e2-9bf8-49b5-b68b-6b45ef65f737-tmp-dir\") pod \"b70b20e2-9bf8-49b5-b68b-6b45ef65f737\" (UID: \"b70b20e2-9bf8-49b5-b68b-6b45ef65f737\") " Nov 01 16:28:18.531197 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:18.531152 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b70b20e2-9bf8-49b5-b68b-6b45ef65f737-tls-certs\" (UniqueName: \"kubernetes.io/secret/b70b20e2-9bf8-49b5-b68b-6b45ef65f737-tls-certs\") pod \"b70b20e2-9bf8-49b5-b68b-6b45ef65f737\" (UID: \"b70b20e2-9bf8-49b5-b68b-6b45ef65f737\") " Nov 01 16:28:18.531272 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:18.531243 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b70b20e2-9bf8-49b5-b68b-6b45ef65f737-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b70b20e2-9bf8-49b5-b68b-6b45ef65f737-dshm\") pod \"b70b20e2-9bf8-49b5-b68b-6b45ef65f737\" (UID: \"b70b20e2-9bf8-49b5-b68b-6b45ef65f737\") " Nov 01 16:28:18.531344 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:18.531276 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b70b20e2-9bf8-49b5-b68b-6b45ef65f737-tmp-dir" pod "b70b20e2-9bf8-49b5-b68b-6b45ef65f737" (UID: "b70b20e2-9bf8-49b5-b68b-6b45ef65f737"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:18.531344 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:18.531302 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b70b20e2-9bf8-49b5-b68b-6b45ef65f737-kube-api-access-4krmk\" (UniqueName: \"kubernetes.io/projected/b70b20e2-9bf8-49b5-b68b-6b45ef65f737-kube-api-access-4krmk\") pod \"b70b20e2-9bf8-49b5-b68b-6b45ef65f737\" (UID: \"b70b20e2-9bf8-49b5-b68b-6b45ef65f737\") " Nov 01 16:28:18.531457 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:18.531391 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b70b20e2-9bf8-49b5-b68b-6b45ef65f737-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b70b20e2-9bf8-49b5-b68b-6b45ef65f737-model-cache\") pod \"b70b20e2-9bf8-49b5-b68b-6b45ef65f737\" (UID: \"b70b20e2-9bf8-49b5-b68b-6b45ef65f737\") " Nov 01 16:28:18.531514 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:18.531464 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b70b20e2-9bf8-49b5-b68b-6b45ef65f737-home\" (UniqueName: \"kubernetes.io/empty-dir/b70b20e2-9bf8-49b5-b68b-6b45ef65f737-home\") pod \"b70b20e2-9bf8-49b5-b68b-6b45ef65f737\" (UID: \"b70b20e2-9bf8-49b5-b68b-6b45ef65f737\") " Nov 01 16:28:18.531756 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:18.531711 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b70b20e2-9bf8-49b5-b68b-6b45ef65f737-model-cache" pod "b70b20e2-9bf8-49b5-b68b-6b45ef65f737" (UID: "b70b20e2-9bf8-49b5-b68b-6b45ef65f737"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:18.531816 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:18.531756 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b70b20e2-9bf8-49b5-b68b-6b45ef65f737-home" pod "b70b20e2-9bf8-49b5-b68b-6b45ef65f737" (UID: "b70b20e2-9bf8-49b5-b68b-6b45ef65f737"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:18.532391 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:18.532363 4363 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b70b20e2-9bf8-49b5-b68b-6b45ef65f737-model-cache\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:18.532391 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:18.532392 4363 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b70b20e2-9bf8-49b5-b68b-6b45ef65f737-home\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:18.532599 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:18.532407 4363 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b70b20e2-9bf8-49b5-b68b-6b45ef65f737-tmp-dir\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:18.533702 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:18.533680 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b70b20e2-9bf8-49b5-b68b-6b45ef65f737-dshm" pod "b70b20e2-9bf8-49b5-b68b-6b45ef65f737" (UID: "b70b20e2-9bf8-49b5-b68b-6b45ef65f737"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:18.533797 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:18.533698 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b70b20e2-9bf8-49b5-b68b-6b45ef65f737-tls-certs" pod "b70b20e2-9bf8-49b5-b68b-6b45ef65f737" (UID: "b70b20e2-9bf8-49b5-b68b-6b45ef65f737"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:28:18.534594 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:18.534567 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b70b20e2-9bf8-49b5-b68b-6b45ef65f737-kube-api-access-4krmk" pod "b70b20e2-9bf8-49b5-b68b-6b45ef65f737" (UID: "b70b20e2-9bf8-49b5-b68b-6b45ef65f737"). InnerVolumeSpecName "kube-api-access-4krmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:18.634144 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:18.634104 4363 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b70b20e2-9bf8-49b5-b68b-6b45ef65f737-tls-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:18.634144 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:18.634141 4363 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b70b20e2-9bf8-49b5-b68b-6b45ef65f737-dshm\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:18.634434 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:18.634155 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4krmk\" (UniqueName: \"kubernetes.io/projected/b70b20e2-9bf8-49b5-b68b-6b45ef65f737-kube-api-access-4krmk\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:18.821468 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:18.821432 4363 generic.go:356] "Generic (PLEG): container finished" podID="b034b9f6-e3e0-4682-9323-f99ce63ddaf1" containerID="5ebf85af4e623dbba635f82d4d41892bcab41e222c59e9abe96186dbe29d9217" exitCode=0 Nov 01 16:28:18.821779 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:18.821520 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6992258b-cx54c" event={"ID":"b034b9f6-e3e0-4682-9323-f99ce63ddaf1","Type":"ContainerDied","Data":"5ebf85af4e623dbba635f82d4d41892bcab41e222c59e9abe96186dbe29d9217"} Nov 01 16:28:18.823677 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:18.823654 4363 generic.go:356] "Generic (PLEG): container finished" podID="b70b20e2-9bf8-49b5-b68b-6b45ef65f737" containerID="2019bd5e04a1dee1227ac5eca1ca2fa5e54083ae36e220eb275f05af100ad450" exitCode=0 Nov 01 16:28:18.823907 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:18.823682 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9a482b/sub-test-model-9a482b-kserve-d54b499b5-4svj8" event={"ID":"b70b20e2-9bf8-49b5-b68b-6b45ef65f737","Type":"ContainerDied","Data":"2019bd5e04a1dee1227ac5eca1ca2fa5e54083ae36e220eb275f05af100ad450"} Nov 01 16:28:18.824082 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:18.823917 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9a482b/sub-test-model-9a482b-kserve-d54b499b5-4svj8" event={"ID":"b70b20e2-9bf8-49b5-b68b-6b45ef65f737","Type":"ContainerDied","Data":"11444d767cd08157d3e978bd50128a2e32335cb92260e137addf51d03fae300e"} Nov 01 16:28:18.824082 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:18.823942 4363 scope.go:122] "RemoveContainer" containerID="2019bd5e04a1dee1227ac5eca1ca2fa5e54083ae36e220eb275f05af100ad450" Nov 01 16:28:18.824082 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:18.823715 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9a482b/sub-test-model-9a482b-kserve-d54b499b5-4svj8" Nov 01 16:28:18.837559 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:18.837538 4363 scope.go:122] "RemoveContainer" containerID="2019bd5e04a1dee1227ac5eca1ca2fa5e54083ae36e220eb275f05af100ad450" Nov 01 16:28:18.837892 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:28:18.837866 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2019bd5e04a1dee1227ac5eca1ca2fa5e54083ae36e220eb275f05af100ad450\": container with ID starting with 2019bd5e04a1dee1227ac5eca1ca2fa5e54083ae36e220eb275f05af100ad450 not found: ID does not exist" containerID="2019bd5e04a1dee1227ac5eca1ca2fa5e54083ae36e220eb275f05af100ad450" Nov 01 16:28:18.837943 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:18.837905 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2019bd5e04a1dee1227ac5eca1ca2fa5e54083ae36e220eb275f05af100ad450"} err="failed to get container status \"2019bd5e04a1dee1227ac5eca1ca2fa5e54083ae36e220eb275f05af100ad450\": rpc error: code = NotFound desc = could not find container \"2019bd5e04a1dee1227ac5eca1ca2fa5e54083ae36e220eb275f05af100ad450\": container with ID starting with 2019bd5e04a1dee1227ac5eca1ca2fa5e54083ae36e220eb275f05af100ad450 not found: ID does not exist" Nov 01 16:28:18.855925 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:18.854985 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9a482b/sub-test-model-9a482b-kserve-d54b499b5-4svj8"] Nov 01 16:28:18.865349 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:18.865322 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9a482b/sub-test-model-9a482b-kserve-d54b499b5-4svj8"] Nov 01 16:28:19.806908 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:19.806871 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-9a482b/rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl"] Nov 01 16:28:19.808422 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:19.808394 4363 cpu_manager.go:414] "containerMap: removing container" podUID="b70b20e2-9bf8-49b5-b68b-6b45ef65f737" containerName="main" Nov 01 16:28:19.808758 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:19.808420 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b70b20e2-9bf8-49b5-b68b-6b45ef65f737" containerName="main" Nov 01 16:28:19.813568 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:19.808774 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="b70b20e2-9bf8-49b5-b68b-6b45ef65f737" containerName="main" Nov 01 16:28:19.819838 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:19.819818 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9a482b/rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl" Nov 01 16:28:19.823625 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:19.822495 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9a482b/rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl"] Nov 01 16:28:19.829652 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:19.829501 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-9a482b\"/\"openshift-service-ca.crt\"" Nov 01 16:28:19.829652 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:19.829589 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-9a482b\"/\"kube-root-ca.crt\"" Nov 01 16:28:19.829652 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:19.829625 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-9a482b\"/\"rate-test-model-9a482b-kserve-self-signed-certs\"" Nov 01 16:28:19.829821 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:19.829661 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-9a482b\"/\"default-dockercfg-sl2lj\"" Nov 01 16:28:19.870238 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:19.870201 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/190bf9fe-551d-4f76-8709-b637ff81d4a1-dshm\") pod \"rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl\" (UID: \"190bf9fe-551d-4f76-8709-b637ff81d4a1\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl" Nov 01 16:28:19.870238 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:19.870243 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/190bf9fe-551d-4f76-8709-b637ff81d4a1-home\") pod \"rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl\" (UID: \"190bf9fe-551d-4f76-8709-b637ff81d4a1\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl" Nov 01 16:28:19.870495 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:19.870313 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz8x9\" (UniqueName: \"kubernetes.io/projected/190bf9fe-551d-4f76-8709-b637ff81d4a1-kube-api-access-jz8x9\") pod \"rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl\" (UID: \"190bf9fe-551d-4f76-8709-b637ff81d4a1\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl" Nov 01 16:28:19.870495 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:19.870388 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/190bf9fe-551d-4f76-8709-b637ff81d4a1-tls-certs\") pod \"rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl\" (UID: \"190bf9fe-551d-4f76-8709-b637ff81d4a1\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl" Nov 01 16:28:19.870495 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:19.870412 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/190bf9fe-551d-4f76-8709-b637ff81d4a1-model-cache\") pod \"rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl\" (UID: \"190bf9fe-551d-4f76-8709-b637ff81d4a1\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl" Nov 01 16:28:19.870495 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:19.870432 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/190bf9fe-551d-4f76-8709-b637ff81d4a1-tmp-dir\") pod \"rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl\" (UID: \"190bf9fe-551d-4f76-8709-b637ff81d4a1\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl" Nov 01 16:28:20.085105 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:20.074134 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/190bf9fe-551d-4f76-8709-b637ff81d4a1-model-cache\") pod \"rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl\" (UID: \"190bf9fe-551d-4f76-8709-b637ff81d4a1\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl" Nov 01 16:28:20.085376 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:20.085349 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/190bf9fe-551d-4f76-8709-b637ff81d4a1-tmp-dir\") pod \"rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl\" (UID: \"190bf9fe-551d-4f76-8709-b637ff81d4a1\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl" Nov 01 16:28:20.094462 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:20.094420 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/190bf9fe-551d-4f76-8709-b637ff81d4a1-dshm\") pod \"rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl\" (UID: \"190bf9fe-551d-4f76-8709-b637ff81d4a1\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl" Nov 01 16:28:20.108158 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:20.108120 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/190bf9fe-551d-4f76-8709-b637ff81d4a1-home\") pod \"rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl\" (UID: \"190bf9fe-551d-4f76-8709-b637ff81d4a1\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl" Nov 01 16:28:20.121052 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:20.108456 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz8x9\" (UniqueName: \"kubernetes.io/projected/190bf9fe-551d-4f76-8709-b637ff81d4a1-kube-api-access-jz8x9\") pod \"rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl\" (UID: \"190bf9fe-551d-4f76-8709-b637ff81d4a1\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl" Nov 01 16:28:20.121506 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:20.121478 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/190bf9fe-551d-4f76-8709-b637ff81d4a1-tls-certs\") pod \"rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl\" (UID: \"190bf9fe-551d-4f76-8709-b637ff81d4a1\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl" Nov 01 16:28:20.139804 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:20.078643 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/190bf9fe-551d-4f76-8709-b637ff81d4a1-model-cache\") pod \"rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl\" (UID: \"190bf9fe-551d-4f76-8709-b637ff81d4a1\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl" Nov 01 16:28:20.139804 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:20.113408 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/190bf9fe-551d-4f76-8709-b637ff81d4a1-home\") pod \"rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl\" (UID: \"190bf9fe-551d-4f76-8709-b637ff81d4a1\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl" Nov 01 16:28:20.139804 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:20.108644 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/190bf9fe-551d-4f76-8709-b637ff81d4a1-tmp-dir\") pod \"rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl\" (UID: \"190bf9fe-551d-4f76-8709-b637ff81d4a1\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl" Nov 01 16:28:20.159435 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:20.159398 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/190bf9fe-551d-4f76-8709-b637ff81d4a1-tls-certs\") pod \"rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl\" (UID: \"190bf9fe-551d-4f76-8709-b637ff81d4a1\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl" Nov 01 16:28:20.160127 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:20.160103 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz8x9\" (UniqueName: \"kubernetes.io/projected/190bf9fe-551d-4f76-8709-b637ff81d4a1-kube-api-access-jz8x9\") pod \"rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl\" (UID: \"190bf9fe-551d-4f76-8709-b637ff81d4a1\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl" Nov 01 16:28:20.162900 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:20.162872 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/190bf9fe-551d-4f76-8709-b637ff81d4a1-dshm\") pod \"rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl\" (UID: \"190bf9fe-551d-4f76-8709-b637ff81d4a1\") " pod="ai-tenant-e2e-shared-a-w4-9a482b/rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl" Nov 01 16:28:20.258618 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:20.258592 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6992258b-cx54c" Nov 01 16:28:20.342282 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:20.342258 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b034b9f6-e3e0-4682-9323-f99ce63ddaf1-kube-api-access-hd8x9\" (UniqueName: \"kubernetes.io/projected/b034b9f6-e3e0-4682-9323-f99ce63ddaf1-kube-api-access-hd8x9\") pod \"b034b9f6-e3e0-4682-9323-f99ce63ddaf1\" (UID: \"b034b9f6-e3e0-4682-9323-f99ce63ddaf1\") " Nov 01 16:28:20.342440 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:20.342392 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b034b9f6-e3e0-4682-9323-f99ce63ddaf1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b034b9f6-e3e0-4682-9323-f99ce63ddaf1-maas-api-openshift-ca\") pod \"b034b9f6-e3e0-4682-9323-f99ce63ddaf1\" (UID: \"b034b9f6-e3e0-4682-9323-f99ce63ddaf1\") " Nov 01 16:28:20.342510 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:20.342455 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b034b9f6-e3e0-4682-9323-f99ce63ddaf1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b034b9f6-e3e0-4682-9323-f99ce63ddaf1-maas-api-xks-ca\") pod \"b034b9f6-e3e0-4682-9323-f99ce63ddaf1\" (UID: \"b034b9f6-e3e0-4682-9323-f99ce63ddaf1\") " Nov 01 16:28:20.342799 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:20.342762 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b034b9f6-e3e0-4682-9323-f99ce63ddaf1-maas-api-openshift-ca" pod "b034b9f6-e3e0-4682-9323-f99ce63ddaf1" (UID: "b034b9f6-e3e0-4682-9323-f99ce63ddaf1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:28:20.343200 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:20.343179 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b034b9f6-e3e0-4682-9323-f99ce63ddaf1-maas-api-openshift-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:20.344566 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:20.344546 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b034b9f6-e3e0-4682-9323-f99ce63ddaf1-kube-api-access-hd8x9" pod "b034b9f6-e3e0-4682-9323-f99ce63ddaf1" (UID: "b034b9f6-e3e0-4682-9323-f99ce63ddaf1"). InnerVolumeSpecName "kube-api-access-hd8x9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:20.344968 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:20.344945 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b034b9f6-e3e0-4682-9323-f99ce63ddaf1-maas-api-xks-ca" pod "b034b9f6-e3e0-4682-9323-f99ce63ddaf1" (UID: "b034b9f6-e3e0-4682-9323-f99ce63ddaf1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:28:20.456046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:20.455783 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd8x9\" (UniqueName: \"kubernetes.io/projected/b034b9f6-e3e0-4682-9323-f99ce63ddaf1-kube-api-access-hd8x9\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:20.456046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:20.455826 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b034b9f6-e3e0-4682-9323-f99ce63ddaf1-maas-api-xks-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:20.481299 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:20.481248 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9a482b/rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl" Nov 01 16:28:20.805910 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:20.805876 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b70b20e2-9bf8-49b5-b68b-6b45ef65f737" path="/var/lib/kubelet/pods/b70b20e2-9bf8-49b5-b68b-6b45ef65f737/volumes" Nov 01 16:28:20.806458 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:28:20.806416 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod190bf9fe_551d_4f76_8709_b637ff81d4a1.slice/crio-ca4b8a90cc6b5381ab224b2b4608cde76d9e96feccbd1c7a29796f06fa3a7b76 WatchSource:0}: Error finding container ca4b8a90cc6b5381ab224b2b4608cde76d9e96feccbd1c7a29796f06fa3a7b76: Status 404 returned error can't find the container with id ca4b8a90cc6b5381ab224b2b4608cde76d9e96feccbd1c7a29796f06fa3a7b76 Nov 01 16:28:20.806642 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:20.806625 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9a482b/rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl"] Nov 01 16:28:20.868820 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:20.868778 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9a482b/rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl" event={"ID":"190bf9fe-551d-4f76-8709-b637ff81d4a1","Type":"ContainerStarted","Data":"ca4b8a90cc6b5381ab224b2b4608cde76d9e96feccbd1c7a29796f06fa3a7b76"} Nov 01 16:28:20.870563 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:20.870538 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6992258b-cx54c" event={"ID":"b034b9f6-e3e0-4682-9323-f99ce63ddaf1","Type":"ContainerDied","Data":"ff7fed4496c4bddc3ae98f43a2068be566d855e60f4d2179107095ea211dd696"} Nov 01 16:28:20.870683 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:20.870573 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6992258b-cx54c" Nov 01 16:28:20.870683 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:20.870602 4363 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff7fed4496c4bddc3ae98f43a2068be566d855e60f4d2179107095ea211dd696" Nov 01 16:28:21.385093 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:21.385055 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn"] Nov 01 16:28:21.385478 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:21.385426 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" podUID="8744fda2-3d14-48d8-8426-323166cca7a3" containerName="istio-proxy" containerID="cri-o://430f3237ed8e6bf67c1ea07120f26bb5947c6920b5934151c3541be54ff8ba96" gracePeriod=30 Nov 01 16:28:21.393167 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:21.393139 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg"] Nov 01 16:28:21.393564 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:21.393531 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" podUID="f4b349f5-acb2-42b2-8d90-01b2029167f7" containerName="istio-proxy" containerID="cri-o://46f0f59d149b20a47b982a1e8d12504e3ff18c8b7e990e8cff3bb909efe59d41" gracePeriod=30 Nov 01 16:28:21.892311 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:21.892275 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c78897-kkrkx"] Nov 01 16:28:21.892928 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:21.892698 4363 generic.go:356] "Generic (PLEG): container finished" podID="742547a4-fbd7-40d3-a7a3-86fc91fd7cc3" containerID="2d271a9c18d6ff4575a67d700a8b05bbd8356e297a3de9b6e16758d5ff5242e2" exitCode=0 Nov 01 16:28:21.893297 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:21.893277 4363 cpu_manager.go:414] "containerMap: removing container" podUID="b034b9f6-e3e0-4682-9323-f99ce63ddaf1" containerName="revoke-keys" Nov 01 16:28:21.893297 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:21.893296 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b034b9f6-e3e0-4682-9323-f99ce63ddaf1" containerName="revoke-keys" Nov 01 16:28:21.893510 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:21.893492 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="b034b9f6-e3e0-4682-9323-f99ce63ddaf1" containerName="revoke-keys" Nov 01 16:28:21.904055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:21.897976 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" event={"ID":"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3","Type":"ContainerDied","Data":"2d271a9c18d6ff4575a67d700a8b05bbd8356e297a3de9b6e16758d5ff5242e2"} Nov 01 16:28:21.904055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:21.898047 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9a482b/rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl" event={"ID":"190bf9fe-551d-4f76-8709-b637ff81d4a1","Type":"ContainerStarted","Data":"6144fdc0eede1c2cb6f369996f14e9c4ac1a5e2cd85690d87a71143871fc8347"} Nov 01 16:28:21.904055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:21.898712 4363 scope.go:122] "RemoveContainer" containerID="2d271a9c18d6ff4575a67d700a8b05bbd8356e297a3de9b6e16758d5ff5242e2" Nov 01 16:28:21.904055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:21.899505 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c78897-kkrkx" Nov 01 16:28:21.910914 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:21.910461 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:28:21.910914 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:21.910550 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:28:21.910914 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:21.910602 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:28:21.927431 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:21.924584 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c78897-kkrkx"] Nov 01 16:28:21.929534 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:21.929478 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-9a482b/rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl" podStartSLOduration=2.929457067 podStartE2EDuration="2.929457067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:28:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:28:21.929057413 +0000 UTC m=+2435.260795056" watchObservedRunningTime="2026-11-01 16:28:21.929457067 +0000 UTC m=+2435.261194717" Nov 01 16:28:21.987075 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:21.987009 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:28:21.987288 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:21.987086 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:28:22.016082 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:22.016046 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04ee6238-4a5c-477d-b1b2-0442ecc9c037-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c78897-kkrkx\" (UID: \"04ee6238-4a5c-477d-b1b2-0442ecc9c037\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c78897-kkrkx" Nov 01 16:28:22.016359 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:22.016332 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04ee6238-4a5c-477d-b1b2-0442ecc9c037-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c78897-kkrkx\" (UID: \"04ee6238-4a5c-477d-b1b2-0442ecc9c037\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c78897-kkrkx" Nov 01 16:28:22.016594 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:22.016571 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj74k\" (UniqueName: \"kubernetes.io/projected/04ee6238-4a5c-477d-b1b2-0442ecc9c037-kube-api-access-qj74k\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c78897-kkrkx\" (UID: \"04ee6238-4a5c-477d-b1b2-0442ecc9c037\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c78897-kkrkx" Nov 01 16:28:22.121106 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:22.118880 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj74k\" (UniqueName: \"kubernetes.io/projected/04ee6238-4a5c-477d-b1b2-0442ecc9c037-kube-api-access-qj74k\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c78897-kkrkx\" (UID: \"04ee6238-4a5c-477d-b1b2-0442ecc9c037\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c78897-kkrkx" Nov 01 16:28:22.121106 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:22.119203 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04ee6238-4a5c-477d-b1b2-0442ecc9c037-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c78897-kkrkx\" (UID: \"04ee6238-4a5c-477d-b1b2-0442ecc9c037\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c78897-kkrkx" Nov 01 16:28:22.121106 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:22.119386 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04ee6238-4a5c-477d-b1b2-0442ecc9c037-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c78897-kkrkx\" (UID: \"04ee6238-4a5c-477d-b1b2-0442ecc9c037\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c78897-kkrkx" Nov 01 16:28:22.121106 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:22.120273 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04ee6238-4a5c-477d-b1b2-0442ecc9c037-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c78897-kkrkx\" (UID: \"04ee6238-4a5c-477d-b1b2-0442ecc9c037\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c78897-kkrkx" Nov 01 16:28:22.123644 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:22.123610 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04ee6238-4a5c-477d-b1b2-0442ecc9c037-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c78897-kkrkx\" (UID: \"04ee6238-4a5c-477d-b1b2-0442ecc9c037\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c78897-kkrkx" Nov 01 16:28:22.140156 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:22.140123 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj74k\" (UniqueName: \"kubernetes.io/projected/04ee6238-4a5c-477d-b1b2-0442ecc9c037-kube-api-access-qj74k\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c78897-kkrkx\" (UID: \"04ee6238-4a5c-477d-b1b2-0442ecc9c037\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c78897-kkrkx" Nov 01 16:28:22.238474 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:22.238435 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c78897-kkrkx" Nov 01 16:28:22.482655 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:22.482573 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-9a482b/rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl" Nov 01 16:28:22.531060 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:28:22.530986 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04ee6238_4a5c_477d_b1b2_0442ecc9c037.slice/crio-a666b911e8fbfaaf2f4bb3a440425bcd4fe411c47e7359a7350bab2a19f0342e WatchSource:0}: Error finding container a666b911e8fbfaaf2f4bb3a440425bcd4fe411c47e7359a7350bab2a19f0342e: Status 404 returned error can't find the container with id a666b911e8fbfaaf2f4bb3a440425bcd4fe411c47e7359a7350bab2a19f0342e Nov 01 16:28:22.531583 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:22.531558 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c78897-kkrkx"] Nov 01 16:28:22.937739 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:22.937703 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c78897-kkrkx" event={"ID":"04ee6238-4a5c-477d-b1b2-0442ecc9c037","Type":"ContainerStarted","Data":"a666b911e8fbfaaf2f4bb3a440425bcd4fe411c47e7359a7350bab2a19f0342e"} Nov 01 16:28:22.940279 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:22.940250 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" event={"ID":"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3","Type":"ContainerStarted","Data":"5dc6bfe92933edec7e596d52fece68e868ea7ca66853a0220f2a18209a7ec8ff"} Nov 01 16:28:23.283960 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:23.283931 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" Nov 01 16:28:23.735235 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:23.735203 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-6f78d745c9-xgnqg" Nov 01 16:28:23.855757 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:23.855728 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" Nov 01 16:28:23.865814 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:23.856995 4363 patch_prober.go:29] interesting pod/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Nov 01 16:28:23.865814 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:23.865155 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" podUID="742547a4-fbd7-40d3-a7a3-86fc91fd7cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Nov 01 16:28:23.950989 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:23.950951 4363 generic.go:356] "Generic (PLEG): container finished" podID="04ee6238-4a5c-477d-b1b2-0442ecc9c037" containerID="e219a15223593c53cab77d5ffd00c81a0c6504b584e2112e7959169b942025e3" exitCode=0 Nov 01 16:28:23.951543 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:23.951053 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c78897-kkrkx" event={"ID":"04ee6238-4a5c-477d-b1b2-0442ecc9c037","Type":"ContainerDied","Data":"e219a15223593c53cab77d5ffd00c81a0c6504b584e2112e7959169b942025e3"} Nov 01 16:28:24.744706 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:24.744674 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7321d6-5c5749c946-tmb8z" Nov 01 16:28:24.855671 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:24.855632 4363 patch_prober.go:29] interesting pod/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Nov 01 16:28:24.855846 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:24.855703 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" podUID="742547a4-fbd7-40d3-a7a3-86fc91fd7cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Nov 01 16:28:24.870076 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:28:24.870047 4363 logging.go:55] [core] [Channel #11037 SubChannel #11038]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.207:50052", ServerName: "10.134.0.207:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.207:50052: connect: connection refused" Nov 01 16:28:24.902036 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:28:24.901826 4363 logging.go:55] [core] [Channel #11039 SubChannel #11040]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.206:50052", ServerName: "10.134.0.206:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.206:50052: connect: connection refused" Nov 01 16:28:25.324864 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:25.324776 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7225e5-db455946c-psv2h" Nov 01 16:28:25.419175 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:25.419144 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7225e5-d8c8dcb65-ts6np" Nov 01 16:28:25.423908 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:25.423885 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c78897-kkrkx" Nov 01 16:28:25.424590 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:25.424568 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfe20137-e20c-49c2-a99a-8cc8d517003a-kube-api-access-rlnqz\" (UniqueName: \"kubernetes.io/projected/bfe20137-e20c-49c2-a99a-8cc8d517003a-kube-api-access-rlnqz\") pod \"bfe20137-e20c-49c2-a99a-8cc8d517003a\" (UID: \"bfe20137-e20c-49c2-a99a-8cc8d517003a\") " Nov 01 16:28:25.425058 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:25.425041 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfe20137-e20c-49c2-a99a-8cc8d517003a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe20137-e20c-49c2-a99a-8cc8d517003a-plugins-config-volume\") pod \"bfe20137-e20c-49c2-a99a-8cc8d517003a\" (UID: \"bfe20137-e20c-49c2-a99a-8cc8d517003a\") " Nov 01 16:28:25.425484 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:25.425457 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfe20137-e20c-49c2-a99a-8cc8d517003a-plugins-config-volume" pod "bfe20137-e20c-49c2-a99a-8cc8d517003a" (UID: "bfe20137-e20c-49c2-a99a-8cc8d517003a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:28:25.425941 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:25.425923 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe20137-e20c-49c2-a99a-8cc8d517003a-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:25.427374 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:25.427353 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfe20137-e20c-49c2-a99a-8cc8d517003a-kube-api-access-rlnqz" pod "bfe20137-e20c-49c2-a99a-8cc8d517003a" (UID: "bfe20137-e20c-49c2-a99a-8cc8d517003a"). InnerVolumeSpecName "kube-api-access-rlnqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:25.527368 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:25.527328 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04ee6238-4a5c-477d-b1b2-0442ecc9c037-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04ee6238-4a5c-477d-b1b2-0442ecc9c037-maas-api-xks-ca\") pod \"04ee6238-4a5c-477d-b1b2-0442ecc9c037\" (UID: \"04ee6238-4a5c-477d-b1b2-0442ecc9c037\") " Nov 01 16:28:25.528475 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:25.528445 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3899829-a9d3-470c-8d39-befbe630cb96-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3899829-a9d3-470c-8d39-befbe630cb96-plugins-config-volume\") pod \"f3899829-a9d3-470c-8d39-befbe630cb96\" (UID: \"f3899829-a9d3-470c-8d39-befbe630cb96\") " Nov 01 16:28:25.528621 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:25.528597 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3899829-a9d3-470c-8d39-befbe630cb96-kube-api-access-sg4rx\" (UniqueName: \"kubernetes.io/projected/f3899829-a9d3-470c-8d39-befbe630cb96-kube-api-access-sg4rx\") pod \"f3899829-a9d3-470c-8d39-befbe630cb96\" (UID: \"f3899829-a9d3-470c-8d39-befbe630cb96\") " Nov 01 16:28:25.528846 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:25.528823 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3899829-a9d3-470c-8d39-befbe630cb96-plugins-config-volume" pod "f3899829-a9d3-470c-8d39-befbe630cb96" (UID: "f3899829-a9d3-470c-8d39-befbe630cb96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:28:25.528939 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:25.528861 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04ee6238-4a5c-477d-b1b2-0442ecc9c037-kube-api-access-qj74k\" (UniqueName: \"kubernetes.io/projected/04ee6238-4a5c-477d-b1b2-0442ecc9c037-kube-api-access-qj74k\") pod \"04ee6238-4a5c-477d-b1b2-0442ecc9c037\" (UID: \"04ee6238-4a5c-477d-b1b2-0442ecc9c037\") " Nov 01 16:28:25.529555 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:25.529532 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04ee6238-4a5c-477d-b1b2-0442ecc9c037-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04ee6238-4a5c-477d-b1b2-0442ecc9c037-maas-api-openshift-ca\") pod \"04ee6238-4a5c-477d-b1b2-0442ecc9c037\" (UID: \"04ee6238-4a5c-477d-b1b2-0442ecc9c037\") " Nov 01 16:28:25.530118 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:25.530090 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04ee6238-4a5c-477d-b1b2-0442ecc9c037-maas-api-openshift-ca" pod "04ee6238-4a5c-477d-b1b2-0442ecc9c037" (UID: "04ee6238-4a5c-477d-b1b2-0442ecc9c037"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:28:25.530465 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:25.530440 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04ee6238-4a5c-477d-b1b2-0442ecc9c037-maas-api-openshift-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:25.530555 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:25.530473 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3899829-a9d3-470c-8d39-befbe630cb96-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:25.530555 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:25.530489 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlnqz\" (UniqueName: \"kubernetes.io/projected/bfe20137-e20c-49c2-a99a-8cc8d517003a-kube-api-access-rlnqz\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:25.531360 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:25.531337 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3899829-a9d3-470c-8d39-befbe630cb96-kube-api-access-sg4rx" pod "f3899829-a9d3-470c-8d39-befbe630cb96" (UID: "f3899829-a9d3-470c-8d39-befbe630cb96"). InnerVolumeSpecName "kube-api-access-sg4rx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:25.532600 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:25.532561 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04ee6238-4a5c-477d-b1b2-0442ecc9c037-kube-api-access-qj74k" pod "04ee6238-4a5c-477d-b1b2-0442ecc9c037" (UID: "04ee6238-4a5c-477d-b1b2-0442ecc9c037"). InnerVolumeSpecName "kube-api-access-qj74k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:25.534647 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:25.534474 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04ee6238-4a5c-477d-b1b2-0442ecc9c037-maas-api-xks-ca" pod "04ee6238-4a5c-477d-b1b2-0442ecc9c037" (UID: "04ee6238-4a5c-477d-b1b2-0442ecc9c037"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:28:25.632400 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:25.632364 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj74k\" (UniqueName: \"kubernetes.io/projected/04ee6238-4a5c-477d-b1b2-0442ecc9c037-kube-api-access-qj74k\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:25.632400 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:25.632398 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04ee6238-4a5c-477d-b1b2-0442ecc9c037-maas-api-xks-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:25.632625 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:25.632412 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg4rx\" (UniqueName: \"kubernetes.io/projected/f3899829-a9d3-470c-8d39-befbe630cb96-kube-api-access-sg4rx\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:25.759792 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:25.759759 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:28:25.759964 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:25.759813 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" podUID="8744fda2-3d14-48d8-8426-323166cca7a3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:28:25.873076 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:25.869909 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7225e5-d8c8dcb65-ts6np" podUID="f3899829-a9d3-470c-8d39-befbe630cb96" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.207:50052\" within 1s: context deadline exceeded" Nov 01 16:28:25.873076 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:25.870037 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7225e5-d8c8dcb65-ts6np" Nov 01 16:28:25.882037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:25.881895 4363 patch_prober.go:29] interesting pod/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Nov 01 16:28:25.882037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:25.881950 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" podUID="742547a4-fbd7-40d3-a7a3-86fc91fd7cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Nov 01 16:28:25.905474 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:25.905178 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7225e5-db455946c-psv2h" podUID="bfe20137-e20c-49c2-a99a-8cc8d517003a" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.206:50052\" within 1s: context deadline exceeded" Nov 01 16:28:26.044599 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.044569 4363 generic.go:356] "Generic (PLEG): container finished" podID="bfe20137-e20c-49c2-a99a-8cc8d517003a" containerID="0f8fdaa78b6a45c03910823d799944cc8ee1aaaecbee652ec1443ff4a06291fc" exitCode=0 Nov 01 16:28:26.044796 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.044625 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7225e5-db455946c-psv2h" Nov 01 16:28:26.044796 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.044655 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7225e5-db455946c-psv2h" event={"ID":"bfe20137-e20c-49c2-a99a-8cc8d517003a","Type":"ContainerDied","Data":"0f8fdaa78b6a45c03910823d799944cc8ee1aaaecbee652ec1443ff4a06291fc"} Nov 01 16:28:26.044796 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.044711 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7225e5-db455946c-psv2h" event={"ID":"bfe20137-e20c-49c2-a99a-8cc8d517003a","Type":"ContainerDied","Data":"c6f4abdf0744a785578a003cb875ebf56eacbd579a6a8f129a571734b046f318"} Nov 01 16:28:26.044796 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.044742 4363 scope.go:122] "RemoveContainer" containerID="0f8fdaa78b6a45c03910823d799944cc8ee1aaaecbee652ec1443ff4a06291fc" Nov 01 16:28:26.046232 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.046209 4363 generic.go:356] "Generic (PLEG): container finished" podID="f3899829-a9d3-470c-8d39-befbe630cb96" containerID="a18883ebb081f95e165615cf0f1830fd5fb5d86d36a3d8b9cdccb209921558d0" exitCode=0 Nov 01 16:28:26.046355 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.046250 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7225e5-d8c8dcb65-ts6np" Nov 01 16:28:26.046355 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.046256 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7225e5-d8c8dcb65-ts6np" event={"ID":"f3899829-a9d3-470c-8d39-befbe630cb96","Type":"ContainerDied","Data":"a18883ebb081f95e165615cf0f1830fd5fb5d86d36a3d8b9cdccb209921558d0"} Nov 01 16:28:26.046355 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.046311 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7225e5-d8c8dcb65-ts6np" event={"ID":"f3899829-a9d3-470c-8d39-befbe630cb96","Type":"ContainerDied","Data":"3d90fc0a9a1e18a3699d38e329105518445506801f6c8d5c2e434fc2aaf73864"} Nov 01 16:28:26.048090 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.048064 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c78897-kkrkx" event={"ID":"04ee6238-4a5c-477d-b1b2-0442ecc9c037","Type":"ContainerDied","Data":"a666b911e8fbfaaf2f4bb3a440425bcd4fe411c47e7359a7350bab2a19f0342e"} Nov 01 16:28:26.048243 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.048110 4363 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a666b911e8fbfaaf2f4bb3a440425bcd4fe411c47e7359a7350bab2a19f0342e" Nov 01 16:28:26.048243 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.048114 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c78897-kkrkx" Nov 01 16:28:26.058536 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.058521 4363 scope.go:122] "RemoveContainer" containerID="0f8fdaa78b6a45c03910823d799944cc8ee1aaaecbee652ec1443ff4a06291fc" Nov 01 16:28:26.058922 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:28:26.058905 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f8fdaa78b6a45c03910823d799944cc8ee1aaaecbee652ec1443ff4a06291fc\": container with ID starting with 0f8fdaa78b6a45c03910823d799944cc8ee1aaaecbee652ec1443ff4a06291fc not found: ID does not exist" containerID="0f8fdaa78b6a45c03910823d799944cc8ee1aaaecbee652ec1443ff4a06291fc" Nov 01 16:28:26.058971 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.058931 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f8fdaa78b6a45c03910823d799944cc8ee1aaaecbee652ec1443ff4a06291fc"} err="failed to get container status \"0f8fdaa78b6a45c03910823d799944cc8ee1aaaecbee652ec1443ff4a06291fc\": rpc error: code = NotFound desc = could not find container \"0f8fdaa78b6a45c03910823d799944cc8ee1aaaecbee652ec1443ff4a06291fc\": container with ID starting with 0f8fdaa78b6a45c03910823d799944cc8ee1aaaecbee652ec1443ff4a06291fc not found: ID does not exist" Nov 01 16:28:26.058971 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.058948 4363 scope.go:122] "RemoveContainer" containerID="a18883ebb081f95e165615cf0f1830fd5fb5d86d36a3d8b9cdccb209921558d0" Nov 01 16:28:26.069967 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.069953 4363 scope.go:122] "RemoveContainer" containerID="a18883ebb081f95e165615cf0f1830fd5fb5d86d36a3d8b9cdccb209921558d0" Nov 01 16:28:26.070310 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:28:26.070287 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a18883ebb081f95e165615cf0f1830fd5fb5d86d36a3d8b9cdccb209921558d0\": container with ID starting with a18883ebb081f95e165615cf0f1830fd5fb5d86d36a3d8b9cdccb209921558d0 not found: ID does not exist" containerID="a18883ebb081f95e165615cf0f1830fd5fb5d86d36a3d8b9cdccb209921558d0" Nov 01 16:28:26.070408 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.070314 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a18883ebb081f95e165615cf0f1830fd5fb5d86d36a3d8b9cdccb209921558d0"} err="failed to get container status \"a18883ebb081f95e165615cf0f1830fd5fb5d86d36a3d8b9cdccb209921558d0\": rpc error: code = NotFound desc = could not find container \"a18883ebb081f95e165615cf0f1830fd5fb5d86d36a3d8b9cdccb209921558d0\": container with ID starting with a18883ebb081f95e165615cf0f1830fd5fb5d86d36a3d8b9cdccb209921558d0 not found: ID does not exist" Nov 01 16:28:26.073592 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.073555 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7225e5-db455946c-psv2h"] Nov 01 16:28:26.113158 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.113123 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7225e5-db455946c-psv2h"] Nov 01 16:28:26.118079 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.118048 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7225e5-d8c8dcb65-ts6np"] Nov 01 16:28:26.122464 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.122441 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7225e5-d8c8dcb65-ts6np"] Nov 01 16:28:26.524346 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:28:26.524322 4363 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8744fda2_3d14_48d8_8426_323166cca7a3.slice/crio-conmon-430f3237ed8e6bf67c1ea07120f26bb5947c6920b5934151c3541be54ff8ba96.scope\": RecentStats: unable to find data in memory cache]" Nov 01 16:28:26.719098 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.717180 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a498ae42-krb6v"] Nov 01 16:28:26.723731 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.719281 4363 cpu_manager.go:414] "containerMap: removing container" podUID="04ee6238-4a5c-477d-b1b2-0442ecc9c037" containerName="revoke-keys" Nov 01 16:28:26.723731 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.719307 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04ee6238-4a5c-477d-b1b2-0442ecc9c037" containerName="revoke-keys" Nov 01 16:28:26.723731 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.719489 4363 cpu_manager.go:414] "containerMap: removing container" podUID="f3899829-a9d3-470c-8d39-befbe630cb96" containerName="payload-processing" Nov 01 16:28:26.723731 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.719501 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3899829-a9d3-470c-8d39-befbe630cb96" containerName="payload-processing" Nov 01 16:28:26.723731 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.719539 4363 cpu_manager.go:414] "containerMap: removing container" podUID="bfe20137-e20c-49c2-a99a-8cc8d517003a" containerName="payload-pre-processing" Nov 01 16:28:26.723731 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.719549 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfe20137-e20c-49c2-a99a-8cc8d517003a" containerName="payload-pre-processing" Nov 01 16:28:26.723731 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.719917 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3899829-a9d3-470c-8d39-befbe630cb96" containerName="payload-processing" Nov 01 16:28:26.723731 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.719944 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfe20137-e20c-49c2-a99a-8cc8d517003a" containerName="payload-pre-processing" Nov 01 16:28:26.723731 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.719965 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="04ee6238-4a5c-477d-b1b2-0442ecc9c037" containerName="revoke-keys" Nov 01 16:28:26.729243 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.728705 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a498ae42-krb6v" Nov 01 16:28:26.734243 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.734215 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:28:26.739403 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.734635 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:28:26.739631 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.736034 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a498ae42-krb6v"] Nov 01 16:28:26.772232 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.772195 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/669d4672-f30d-4c7f-9a8c-768a472da306-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a498ae42-krb6v\" (UID: \"669d4672-f30d-4c7f-9a8c-768a472da306\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a498ae42-krb6v" Nov 01 16:28:26.772404 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.772372 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/669d4672-f30d-4c7f-9a8c-768a472da306-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a498ae42-krb6v\" (UID: \"669d4672-f30d-4c7f-9a8c-768a472da306\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a498ae42-krb6v" Nov 01 16:28:26.772511 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.772487 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6db8\" (UniqueName: \"kubernetes.io/projected/669d4672-f30d-4c7f-9a8c-768a472da306-kube-api-access-q6db8\") pod \"maas-api-revoke-sub-a498ae42-krb6v\" (UID: \"669d4672-f30d-4c7f-9a8c-768a472da306\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a498ae42-krb6v" Nov 01 16:28:26.842413 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.842247 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfe20137-e20c-49c2-a99a-8cc8d517003a" path="/var/lib/kubelet/pods/bfe20137-e20c-49c2-a99a-8cc8d517003a/volumes" Nov 01 16:28:26.843059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.842973 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3899829-a9d3-470c-8d39-befbe630cb96" path="/var/lib/kubelet/pods/f3899829-a9d3-470c-8d39-befbe630cb96/volumes" Nov 01 16:28:26.843519 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.843495 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-104b6e-sjwz4"] Nov 01 16:28:26.855492 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.855129 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-104b6e-sjwz4"] Nov 01 16:28:26.855492 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.855240 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-104b6e-sjwz4" Nov 01 16:28:26.855782 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.855760 4363 patch_prober.go:29] interesting pod/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Nov 01 16:28:26.855917 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.855893 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" podUID="742547a4-fbd7-40d3-a7a3-86fc91fd7cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Nov 01 16:28:26.940085 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.939995 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60da33d7-f926-4e51-a32e-ea1125d706be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-104b6e-sjwz4\" (UID: \"60da33d7-f926-4e51-a32e-ea1125d706be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-104b6e-sjwz4" Nov 01 16:28:26.940277 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.940129 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/669d4672-f30d-4c7f-9a8c-768a472da306-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a498ae42-krb6v\" (UID: \"669d4672-f30d-4c7f-9a8c-768a472da306\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a498ae42-krb6v" Nov 01 16:28:26.940358 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.940329 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60da33d7-f926-4e51-a32e-ea1125d706be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-104b6e-sjwz4\" (UID: \"60da33d7-f926-4e51-a32e-ea1125d706be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-104b6e-sjwz4" Nov 01 16:28:26.940518 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.940380 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lc2b\" (UniqueName: \"kubernetes.io/projected/60da33d7-f926-4e51-a32e-ea1125d706be-kube-api-access-4lc2b\") pod \"maas-api-revoke-keys-e2e-ipp-a-104b6e-sjwz4\" (UID: \"60da33d7-f926-4e51-a32e-ea1125d706be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-104b6e-sjwz4" Nov 01 16:28:26.940518 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.940485 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/669d4672-f30d-4c7f-9a8c-768a472da306-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a498ae42-krb6v\" (UID: \"669d4672-f30d-4c7f-9a8c-768a472da306\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a498ae42-krb6v" Nov 01 16:28:26.940660 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.940634 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6db8\" (UniqueName: \"kubernetes.io/projected/669d4672-f30d-4c7f-9a8c-768a472da306-kube-api-access-q6db8\") pod \"maas-api-revoke-sub-a498ae42-krb6v\" (UID: \"669d4672-f30d-4c7f-9a8c-768a472da306\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a498ae42-krb6v" Nov 01 16:28:26.943384 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.941264 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/669d4672-f30d-4c7f-9a8c-768a472da306-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a498ae42-krb6v\" (UID: \"669d4672-f30d-4c7f-9a8c-768a472da306\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a498ae42-krb6v" Nov 01 16:28:26.945619 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.945596 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/669d4672-f30d-4c7f-9a8c-768a472da306-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a498ae42-krb6v\" (UID: \"669d4672-f30d-4c7f-9a8c-768a472da306\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a498ae42-krb6v" Nov 01 16:28:26.968006 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.967766 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6db8\" (UniqueName: \"kubernetes.io/projected/669d4672-f30d-4c7f-9a8c-768a472da306-kube-api-access-q6db8\") pod \"maas-api-revoke-sub-a498ae42-krb6v\" (UID: \"669d4672-f30d-4c7f-9a8c-768a472da306\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a498ae42-krb6v" Nov 01 16:28:26.973242 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:26.973184 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" Nov 01 16:28:27.044923 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.043346 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60da33d7-f926-4e51-a32e-ea1125d706be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-104b6e-sjwz4\" (UID: \"60da33d7-f926-4e51-a32e-ea1125d706be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-104b6e-sjwz4" Nov 01 16:28:27.044923 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.043496 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60da33d7-f926-4e51-a32e-ea1125d706be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-104b6e-sjwz4\" (UID: \"60da33d7-f926-4e51-a32e-ea1125d706be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-104b6e-sjwz4" Nov 01 16:28:27.044923 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.043543 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lc2b\" (UniqueName: \"kubernetes.io/projected/60da33d7-f926-4e51-a32e-ea1125d706be-kube-api-access-4lc2b\") pod \"maas-api-revoke-keys-e2e-ipp-a-104b6e-sjwz4\" (UID: \"60da33d7-f926-4e51-a32e-ea1125d706be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-104b6e-sjwz4" Nov 01 16:28:27.045207 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.045164 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60da33d7-f926-4e51-a32e-ea1125d706be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-104b6e-sjwz4\" (UID: \"60da33d7-f926-4e51-a32e-ea1125d706be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-104b6e-sjwz4" Nov 01 16:28:27.047231 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.047212 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60da33d7-f926-4e51-a32e-ea1125d706be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-104b6e-sjwz4\" (UID: \"60da33d7-f926-4e51-a32e-ea1125d706be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-104b6e-sjwz4" Nov 01 16:28:27.051825 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.051800 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lc2b\" (UniqueName: \"kubernetes.io/projected/60da33d7-f926-4e51-a32e-ea1125d706be-kube-api-access-4lc2b\") pod \"maas-api-revoke-keys-e2e-ipp-a-104b6e-sjwz4\" (UID: \"60da33d7-f926-4e51-a32e-ea1125d706be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-104b6e-sjwz4" Nov 01 16:28:27.058119 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.058097 4363 generic.go:356] "Generic (PLEG): container finished" podID="8744fda2-3d14-48d8-8426-323166cca7a3" containerID="430f3237ed8e6bf67c1ea07120f26bb5947c6920b5934151c3541be54ff8ba96" exitCode=0 Nov 01 16:28:27.058285 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.058268 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" event={"ID":"8744fda2-3d14-48d8-8426-323166cca7a3","Type":"ContainerDied","Data":"430f3237ed8e6bf67c1ea07120f26bb5947c6920b5934151c3541be54ff8ba96"} Nov 01 16:28:27.058329 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.058313 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" event={"ID":"8744fda2-3d14-48d8-8426-323166cca7a3","Type":"ContainerDied","Data":"7285936034e376093598c2c930467616717dbefe0c0f84f62c67793efd1a8b36"} Nov 01 16:28:27.058365 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.058334 4363 scope.go:122] "RemoveContainer" containerID="430f3237ed8e6bf67c1ea07120f26bb5947c6920b5934151c3541be54ff8ba96" Nov 01 16:28:27.058453 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.058443 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn" Nov 01 16:28:27.061077 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.061052 4363 generic.go:356] "Generic (PLEG): container finished" podID="f4b349f5-acb2-42b2-8d90-01b2029167f7" containerID="46f0f59d149b20a47b982a1e8d12504e3ff18c8b7e990e8cff3bb909efe59d41" exitCode=0 Nov 01 16:28:27.061190 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.061116 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" event={"ID":"f4b349f5-acb2-42b2-8d90-01b2029167f7","Type":"ContainerDied","Data":"46f0f59d149b20a47b982a1e8d12504e3ff18c8b7e990e8cff3bb909efe59d41"} Nov 01 16:28:27.061190 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.061151 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" event={"ID":"f4b349f5-acb2-42b2-8d90-01b2029167f7","Type":"ContainerDied","Data":"67969328a1c7f2040039b4136efd9a6065a0e8ec3ae5dbe6c9782340c444130a"} Nov 01 16:28:27.061190 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.061166 4363 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67969328a1c7f2040039b4136efd9a6065a0e8ec3ae5dbe6c9782340c444130a" Nov 01 16:28:27.065861 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.065840 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a498ae42-krb6v" Nov 01 16:28:27.125950 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.125112 4363 scope.go:122] "RemoveContainer" containerID="430f3237ed8e6bf67c1ea07120f26bb5947c6920b5934151c3541be54ff8ba96" Nov 01 16:28:27.125950 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:28:27.125717 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"430f3237ed8e6bf67c1ea07120f26bb5947c6920b5934151c3541be54ff8ba96\": container with ID starting with 430f3237ed8e6bf67c1ea07120f26bb5947c6920b5934151c3541be54ff8ba96 not found: ID does not exist" containerID="430f3237ed8e6bf67c1ea07120f26bb5947c6920b5934151c3541be54ff8ba96" Nov 01 16:28:27.125950 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.125761 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"430f3237ed8e6bf67c1ea07120f26bb5947c6920b5934151c3541be54ff8ba96"} err="failed to get container status \"430f3237ed8e6bf67c1ea07120f26bb5947c6920b5934151c3541be54ff8ba96\": rpc error: code = NotFound desc = could not find container \"430f3237ed8e6bf67c1ea07120f26bb5947c6920b5934151c3541be54ff8ba96\": container with ID starting with 430f3237ed8e6bf67c1ea07120f26bb5947c6920b5934151c3541be54ff8ba96 not found: ID does not exist" Nov 01 16:28:27.125950 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.125788 4363 scope.go:122] "RemoveContainer" containerID="db8bd7437ddcdc6617b7e6c3dc9d860fab8ad93befb0669687b273a8d20eed82" Nov 01 16:28:27.129428 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.129402 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-104b6e-sjwz4" Nov 01 16:28:27.134333 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.133996 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" Nov 01 16:28:27.144701 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.144678 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8744fda2-3d14-48d8-8426-323166cca7a3-istio-token\" (UniqueName: \"kubernetes.io/projected/8744fda2-3d14-48d8-8426-323166cca7a3-istio-token\") pod \"8744fda2-3d14-48d8-8426-323166cca7a3\" (UID: \"8744fda2-3d14-48d8-8426-323166cca7a3\") " Nov 01 16:28:27.144829 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.144736 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8744fda2-3d14-48d8-8426-323166cca7a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8744fda2-3d14-48d8-8426-323166cca7a3-credential-socket\") pod \"8744fda2-3d14-48d8-8426-323166cca7a3\" (UID: \"8744fda2-3d14-48d8-8426-323166cca7a3\") " Nov 01 16:28:27.144891 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.144833 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8744fda2-3d14-48d8-8426-323166cca7a3-kube-api-access-cqkwv\" (UniqueName: \"kubernetes.io/projected/8744fda2-3d14-48d8-8426-323166cca7a3-kube-api-access-cqkwv\") pod \"8744fda2-3d14-48d8-8426-323166cca7a3\" (UID: \"8744fda2-3d14-48d8-8426-323166cca7a3\") " Nov 01 16:28:27.145054 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.145039 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8744fda2-3d14-48d8-8426-323166cca7a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8744fda2-3d14-48d8-8426-323166cca7a3-workload-socket\") pod \"8744fda2-3d14-48d8-8426-323166cca7a3\" (UID: \"8744fda2-3d14-48d8-8426-323166cca7a3\") " Nov 01 16:28:27.145131 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.145101 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8744fda2-3d14-48d8-8426-323166cca7a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8744fda2-3d14-48d8-8426-323166cca7a3-istio-envoy\") pod \"8744fda2-3d14-48d8-8426-323166cca7a3\" (UID: \"8744fda2-3d14-48d8-8426-323166cca7a3\") " Nov 01 16:28:27.145185 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.145138 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8744fda2-3d14-48d8-8426-323166cca7a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8744fda2-3d14-48d8-8426-323166cca7a3-workload-certs\") pod \"8744fda2-3d14-48d8-8426-323166cca7a3\" (UID: \"8744fda2-3d14-48d8-8426-323166cca7a3\") " Nov 01 16:28:27.145236 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.145208 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8744fda2-3d14-48d8-8426-323166cca7a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8744fda2-3d14-48d8-8426-323166cca7a3-istio-podinfo\") pod \"8744fda2-3d14-48d8-8426-323166cca7a3\" (UID: \"8744fda2-3d14-48d8-8426-323166cca7a3\") " Nov 01 16:28:27.145295 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.145264 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8744fda2-3d14-48d8-8426-323166cca7a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8744fda2-3d14-48d8-8426-323166cca7a3-istiod-ca-cert\") pod \"8744fda2-3d14-48d8-8426-323166cca7a3\" (UID: \"8744fda2-3d14-48d8-8426-323166cca7a3\") " Nov 01 16:28:27.145355 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.145298 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8744fda2-3d14-48d8-8426-323166cca7a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8744fda2-3d14-48d8-8426-323166cca7a3-istio-data\") pod \"8744fda2-3d14-48d8-8426-323166cca7a3\" (UID: \"8744fda2-3d14-48d8-8426-323166cca7a3\") " Nov 01 16:28:27.146296 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.146276 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8744fda2-3d14-48d8-8426-323166cca7a3-credential-socket" pod "8744fda2-3d14-48d8-8426-323166cca7a3" (UID: "8744fda2-3d14-48d8-8426-323166cca7a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:27.146601 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.146583 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8744fda2-3d14-48d8-8426-323166cca7a3-workload-socket" pod "8744fda2-3d14-48d8-8426-323166cca7a3" (UID: "8744fda2-3d14-48d8-8426-323166cca7a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:27.146683 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.146640 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8744fda2-3d14-48d8-8426-323166cca7a3-istio-data" pod "8744fda2-3d14-48d8-8426-323166cca7a3" (UID: "8744fda2-3d14-48d8-8426-323166cca7a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:27.146865 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.146832 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8744fda2-3d14-48d8-8426-323166cca7a3-workload-certs" pod "8744fda2-3d14-48d8-8426-323166cca7a3" (UID: "8744fda2-3d14-48d8-8426-323166cca7a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:27.146965 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.146925 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8744fda2-3d14-48d8-8426-323166cca7a3-istiod-ca-cert" pod "8744fda2-3d14-48d8-8426-323166cca7a3" (UID: "8744fda2-3d14-48d8-8426-323166cca7a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:28:27.148693 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.148657 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8744fda2-3d14-48d8-8426-323166cca7a3-istio-envoy" pod "8744fda2-3d14-48d8-8426-323166cca7a3" (UID: "8744fda2-3d14-48d8-8426-323166cca7a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:27.148980 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.148957 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8744fda2-3d14-48d8-8426-323166cca7a3-kube-api-access-cqkwv" pod "8744fda2-3d14-48d8-8426-323166cca7a3" (UID: "8744fda2-3d14-48d8-8426-323166cca7a3"). InnerVolumeSpecName "kube-api-access-cqkwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:27.149188 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.149076 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8744fda2-3d14-48d8-8426-323166cca7a3-istio-podinfo" pod "8744fda2-3d14-48d8-8426-323166cca7a3" (UID: "8744fda2-3d14-48d8-8426-323166cca7a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:28:27.149188 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.149110 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8744fda2-3d14-48d8-8426-323166cca7a3-istio-token" pod "8744fda2-3d14-48d8-8426-323166cca7a3" (UID: "8744fda2-3d14-48d8-8426-323166cca7a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:27.248791 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.248738 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4b349f5-acb2-42b2-8d90-01b2029167f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4b349f5-acb2-42b2-8d90-01b2029167f7-istio-data\") pod \"f4b349f5-acb2-42b2-8d90-01b2029167f7\" (UID: \"f4b349f5-acb2-42b2-8d90-01b2029167f7\") " Nov 01 16:28:27.248915 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.248794 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4b349f5-acb2-42b2-8d90-01b2029167f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4b349f5-acb2-42b2-8d90-01b2029167f7-istio-envoy\") pod \"f4b349f5-acb2-42b2-8d90-01b2029167f7\" (UID: \"f4b349f5-acb2-42b2-8d90-01b2029167f7\") " Nov 01 16:28:27.248915 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.248859 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4b349f5-acb2-42b2-8d90-01b2029167f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b349f5-acb2-42b2-8d90-01b2029167f7-workload-socket\") pod \"f4b349f5-acb2-42b2-8d90-01b2029167f7\" (UID: \"f4b349f5-acb2-42b2-8d90-01b2029167f7\") " Nov 01 16:28:27.248915 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.248905 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4b349f5-acb2-42b2-8d90-01b2029167f7-kube-api-access-5mmk5\" (UniqueName: \"kubernetes.io/projected/f4b349f5-acb2-42b2-8d90-01b2029167f7-kube-api-access-5mmk5\") pod \"f4b349f5-acb2-42b2-8d90-01b2029167f7\" (UID: \"f4b349f5-acb2-42b2-8d90-01b2029167f7\") " Nov 01 16:28:27.250287 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.250259 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4b349f5-acb2-42b2-8d90-01b2029167f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4b349f5-acb2-42b2-8d90-01b2029167f7-workload-certs\") pod \"f4b349f5-acb2-42b2-8d90-01b2029167f7\" (UID: \"f4b349f5-acb2-42b2-8d90-01b2029167f7\") " Nov 01 16:28:27.250404 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.250343 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4b349f5-acb2-42b2-8d90-01b2029167f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4b349f5-acb2-42b2-8d90-01b2029167f7-istiod-ca-cert\") pod \"f4b349f5-acb2-42b2-8d90-01b2029167f7\" (UID: \"f4b349f5-acb2-42b2-8d90-01b2029167f7\") " Nov 01 16:28:27.250469 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.250417 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4b349f5-acb2-42b2-8d90-01b2029167f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4b349f5-acb2-42b2-8d90-01b2029167f7-istio-podinfo\") pod \"f4b349f5-acb2-42b2-8d90-01b2029167f7\" (UID: \"f4b349f5-acb2-42b2-8d90-01b2029167f7\") " Nov 01 16:28:27.250469 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.250447 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4b349f5-acb2-42b2-8d90-01b2029167f7-istio-token\" (UniqueName: \"kubernetes.io/projected/f4b349f5-acb2-42b2-8d90-01b2029167f7-istio-token\") pod \"f4b349f5-acb2-42b2-8d90-01b2029167f7\" (UID: \"f4b349f5-acb2-42b2-8d90-01b2029167f7\") " Nov 01 16:28:27.250604 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.250495 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4b349f5-acb2-42b2-8d90-01b2029167f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b349f5-acb2-42b2-8d90-01b2029167f7-credential-socket\") pod \"f4b349f5-acb2-42b2-8d90-01b2029167f7\" (UID: \"f4b349f5-acb2-42b2-8d90-01b2029167f7\") " Nov 01 16:28:27.251494 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.251468 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8744fda2-3d14-48d8-8426-323166cca7a3-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:27.251655 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.251501 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8744fda2-3d14-48d8-8426-323166cca7a3-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:27.251655 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.251518 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8744fda2-3d14-48d8-8426-323166cca7a3-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:27.251655 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.251533 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8744fda2-3d14-48d8-8426-323166cca7a3-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:27.251655 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.251548 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8744fda2-3d14-48d8-8426-323166cca7a3-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:27.251655 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.251562 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8744fda2-3d14-48d8-8426-323166cca7a3-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:27.251655 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.251577 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8744fda2-3d14-48d8-8426-323166cca7a3-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:27.251655 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.251591 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqkwv\" (UniqueName: \"kubernetes.io/projected/8744fda2-3d14-48d8-8426-323166cca7a3-kube-api-access-cqkwv\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:27.251655 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.251606 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8744fda2-3d14-48d8-8426-323166cca7a3-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:27.255895 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.255847 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4b349f5-acb2-42b2-8d90-01b2029167f7-istiod-ca-cert" pod "f4b349f5-acb2-42b2-8d90-01b2029167f7" (UID: "f4b349f5-acb2-42b2-8d90-01b2029167f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:28:27.257401 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.256952 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b349f5-acb2-42b2-8d90-01b2029167f7-credential-socket" pod "f4b349f5-acb2-42b2-8d90-01b2029167f7" (UID: "f4b349f5-acb2-42b2-8d90-01b2029167f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:27.257401 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.257104 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b349f5-acb2-42b2-8d90-01b2029167f7-workload-certs" pod "f4b349f5-acb2-42b2-8d90-01b2029167f7" (UID: "f4b349f5-acb2-42b2-8d90-01b2029167f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:27.257401 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.257163 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b349f5-acb2-42b2-8d90-01b2029167f7-workload-socket" pod "f4b349f5-acb2-42b2-8d90-01b2029167f7" (UID: "f4b349f5-acb2-42b2-8d90-01b2029167f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:27.260722 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.259512 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b349f5-acb2-42b2-8d90-01b2029167f7-istio-data" pod "f4b349f5-acb2-42b2-8d90-01b2029167f7" (UID: "f4b349f5-acb2-42b2-8d90-01b2029167f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:27.260722 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.260007 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b349f5-acb2-42b2-8d90-01b2029167f7-istio-envoy" pod "f4b349f5-acb2-42b2-8d90-01b2029167f7" (UID: "f4b349f5-acb2-42b2-8d90-01b2029167f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:27.260997 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.260962 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4b349f5-acb2-42b2-8d90-01b2029167f7-istio-podinfo" pod "f4b349f5-acb2-42b2-8d90-01b2029167f7" (UID: "f4b349f5-acb2-42b2-8d90-01b2029167f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:28:27.261123 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.261053 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4b349f5-acb2-42b2-8d90-01b2029167f7-kube-api-access-5mmk5" pod "f4b349f5-acb2-42b2-8d90-01b2029167f7" (UID: "f4b349f5-acb2-42b2-8d90-01b2029167f7"). InnerVolumeSpecName "kube-api-access-5mmk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:27.262672 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.262133 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4b349f5-acb2-42b2-8d90-01b2029167f7-istio-token" pod "f4b349f5-acb2-42b2-8d90-01b2029167f7" (UID: "f4b349f5-acb2-42b2-8d90-01b2029167f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:27.353805 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.353768 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4b349f5-acb2-42b2-8d90-01b2029167f7-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:27.353930 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.353830 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4b349f5-acb2-42b2-8d90-01b2029167f7-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:27.353930 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.353848 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4b349f5-acb2-42b2-8d90-01b2029167f7-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:27.353930 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.353863 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4b349f5-acb2-42b2-8d90-01b2029167f7-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:27.353930 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.353877 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b349f5-acb2-42b2-8d90-01b2029167f7-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:27.353930 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.353909 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4b349f5-acb2-42b2-8d90-01b2029167f7-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:27.354131 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.353950 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4b349f5-acb2-42b2-8d90-01b2029167f7-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:27.354131 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.353982 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b349f5-acb2-42b2-8d90-01b2029167f7-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:27.354131 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.353998 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mmk5\" (UniqueName: \"kubernetes.io/projected/f4b349f5-acb2-42b2-8d90-01b2029167f7-kube-api-access-5mmk5\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:27.427078 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.422967 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn"] Nov 01 16:28:27.435482 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.435452 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-cvmjn"] Nov 01 16:28:27.438630 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.438604 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a498ae42-krb6v"] Nov 01 16:28:27.439567 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:28:27.439539 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod669d4672_f30d_4c7f_9a8c_768a472da306.slice/crio-72f0babdcd046985bc06fbc04fab6a11e2aed94ac85aa1b5947f8e6301e93efb WatchSource:0}: Error finding container 72f0babdcd046985bc06fbc04fab6a11e2aed94ac85aa1b5947f8e6301e93efb: Status 404 returned error can't find the container with id 72f0babdcd046985bc06fbc04fab6a11e2aed94ac85aa1b5947f8e6301e93efb Nov 01 16:28:27.767678 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.767640 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-104b6e-sjwz4"] Nov 01 16:28:27.855882 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.855825 4363 patch_prober.go:29] interesting pod/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Nov 01 16:28:27.856063 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:27.855911 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" podUID="742547a4-fbd7-40d3-a7a3-86fc91fd7cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Nov 01 16:28:28.073969 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:28.073888 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg" Nov 01 16:28:28.075702 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:28.075667 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-104b6e-sjwz4" event={"ID":"60da33d7-f926-4e51-a32e-ea1125d706be","Type":"ContainerStarted","Data":"48ed0ade0f6d1db78b1a5beeeb982cf94e7340c00ec9bd4e7f4726afe29b25ac"} Nov 01 16:28:28.098038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:28.085268 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a498ae42-krb6v" event={"ID":"669d4672-f30d-4c7f-9a8c-768a472da306","Type":"ContainerStarted","Data":"68b9557e25d5e5185456b38789a0f131bf85af0e5ef7929ef140aac47031c22c"} Nov 01 16:28:28.098038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:28.085325 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a498ae42-krb6v" event={"ID":"669d4672-f30d-4c7f-9a8c-768a472da306","Type":"ContainerStarted","Data":"72f0babdcd046985bc06fbc04fab6a11e2aed94ac85aa1b5947f8e6301e93efb"} Nov 01 16:28:28.134506 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:28.134458 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a498ae42-krb6v" podStartSLOduration=2.134443742 podStartE2EDuration="2.134443742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:28:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:28:28.128714533 +0000 UTC m=+2441.460452185" watchObservedRunningTime="2026-11-01 16:28:28.134443742 +0000 UTC m=+2441.466181392" Nov 01 16:28:28.146549 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:28.146518 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg"] Nov 01 16:28:28.150622 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:28.150596 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-n72bg"] Nov 01 16:28:28.801259 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:28.801230 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8744fda2-3d14-48d8-8426-323166cca7a3" path="/var/lib/kubelet/pods/8744fda2-3d14-48d8-8426-323166cca7a3/volumes" Nov 01 16:28:28.801760 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:28.801745 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4b349f5-acb2-42b2-8d90-01b2029167f7" path="/var/lib/kubelet/pods/f4b349f5-acb2-42b2-8d90-01b2029167f7/volumes" Nov 01 16:28:28.860578 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:28.860554 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" Nov 01 16:28:28.861646 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:28.861626 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" Nov 01 16:28:29.114488 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:29.114399 4363 generic.go:356] "Generic (PLEG): container finished" podID="60da33d7-f926-4e51-a32e-ea1125d706be" containerID="f8415a4045b543deeadededfd9d59061bb7225d0a75bdf187cf98588e9f26564" exitCode=0 Nov 01 16:28:29.114671 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:29.114530 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-104b6e-sjwz4" event={"ID":"60da33d7-f926-4e51-a32e-ea1125d706be","Type":"ContainerDied","Data":"f8415a4045b543deeadededfd9d59061bb7225d0a75bdf187cf98588e9f26564"} Nov 01 16:28:29.116697 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:29.116669 4363 generic.go:356] "Generic (PLEG): container finished" podID="669d4672-f30d-4c7f-9a8c-768a472da306" containerID="68b9557e25d5e5185456b38789a0f131bf85af0e5ef7929ef140aac47031c22c" exitCode=0 Nov 01 16:28:29.116831 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:29.116759 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a498ae42-krb6v" event={"ID":"669d4672-f30d-4c7f-9a8c-768a472da306","Type":"ContainerDied","Data":"68b9557e25d5e5185456b38789a0f131bf85af0e5ef7929ef140aac47031c22c"} Nov 01 16:28:30.438498 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:30.438464 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a498ae42-krb6v" Nov 01 16:28:30.482698 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:30.481741 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-9a482b/rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl" Nov 01 16:28:30.496938 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:30.496793 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-104b6e-sjwz4" Nov 01 16:28:30.518280 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:30.517954 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-9a482b/rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl" Nov 01 16:28:30.544349 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:30.544314 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/669d4672-f30d-4c7f-9a8c-768a472da306-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/669d4672-f30d-4c7f-9a8c-768a472da306-maas-api-openshift-ca\") pod \"669d4672-f30d-4c7f-9a8c-768a472da306\" (UID: \"669d4672-f30d-4c7f-9a8c-768a472da306\") " Nov 01 16:28:30.544517 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:30.544467 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/669d4672-f30d-4c7f-9a8c-768a472da306-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/669d4672-f30d-4c7f-9a8c-768a472da306-maas-api-xks-ca\") pod \"669d4672-f30d-4c7f-9a8c-768a472da306\" (UID: \"669d4672-f30d-4c7f-9a8c-768a472da306\") " Nov 01 16:28:30.544517 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:30.544508 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/669d4672-f30d-4c7f-9a8c-768a472da306-kube-api-access-q6db8\" (UniqueName: \"kubernetes.io/projected/669d4672-f30d-4c7f-9a8c-768a472da306-kube-api-access-q6db8\") pod \"669d4672-f30d-4c7f-9a8c-768a472da306\" (UID: \"669d4672-f30d-4c7f-9a8c-768a472da306\") " Nov 01 16:28:30.544754 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:30.544732 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/669d4672-f30d-4c7f-9a8c-768a472da306-maas-api-openshift-ca" pod "669d4672-f30d-4c7f-9a8c-768a472da306" (UID: "669d4672-f30d-4c7f-9a8c-768a472da306"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:28:30.545289 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:30.545263 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/669d4672-f30d-4c7f-9a8c-768a472da306-maas-api-openshift-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:30.562210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:30.553147 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/669d4672-f30d-4c7f-9a8c-768a472da306-maas-api-xks-ca" pod "669d4672-f30d-4c7f-9a8c-768a472da306" (UID: "669d4672-f30d-4c7f-9a8c-768a472da306"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:28:30.562210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:30.559181 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/669d4672-f30d-4c7f-9a8c-768a472da306-kube-api-access-q6db8" pod "669d4672-f30d-4c7f-9a8c-768a472da306" (UID: "669d4672-f30d-4c7f-9a8c-768a472da306"). InnerVolumeSpecName "kube-api-access-q6db8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:30.646620 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:30.646589 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60da33d7-f926-4e51-a32e-ea1125d706be-kube-api-access-4lc2b\" (UniqueName: \"kubernetes.io/projected/60da33d7-f926-4e51-a32e-ea1125d706be-kube-api-access-4lc2b\") pod \"60da33d7-f926-4e51-a32e-ea1125d706be\" (UID: \"60da33d7-f926-4e51-a32e-ea1125d706be\") " Nov 01 16:28:30.646780 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:30.646763 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60da33d7-f926-4e51-a32e-ea1125d706be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60da33d7-f926-4e51-a32e-ea1125d706be-maas-api-xks-ca\") pod \"60da33d7-f926-4e51-a32e-ea1125d706be\" (UID: \"60da33d7-f926-4e51-a32e-ea1125d706be\") " Nov 01 16:28:30.646839 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:30.646827 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60da33d7-f926-4e51-a32e-ea1125d706be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60da33d7-f926-4e51-a32e-ea1125d706be-maas-api-openshift-ca\") pod \"60da33d7-f926-4e51-a32e-ea1125d706be\" (UID: \"60da33d7-f926-4e51-a32e-ea1125d706be\") " Nov 01 16:28:30.647227 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:30.647202 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60da33d7-f926-4e51-a32e-ea1125d706be-maas-api-openshift-ca" pod "60da33d7-f926-4e51-a32e-ea1125d706be" (UID: "60da33d7-f926-4e51-a32e-ea1125d706be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:28:30.647562 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:30.647544 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/669d4672-f30d-4c7f-9a8c-768a472da306-maas-api-xks-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:30.647645 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:30.647567 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6db8\" (UniqueName: \"kubernetes.io/projected/669d4672-f30d-4c7f-9a8c-768a472da306-kube-api-access-q6db8\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:30.647645 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:30.647582 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60da33d7-f926-4e51-a32e-ea1125d706be-maas-api-openshift-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:30.648858 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:30.648835 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60da33d7-f926-4e51-a32e-ea1125d706be-kube-api-access-4lc2b" pod "60da33d7-f926-4e51-a32e-ea1125d706be" (UID: "60da33d7-f926-4e51-a32e-ea1125d706be"). InnerVolumeSpecName "kube-api-access-4lc2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:30.648944 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:30.648925 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60da33d7-f926-4e51-a32e-ea1125d706be-maas-api-xks-ca" pod "60da33d7-f926-4e51-a32e-ea1125d706be" (UID: "60da33d7-f926-4e51-a32e-ea1125d706be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:28:30.748708 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:30.748630 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60da33d7-f926-4e51-a32e-ea1125d706be-maas-api-xks-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:30.748708 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:30.748656 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lc2b\" (UniqueName: \"kubernetes.io/projected/60da33d7-f926-4e51-a32e-ea1125d706be-kube-api-access-4lc2b\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:31.140052 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:31.139992 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-104b6e-sjwz4" Nov 01 16:28:31.140236 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:31.140059 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-104b6e-sjwz4" event={"ID":"60da33d7-f926-4e51-a32e-ea1125d706be","Type":"ContainerDied","Data":"48ed0ade0f6d1db78b1a5beeeb982cf94e7340c00ec9bd4e7f4726afe29b25ac"} Nov 01 16:28:31.140236 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:31.140105 4363 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48ed0ade0f6d1db78b1a5beeeb982cf94e7340c00ec9bd4e7f4726afe29b25ac" Nov 01 16:28:31.141502 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:31.141485 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a498ae42-krb6v" Nov 01 16:28:31.141615 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:31.141527 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a498ae42-krb6v" event={"ID":"669d4672-f30d-4c7f-9a8c-768a472da306","Type":"ContainerDied","Data":"72f0babdcd046985bc06fbc04fab6a11e2aed94ac85aa1b5947f8e6301e93efb"} Nov 01 16:28:31.141615 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:31.141569 4363 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72f0babdcd046985bc06fbc04fab6a11e2aed94ac85aa1b5947f8e6301e93efb" Nov 01 16:28:31.154200 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:31.154177 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-9a482b/rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl" Nov 01 16:28:33.044808 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.044771 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5"] Nov 01 16:28:33.045591 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.045575 4363 cpu_manager.go:414] "containerMap: removing container" podUID="8744fda2-3d14-48d8-8426-323166cca7a3" containerName="istio-proxy" Nov 01 16:28:33.045643 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.045592 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8744fda2-3d14-48d8-8426-323166cca7a3" containerName="istio-proxy" Nov 01 16:28:33.045643 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.045610 4363 cpu_manager.go:414] "containerMap: removing container" podUID="f4b349f5-acb2-42b2-8d90-01b2029167f7" containerName="istio-proxy" Nov 01 16:28:33.045643 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.045615 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4b349f5-acb2-42b2-8d90-01b2029167f7" containerName="istio-proxy" Nov 01 16:28:33.045643 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.045639 4363 cpu_manager.go:414] "containerMap: removing container" podUID="669d4672-f30d-4c7f-9a8c-768a472da306" containerName="revoke-keys" Nov 01 16:28:33.045643 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.045644 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="669d4672-f30d-4c7f-9a8c-768a472da306" containerName="revoke-keys" Nov 01 16:28:33.045802 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.045663 4363 cpu_manager.go:414] "containerMap: removing container" podUID="60da33d7-f926-4e51-a32e-ea1125d706be" containerName="revoke-keys" Nov 01 16:28:33.045802 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.045668 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60da33d7-f926-4e51-a32e-ea1125d706be" containerName="revoke-keys" Nov 01 16:28:33.045802 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.045778 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="669d4672-f30d-4c7f-9a8c-768a472da306" containerName="revoke-keys" Nov 01 16:28:33.045802 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.045790 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="60da33d7-f926-4e51-a32e-ea1125d706be" containerName="revoke-keys" Nov 01 16:28:33.045802 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.045797 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4b349f5-acb2-42b2-8d90-01b2029167f7" containerName="istio-proxy" Nov 01 16:28:33.046059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.045806 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="8744fda2-3d14-48d8-8426-323166cca7a3" containerName="istio-proxy" Nov 01 16:28:33.046059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.045812 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4b349f5-acb2-42b2-8d90-01b2029167f7" containerName="istio-proxy" Nov 01 16:28:33.050116 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.050096 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" Nov 01 16:28:33.061146 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.061125 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5"] Nov 01 16:28:33.213683 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.213645 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl88k\" (UniqueName: \"kubernetes.io/projected/ca74811d-13fc-4a4c-bfd9-9e787392ffce-kube-api-access-gl88k\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5\" (UID: \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" Nov 01 16:28:33.213865 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.213709 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca74811d-13fc-4a4c-bfd9-9e787392ffce-workload-certs\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5\" (UID: \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" Nov 01 16:28:33.213865 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.213835 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca74811d-13fc-4a4c-bfd9-9e787392ffce-credential-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5\" (UID: \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" Nov 01 16:28:33.213988 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.213930 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca74811d-13fc-4a4c-bfd9-9e787392ffce-istio-data\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5\" (UID: \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" Nov 01 16:28:33.214097 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.214053 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca74811d-13fc-4a4c-bfd9-9e787392ffce-workload-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5\" (UID: \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" Nov 01 16:28:33.214168 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.214097 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca74811d-13fc-4a4c-bfd9-9e787392ffce-istiod-ca-cert\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5\" (UID: \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" Nov 01 16:28:33.214232 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.214193 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca74811d-13fc-4a4c-bfd9-9e787392ffce-istio-podinfo\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5\" (UID: \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" Nov 01 16:28:33.214232 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.214223 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca74811d-13fc-4a4c-bfd9-9e787392ffce-istio-envoy\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5\" (UID: \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" Nov 01 16:28:33.214332 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.214275 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca74811d-13fc-4a4c-bfd9-9e787392ffce-istio-token\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5\" (UID: \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" Nov 01 16:28:33.315991 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.315914 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca74811d-13fc-4a4c-bfd9-9e787392ffce-workload-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5\" (UID: \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" Nov 01 16:28:33.315991 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.315980 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca74811d-13fc-4a4c-bfd9-9e787392ffce-istiod-ca-cert\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5\" (UID: \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" Nov 01 16:28:33.316272 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.316098 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca74811d-13fc-4a4c-bfd9-9e787392ffce-istio-podinfo\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5\" (UID: \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" Nov 01 16:28:33.316272 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.316142 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca74811d-13fc-4a4c-bfd9-9e787392ffce-istio-envoy\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5\" (UID: \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" Nov 01 16:28:33.316272 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.316210 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca74811d-13fc-4a4c-bfd9-9e787392ffce-istio-token\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5\" (UID: \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" Nov 01 16:28:33.316440 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.316288 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl88k\" (UniqueName: \"kubernetes.io/projected/ca74811d-13fc-4a4c-bfd9-9e787392ffce-kube-api-access-gl88k\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5\" (UID: \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" Nov 01 16:28:33.316440 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.316359 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca74811d-13fc-4a4c-bfd9-9e787392ffce-workload-certs\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5\" (UID: \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" Nov 01 16:28:33.316440 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.316393 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca74811d-13fc-4a4c-bfd9-9e787392ffce-workload-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5\" (UID: \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" Nov 01 16:28:33.316440 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.316413 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca74811d-13fc-4a4c-bfd9-9e787392ffce-credential-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5\" (UID: \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" Nov 01 16:28:33.316817 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.316482 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca74811d-13fc-4a4c-bfd9-9e787392ffce-istio-data\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5\" (UID: \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" Nov 01 16:28:33.316882 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.316834 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca74811d-13fc-4a4c-bfd9-9e787392ffce-istio-data\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5\" (UID: \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" Nov 01 16:28:33.316882 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.316867 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca74811d-13fc-4a4c-bfd9-9e787392ffce-istiod-ca-cert\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5\" (UID: \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" Nov 01 16:28:33.317031 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.316907 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca74811d-13fc-4a4c-bfd9-9e787392ffce-workload-certs\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5\" (UID: \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" Nov 01 16:28:33.317189 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.317158 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca74811d-13fc-4a4c-bfd9-9e787392ffce-credential-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5\" (UID: \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" Nov 01 16:28:33.318844 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.318820 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca74811d-13fc-4a4c-bfd9-9e787392ffce-istio-envoy\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5\" (UID: \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" Nov 01 16:28:33.319268 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.319244 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca74811d-13fc-4a4c-bfd9-9e787392ffce-istio-podinfo\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5\" (UID: \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" Nov 01 16:28:33.324387 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.324364 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl88k\" (UniqueName: \"kubernetes.io/projected/ca74811d-13fc-4a4c-bfd9-9e787392ffce-kube-api-access-gl88k\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5\" (UID: \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" Nov 01 16:28:33.324848 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.324826 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca74811d-13fc-4a4c-bfd9-9e787392ffce-istio-token\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5\" (UID: \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" Nov 01 16:28:33.367274 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.367247 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" Nov 01 16:28:33.613492 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.613443 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p"] Nov 01 16:28:33.614554 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.614527 4363 cpu_manager.go:414] "containerMap: removing container" podUID="f4b349f5-acb2-42b2-8d90-01b2029167f7" containerName="istio-proxy" Nov 01 16:28:33.614554 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.614551 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4b349f5-acb2-42b2-8d90-01b2029167f7" containerName="istio-proxy" Nov 01 16:28:33.620121 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.620078 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" Nov 01 16:28:33.649291 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.649244 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p"] Nov 01 16:28:33.662292 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:28:33.662264 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca74811d_13fc_4a4c_bfd9_9e787392ffce.slice/crio-27fc6c87658b6433dd3dc5398c0fdd7e4d461e43162765efd466035a75d23b19 WatchSource:0}: Error finding container 27fc6c87658b6433dd3dc5398c0fdd7e4d461e43162765efd466035a75d23b19: Status 404 returned error can't find the container with id 27fc6c87658b6433dd3dc5398c0fdd7e4d461e43162765efd466035a75d23b19 Nov 01 16:28:33.664475 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.664383 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5"] Nov 01 16:28:33.666875 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.666833 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:28:33.666960 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.666921 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:28:33.667011 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.666965 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:28:33.723285 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.723257 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c176da5b-fd83-454e-bbb7-bb793807cb67-istio-token\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p\" (UID: \"c176da5b-fd83-454e-bbb7-bb793807cb67\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" Nov 01 16:28:33.723406 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.723384 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c176da5b-fd83-454e-bbb7-bb793807cb67-workload-certs\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p\" (UID: \"c176da5b-fd83-454e-bbb7-bb793807cb67\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" Nov 01 16:28:33.723473 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.723439 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c176da5b-fd83-454e-bbb7-bb793807cb67-credential-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p\" (UID: \"c176da5b-fd83-454e-bbb7-bb793807cb67\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" Nov 01 16:28:33.723548 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.723497 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c176da5b-fd83-454e-bbb7-bb793807cb67-istiod-ca-cert\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p\" (UID: \"c176da5b-fd83-454e-bbb7-bb793807cb67\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" Nov 01 16:28:33.723616 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.723592 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx8dc\" (UniqueName: \"kubernetes.io/projected/c176da5b-fd83-454e-bbb7-bb793807cb67-kube-api-access-fx8dc\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p\" (UID: \"c176da5b-fd83-454e-bbb7-bb793807cb67\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" Nov 01 16:28:33.723672 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.723639 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c176da5b-fd83-454e-bbb7-bb793807cb67-istio-podinfo\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p\" (UID: \"c176da5b-fd83-454e-bbb7-bb793807cb67\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" Nov 01 16:28:33.723732 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.723694 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c176da5b-fd83-454e-bbb7-bb793807cb67-istio-data\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p\" (UID: \"c176da5b-fd83-454e-bbb7-bb793807cb67\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" Nov 01 16:28:33.723807 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.723788 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c176da5b-fd83-454e-bbb7-bb793807cb67-istio-envoy\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p\" (UID: \"c176da5b-fd83-454e-bbb7-bb793807cb67\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" Nov 01 16:28:33.723865 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.723844 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c176da5b-fd83-454e-bbb7-bb793807cb67-workload-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p\" (UID: \"c176da5b-fd83-454e-bbb7-bb793807cb67\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" Nov 01 16:28:33.825025 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.824975 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c176da5b-fd83-454e-bbb7-bb793807cb67-credential-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p\" (UID: \"c176da5b-fd83-454e-bbb7-bb793807cb67\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" Nov 01 16:28:33.825221 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.825088 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c176da5b-fd83-454e-bbb7-bb793807cb67-istiod-ca-cert\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p\" (UID: \"c176da5b-fd83-454e-bbb7-bb793807cb67\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" Nov 01 16:28:33.825221 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.825162 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx8dc\" (UniqueName: \"kubernetes.io/projected/c176da5b-fd83-454e-bbb7-bb793807cb67-kube-api-access-fx8dc\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p\" (UID: \"c176da5b-fd83-454e-bbb7-bb793807cb67\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" Nov 01 16:28:33.825348 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.825227 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c176da5b-fd83-454e-bbb7-bb793807cb67-istio-podinfo\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p\" (UID: \"c176da5b-fd83-454e-bbb7-bb793807cb67\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" Nov 01 16:28:33.825348 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.825253 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c176da5b-fd83-454e-bbb7-bb793807cb67-istio-data\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p\" (UID: \"c176da5b-fd83-454e-bbb7-bb793807cb67\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" Nov 01 16:28:33.825348 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.825301 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c176da5b-fd83-454e-bbb7-bb793807cb67-istio-envoy\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p\" (UID: \"c176da5b-fd83-454e-bbb7-bb793807cb67\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" Nov 01 16:28:33.825348 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.825325 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c176da5b-fd83-454e-bbb7-bb793807cb67-workload-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p\" (UID: \"c176da5b-fd83-454e-bbb7-bb793807cb67\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" Nov 01 16:28:33.825556 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.825399 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c176da5b-fd83-454e-bbb7-bb793807cb67-credential-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p\" (UID: \"c176da5b-fd83-454e-bbb7-bb793807cb67\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" Nov 01 16:28:33.825556 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.825459 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c176da5b-fd83-454e-bbb7-bb793807cb67-istio-token\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p\" (UID: \"c176da5b-fd83-454e-bbb7-bb793807cb67\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" Nov 01 16:28:33.825674 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.825587 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c176da5b-fd83-454e-bbb7-bb793807cb67-workload-certs\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p\" (UID: \"c176da5b-fd83-454e-bbb7-bb793807cb67\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" Nov 01 16:28:33.825674 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.825641 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c176da5b-fd83-454e-bbb7-bb793807cb67-workload-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p\" (UID: \"c176da5b-fd83-454e-bbb7-bb793807cb67\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" Nov 01 16:28:33.825850 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.825824 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c176da5b-fd83-454e-bbb7-bb793807cb67-istiod-ca-cert\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p\" (UID: \"c176da5b-fd83-454e-bbb7-bb793807cb67\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" Nov 01 16:28:33.826106 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.826076 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c176da5b-fd83-454e-bbb7-bb793807cb67-istio-data\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p\" (UID: \"c176da5b-fd83-454e-bbb7-bb793807cb67\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" Nov 01 16:28:33.826251 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.826225 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c176da5b-fd83-454e-bbb7-bb793807cb67-workload-certs\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p\" (UID: \"c176da5b-fd83-454e-bbb7-bb793807cb67\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" Nov 01 16:28:33.828403 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.828378 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c176da5b-fd83-454e-bbb7-bb793807cb67-istio-envoy\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p\" (UID: \"c176da5b-fd83-454e-bbb7-bb793807cb67\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" Nov 01 16:28:33.828532 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.828516 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c176da5b-fd83-454e-bbb7-bb793807cb67-istio-podinfo\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p\" (UID: \"c176da5b-fd83-454e-bbb7-bb793807cb67\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" Nov 01 16:28:33.838467 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.838442 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c176da5b-fd83-454e-bbb7-bb793807cb67-istio-token\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p\" (UID: \"c176da5b-fd83-454e-bbb7-bb793807cb67\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" Nov 01 16:28:33.838603 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.838559 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx8dc\" (UniqueName: \"kubernetes.io/projected/c176da5b-fd83-454e-bbb7-bb793807cb67-kube-api-access-fx8dc\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p\" (UID: \"c176da5b-fd83-454e-bbb7-bb793807cb67\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" Nov 01 16:28:33.966405 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:33.966325 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" Nov 01 16:28:34.198356 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:34.198318 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" event={"ID":"ca74811d-13fc-4a4c-bfd9-9e787392ffce","Type":"ContainerStarted","Data":"51923b7f22e4e169f57c4d9a5034691b313bee36af92af44172f4704b8d9361d"} Nov 01 16:28:34.198846 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:34.198401 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" event={"ID":"ca74811d-13fc-4a4c-bfd9-9e787392ffce","Type":"ContainerStarted","Data":"27fc6c87658b6433dd3dc5398c0fdd7e4d461e43162765efd466035a75d23b19"} Nov 01 16:28:34.218641 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:34.218482 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" podStartSLOduration=1.218462257 podStartE2EDuration="1.218462257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:28:34.216497865 +0000 UTC m=+2447.548235516" watchObservedRunningTime="2026-11-01 16:28:34.218462257 +0000 UTC m=+2447.550199911" Nov 01 16:28:34.294889 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:34.294846 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p"] Nov 01 16:28:34.304194 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:34.304159 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:28:34.304318 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:34.304229 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:28:34.304318 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:34.304266 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:28:34.368063 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:34.367969 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" Nov 01 16:28:34.369561 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:34.369533 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Nov 01 16:28:34.369680 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:34.369608 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" podUID="ca74811d-13fc-4a4c-bfd9-9e787392ffce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Nov 01 16:28:35.247041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:35.241627 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" event={"ID":"c176da5b-fd83-454e-bbb7-bb793807cb67","Type":"ContainerStarted","Data":"b001a90e25bfaed41da951cfe9477775e6ef5e21487f02cb3e23de8bd91803a1"} Nov 01 16:28:35.247041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:35.241697 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" event={"ID":"c176da5b-fd83-454e-bbb7-bb793807cb67","Type":"ContainerStarted","Data":"1158919e402f00a9784f37e3298d3e33680cbc1b2753946e4c83030b28046cb4"} Nov 01 16:28:35.303207 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:35.302433 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" podStartSLOduration=2.30240798 podStartE2EDuration="2.30240798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:28:35.265706102 +0000 UTC m=+2448.597443754" watchObservedRunningTime="2026-11-01 16:28:35.30240798 +0000 UTC m=+2448.634145631" Nov 01 16:28:35.368607 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:35.368566 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Nov 01 16:28:35.368797 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:35.368630 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" podUID="ca74811d-13fc-4a4c-bfd9-9e787392ffce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Nov 01 16:28:35.970043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:35.969964 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" Nov 01 16:28:35.971176 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:35.971137 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Nov 01 16:28:35.971230 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:35.971202 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" podUID="c176da5b-fd83-454e-bbb7-bb793807cb67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Nov 01 16:28:36.368304 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:36.368268 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Nov 01 16:28:36.368716 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:36.368318 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" podUID="ca74811d-13fc-4a4c-bfd9-9e787392ffce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Nov 01 16:28:36.970123 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:36.970070 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Nov 01 16:28:36.970315 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:36.970143 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" podUID="c176da5b-fd83-454e-bbb7-bb793807cb67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Nov 01 16:28:37.368441 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:37.368410 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Nov 01 16:28:37.368889 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:37.368460 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" podUID="ca74811d-13fc-4a4c-bfd9-9e787392ffce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Nov 01 16:28:37.461236 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:37.457045 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr"] Nov 01 16:28:37.464353 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:37.464332 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" Nov 01 16:28:37.519719 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:37.519631 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr"] Nov 01 16:28:37.550750 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:37.548811 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgqsk\" (UniqueName: \"kubernetes.io/projected/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-kube-api-access-fgqsk\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr\" (UID: \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" Nov 01 16:28:37.550750 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:37.548869 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-istiod-ca-cert\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr\" (UID: \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" Nov 01 16:28:37.550750 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:37.548915 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-istio-token\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr\" (UID: \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" Nov 01 16:28:37.550750 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:37.548995 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-istio-envoy\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr\" (UID: \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" Nov 01 16:28:37.550750 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:37.549026 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-workload-certs\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr\" (UID: \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" Nov 01 16:28:37.550750 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:37.549088 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-istio-data\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr\" (UID: \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" Nov 01 16:28:37.550750 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:37.549152 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-istio-podinfo\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr\" (UID: \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" Nov 01 16:28:37.550750 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:37.549183 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-credential-socket\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr\" (UID: \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" Nov 01 16:28:37.550750 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:37.549248 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-workload-socket\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr\" (UID: \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" Nov 01 16:28:37.653414 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:37.653306 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-istio-envoy\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr\" (UID: \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" Nov 01 16:28:37.653820 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:37.653796 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-workload-certs\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr\" (UID: \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" Nov 01 16:28:37.653925 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:37.653886 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-istio-data\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr\" (UID: \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" Nov 01 16:28:37.653990 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:37.653979 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-istio-podinfo\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr\" (UID: \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" Nov 01 16:28:37.654086 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:37.654047 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-credential-socket\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr\" (UID: \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" Nov 01 16:28:37.654161 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:37.654142 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-workload-socket\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr\" (UID: \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" Nov 01 16:28:37.654238 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:37.654221 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgqsk\" (UniqueName: \"kubernetes.io/projected/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-kube-api-access-fgqsk\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr\" (UID: \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" Nov 01 16:28:37.654295 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:37.654273 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-istiod-ca-cert\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr\" (UID: \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" Nov 01 16:28:37.654357 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:37.654337 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-istio-token\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr\" (UID: \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" Nov 01 16:28:37.654733 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:37.654687 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-credential-socket\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr\" (UID: \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" Nov 01 16:28:37.655000 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:37.654977 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-workload-socket\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr\" (UID: \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" Nov 01 16:28:37.655117 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:37.655032 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-workload-certs\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr\" (UID: \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" Nov 01 16:28:37.655336 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:37.655311 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-istio-data\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr\" (UID: \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" Nov 01 16:28:37.655867 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:37.655841 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-istiod-ca-cert\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr\" (UID: \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" Nov 01 16:28:37.656571 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:37.656545 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-istio-envoy\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr\" (UID: \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" Nov 01 16:28:37.658633 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:37.658604 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-istio-podinfo\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr\" (UID: \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" Nov 01 16:28:37.662844 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:37.662783 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-istio-token\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr\" (UID: \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" Nov 01 16:28:37.663814 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:37.663790 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgqsk\" (UniqueName: \"kubernetes.io/projected/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-kube-api-access-fgqsk\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr\" (UID: \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" Nov 01 16:28:37.853605 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:37.853526 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" Nov 01 16:28:37.994454 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:37.979108 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Nov 01 16:28:37.994454 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:37.992143 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" podUID="c176da5b-fd83-454e-bbb7-bb793807cb67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Nov 01 16:28:38.367835 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:38.367792 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Nov 01 16:28:38.368074 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:38.367853 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" podUID="ca74811d-13fc-4a4c-bfd9-9e787392ffce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Nov 01 16:28:38.440445 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:38.439299 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr"] Nov 01 16:28:38.444972 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:28:38.444933 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod635f1ba7_95b4_4c9c_9dc4_106eca4a54ad.slice/crio-371e74b5cecac32fbfeaad1ffc9017f4971abd095d9e0a82a18f26ad6a09d779 WatchSource:0}: Error finding container 371e74b5cecac32fbfeaad1ffc9017f4971abd095d9e0a82a18f26ad6a09d779: Status 404 returned error can't find the container with id 371e74b5cecac32fbfeaad1ffc9017f4971abd095d9e0a82a18f26ad6a09d779 Nov 01 16:28:38.447801 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:38.447746 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:28:38.448050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:38.448004 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:28:38.448212 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:38.448189 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:28:38.970427 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:38.970375 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Nov 01 16:28:38.970650 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:38.970449 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" podUID="c176da5b-fd83-454e-bbb7-bb793807cb67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Nov 01 16:28:39.368621 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:39.368581 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Nov 01 16:28:39.368792 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:39.368642 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" podUID="ca74811d-13fc-4a4c-bfd9-9e787392ffce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Nov 01 16:28:39.370922 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:39.370893 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" event={"ID":"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad","Type":"ContainerStarted","Data":"ccd7813e01b568a03a63370557d80a54b0ed2c782f593853761de6b9f65d0df8"} Nov 01 16:28:39.371095 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:39.370951 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" event={"ID":"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad","Type":"ContainerStarted","Data":"371e74b5cecac32fbfeaad1ffc9017f4971abd095d9e0a82a18f26ad6a09d779"} Nov 01 16:28:39.868041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:39.867034 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" Nov 01 16:28:39.868041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:39.867187 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body= Nov 01 16:28:39.868041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:39.867230 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" podUID="635f1ba7-95b4-4c9c-9dc4-106eca4a54ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" Nov 01 16:28:39.969715 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:39.969680 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Nov 01 16:28:39.969944 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:39.969735 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" podUID="c176da5b-fd83-454e-bbb7-bb793807cb67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Nov 01 16:28:40.395394 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:40.395356 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Nov 01 16:28:40.395687 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:40.395649 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" podUID="ca74811d-13fc-4a4c-bfd9-9e787392ffce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Nov 01 16:28:40.721182 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:40.717691 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" podStartSLOduration=3.717674453 podStartE2EDuration="3.717674453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:28:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:28:39.407368849 +0000 UTC m=+2452.739106512" watchObservedRunningTime="2026-11-01 16:28:40.717674453 +0000 UTC m=+2454.049412096" Nov 01 16:28:40.747119 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:40.747074 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c78897-kkrkx"] Nov 01 16:28:40.753941 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:40.753909 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c78897-kkrkx"] Nov 01 16:28:40.810697 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:40.810659 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04ee6238-4a5c-477d-b1b2-0442ecc9c037" path="/var/lib/kubelet/pods/04ee6238-4a5c-477d-b1b2-0442ecc9c037/volumes" Nov 01 16:28:40.856919 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:40.856463 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body= Nov 01 16:28:40.856919 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:40.856522 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" podUID="635f1ba7-95b4-4c9c-9dc4-106eca4a54ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" Nov 01 16:28:41.020793 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:41.020110 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Nov 01 16:28:41.020793 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:41.020182 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" podUID="c176da5b-fd83-454e-bbb7-bb793807cb67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Nov 01 16:28:41.854875 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:41.854836 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body= Nov 01 16:28:41.855263 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:41.855229 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" podUID="635f1ba7-95b4-4c9c-9dc4-106eca4a54ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" Nov 01 16:28:42.041475 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:42.041439 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" Nov 01 16:28:42.044115 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:42.044046 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" Nov 01 16:28:42.045272 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:42.045252 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" Nov 01 16:28:42.274469 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:42.274329 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" Nov 01 16:28:42.277391 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:42.275823 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" Nov 01 16:28:42.282340 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:42.282316 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" Nov 01 16:28:42.854194 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:42.854161 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body= Nov 01 16:28:42.854397 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:42.854232 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" podUID="635f1ba7-95b4-4c9c-9dc4-106eca4a54ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" Nov 01 16:28:43.854667 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:43.854630 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body= Nov 01 16:28:43.855076 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:43.854681 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" podUID="635f1ba7-95b4-4c9c-9dc4-106eca4a54ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" Nov 01 16:28:44.872051 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:44.870111 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body= Nov 01 16:28:44.872051 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:44.870184 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" podUID="635f1ba7-95b4-4c9c-9dc4-106eca4a54ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" Nov 01 16:28:45.854423 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:45.854385 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body= Nov 01 16:28:45.854584 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:45.854469 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" podUID="635f1ba7-95b4-4c9c-9dc4-106eca4a54ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" Nov 01 16:28:46.859342 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:46.859311 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" Nov 01 16:28:46.859754 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:46.859713 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" Nov 01 16:28:46.860420 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:46.860400 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" Nov 01 16:28:48.336303 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:48.336214 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b"] Nov 01 16:28:48.353037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:48.352990 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc"] Nov 01 16:28:48.373152 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:48.373118 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5"] Nov 01 16:28:48.373663 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:48.373626 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" podUID="ca74811d-13fc-4a4c-bfd9-9e787392ffce" containerName="istio-proxy" containerID="cri-o://51923b7f22e4e169f57c4d9a5034691b313bee36af92af44172f4704b8d9361d" gracePeriod=30 Nov 01 16:28:48.425048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:48.359278 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" podUID="1b180ef5-70b3-4bf2-901b-db280a232bde" containerName="istio-proxy" containerID="cri-o://82aaead6ae9f4be0f0297f718b4fa0f607322da5fe981cdcb7c21aac92d6e198" gracePeriod=30 Nov 01 16:28:48.673806 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:48.376417 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" podUID="77a6ec04-31e5-4d7e-a829-a4a0704de9db" containerName="istio-proxy" containerID="cri-o://fa3edfd3e8a5051dd7849547b19960f645038e9203313e3cc18a81bc2ce8f196" gracePeriod=30 Nov 01 16:28:48.748628 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:28:48.748589 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:28:48.993957 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:48.993870 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6b0e0f-4lpmp"] Nov 01 16:28:49.000918 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:49.000877 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6b0e0f-4lpmp" Nov 01 16:28:49.006887 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:49.006856 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6b0e0f-4lpmp"] Nov 01 16:28:49.008726 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:49.007783 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:28:49.008726 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:49.007887 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:28:49.179549 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:49.179507 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h9hp\" (UniqueName: \"kubernetes.io/projected/8b881d53-3472-4c0a-99a2-4b0f2b693a07-kube-api-access-4h9hp\") pod \"maas-api-revoke-keys-e2e-worker-w2-6b0e0f-4lpmp\" (UID: \"8b881d53-3472-4c0a-99a2-4b0f2b693a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6b0e0f-4lpmp" Nov 01 16:28:49.179753 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:49.179727 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b881d53-3472-4c0a-99a2-4b0f2b693a07-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-6b0e0f-4lpmp\" (UID: \"8b881d53-3472-4c0a-99a2-4b0f2b693a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6b0e0f-4lpmp" Nov 01 16:28:49.179887 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:49.179867 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b881d53-3472-4c0a-99a2-4b0f2b693a07-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-6b0e0f-4lpmp\" (UID: \"8b881d53-3472-4c0a-99a2-4b0f2b693a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6b0e0f-4lpmp" Nov 01 16:28:49.281323 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:49.281234 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h9hp\" (UniqueName: \"kubernetes.io/projected/8b881d53-3472-4c0a-99a2-4b0f2b693a07-kube-api-access-4h9hp\") pod \"maas-api-revoke-keys-e2e-worker-w2-6b0e0f-4lpmp\" (UID: \"8b881d53-3472-4c0a-99a2-4b0f2b693a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6b0e0f-4lpmp" Nov 01 16:28:49.281505 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:49.281393 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b881d53-3472-4c0a-99a2-4b0f2b693a07-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-6b0e0f-4lpmp\" (UID: \"8b881d53-3472-4c0a-99a2-4b0f2b693a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6b0e0f-4lpmp" Nov 01 16:28:49.281505 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:49.281477 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b881d53-3472-4c0a-99a2-4b0f2b693a07-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-6b0e0f-4lpmp\" (UID: \"8b881d53-3472-4c0a-99a2-4b0f2b693a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6b0e0f-4lpmp" Nov 01 16:28:49.282458 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:49.282425 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b881d53-3472-4c0a-99a2-4b0f2b693a07-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-6b0e0f-4lpmp\" (UID: \"8b881d53-3472-4c0a-99a2-4b0f2b693a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6b0e0f-4lpmp" Nov 01 16:28:49.289528 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:49.289501 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b881d53-3472-4c0a-99a2-4b0f2b693a07-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-6b0e0f-4lpmp\" (UID: \"8b881d53-3472-4c0a-99a2-4b0f2b693a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6b0e0f-4lpmp" Nov 01 16:28:49.289830 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:49.289805 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h9hp\" (UniqueName: \"kubernetes.io/projected/8b881d53-3472-4c0a-99a2-4b0f2b693a07-kube-api-access-4h9hp\") pod \"maas-api-revoke-keys-e2e-worker-w2-6b0e0f-4lpmp\" (UID: \"8b881d53-3472-4c0a-99a2-4b0f2b693a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6b0e0f-4lpmp" Nov 01 16:28:49.328779 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:49.328742 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6b0e0f-4lpmp" Nov 01 16:28:49.644318 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:28:49.644282 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b881d53_3472_4c0a_99a2_4b0f2b693a07.slice/crio-a3d681a33060f45915851ad935e1b19e8f3c8eb69593a927df128499b9598a29 WatchSource:0}: Error finding container a3d681a33060f45915851ad935e1b19e8f3c8eb69593a927df128499b9598a29: Status 404 returned error can't find the container with id a3d681a33060f45915851ad935e1b19e8f3c8eb69593a927df128499b9598a29 Nov 01 16:28:49.644716 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:49.644379 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6b0e0f-4lpmp"] Nov 01 16:28:49.801291 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:49.801249 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6b0e0f-4lpmp" event={"ID":"8b881d53-3472-4c0a-99a2-4b0f2b693a07","Type":"ContainerStarted","Data":"a3d681a33060f45915851ad935e1b19e8f3c8eb69593a927df128499b9598a29"} Nov 01 16:28:50.027758 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:50.027718 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-675cc9cb59-vmxp4"] Nov 01 16:28:50.037892 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:50.037831 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-675cc9cb59-vmxp4" podUID="2d97c2a5-f8a5-4b3f-aae1-2ba5295a7c90" containerName="payload-pre-processing" containerID="cri-o://31bdf252913009e0f8bcc87794e4ad72968c4837dad4f661857658a042ea41b7" gracePeriod=30 Nov 01 16:28:50.311178 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:50.281514 4363 scope.go:122] "RemoveContainer" containerID="46f0f59d149b20a47b982a1e8d12504e3ff18c8b7e990e8cff3bb909efe59d41" Nov 01 16:28:50.542475 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:50.540382 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-104b6e-sjwz4"] Nov 01 16:28:50.574810 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:50.568063 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-104b6e-sjwz4"] Nov 01 16:28:50.816978 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:50.816944 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60da33d7-f926-4e51-a32e-ea1125d706be" path="/var/lib/kubelet/pods/60da33d7-f926-4e51-a32e-ea1125d706be/volumes" Nov 01 16:28:50.817525 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:50.817496 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6b0e0f-4lpmp" event={"ID":"8b881d53-3472-4c0a-99a2-4b0f2b693a07","Type":"ContainerStarted","Data":"e977be825bddf57fb10c25ad1f6847116d0811c06518d8add2872ce6b8d784be"} Nov 01 16:28:50.818071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:50.818009 4363 generic.go:356] "Generic (PLEG): container finished" podID="2d97c2a5-f8a5-4b3f-aae1-2ba5295a7c90" containerID="31bdf252913009e0f8bcc87794e4ad72968c4837dad4f661857658a042ea41b7" exitCode=0 Nov 01 16:28:50.818174 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:50.818143 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-675cc9cb59-vmxp4" event={"ID":"2d97c2a5-f8a5-4b3f-aae1-2ba5295a7c90","Type":"ContainerDied","Data":"31bdf252913009e0f8bcc87794e4ad72968c4837dad4f661857658a042ea41b7"} Nov 01 16:28:50.836655 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:50.836600 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6b0e0f-4lpmp" podStartSLOduration=2.836586911 podStartE2EDuration="2.836586911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:28:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:28:50.831749209 +0000 UTC m=+2464.163486860" watchObservedRunningTime="2026-11-01 16:28:50.836586911 +0000 UTC m=+2464.168324621" Nov 01 16:28:50.845478 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:50.845459 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-675cc9cb59-vmxp4" Nov 01 16:28:51.021349 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:51.021316 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d97c2a5-f8a5-4b3f-aae1-2ba5295a7c90-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d97c2a5-f8a5-4b3f-aae1-2ba5295a7c90-plugins-config-volume\") pod \"2d97c2a5-f8a5-4b3f-aae1-2ba5295a7c90\" (UID: \"2d97c2a5-f8a5-4b3f-aae1-2ba5295a7c90\") " Nov 01 16:28:51.021541 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:51.021484 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d97c2a5-f8a5-4b3f-aae1-2ba5295a7c90-kube-api-access-flsp4\" (UniqueName: \"kubernetes.io/projected/2d97c2a5-f8a5-4b3f-aae1-2ba5295a7c90-kube-api-access-flsp4\") pod \"2d97c2a5-f8a5-4b3f-aae1-2ba5295a7c90\" (UID: \"2d97c2a5-f8a5-4b3f-aae1-2ba5295a7c90\") " Nov 01 16:28:51.021773 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:51.021735 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d97c2a5-f8a5-4b3f-aae1-2ba5295a7c90-plugins-config-volume" pod "2d97c2a5-f8a5-4b3f-aae1-2ba5295a7c90" (UID: "2d97c2a5-f8a5-4b3f-aae1-2ba5295a7c90"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:28:51.022242 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:51.022220 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d97c2a5-f8a5-4b3f-aae1-2ba5295a7c90-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:51.023798 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:51.023770 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d97c2a5-f8a5-4b3f-aae1-2ba5295a7c90-kube-api-access-flsp4" pod "2d97c2a5-f8a5-4b3f-aae1-2ba5295a7c90" (UID: "2d97c2a5-f8a5-4b3f-aae1-2ba5295a7c90"). InnerVolumeSpecName "kube-api-access-flsp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:51.124305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:51.124269 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flsp4\" (UniqueName: \"kubernetes.io/projected/2d97c2a5-f8a5-4b3f-aae1-2ba5295a7c90-kube-api-access-flsp4\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:51.828454 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:51.828425 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-675cc9cb59-vmxp4" Nov 01 16:28:51.828890 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:51.828459 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-675cc9cb59-vmxp4" event={"ID":"2d97c2a5-f8a5-4b3f-aae1-2ba5295a7c90","Type":"ContainerDied","Data":"fe287acd573f50d78e998021dff089f24e85189967403f698338466284a1ce7c"} Nov 01 16:28:51.828890 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:51.828526 4363 scope.go:122] "RemoveContainer" containerID="31bdf252913009e0f8bcc87794e4ad72968c4837dad4f661857658a042ea41b7" Nov 01 16:28:51.830577 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:51.830550 4363 generic.go:356] "Generic (PLEG): container finished" podID="8b881d53-3472-4c0a-99a2-4b0f2b693a07" containerID="e977be825bddf57fb10c25ad1f6847116d0811c06518d8add2872ce6b8d784be" exitCode=0 Nov 01 16:28:51.830720 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:51.830625 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6b0e0f-4lpmp" event={"ID":"8b881d53-3472-4c0a-99a2-4b0f2b693a07","Type":"ContainerDied","Data":"e977be825bddf57fb10c25ad1f6847116d0811c06518d8add2872ce6b8d784be"} Nov 01 16:28:51.862984 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:51.862961 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-675cc9cb59-vmxp4"] Nov 01 16:28:51.868622 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:51.868592 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-675cc9cb59-vmxp4"] Nov 01 16:28:52.898970 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:52.898926 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d97c2a5-f8a5-4b3f-aae1-2ba5295a7c90" path="/var/lib/kubelet/pods/2d97c2a5-f8a5-4b3f-aae1-2ba5295a7c90/volumes" Nov 01 16:28:53.222917 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:53.222886 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6b0e0f-4lpmp" Nov 01 16:28:53.314815 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:53.314780 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b881d53-3472-4c0a-99a2-4b0f2b693a07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b881d53-3472-4c0a-99a2-4b0f2b693a07-maas-api-openshift-ca\") pod \"8b881d53-3472-4c0a-99a2-4b0f2b693a07\" (UID: \"8b881d53-3472-4c0a-99a2-4b0f2b693a07\") " Nov 01 16:28:53.315048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:53.314833 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b881d53-3472-4c0a-99a2-4b0f2b693a07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b881d53-3472-4c0a-99a2-4b0f2b693a07-maas-api-xks-ca\") pod \"8b881d53-3472-4c0a-99a2-4b0f2b693a07\" (UID: \"8b881d53-3472-4c0a-99a2-4b0f2b693a07\") " Nov 01 16:28:53.315048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:53.314907 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b881d53-3472-4c0a-99a2-4b0f2b693a07-kube-api-access-4h9hp\" (UniqueName: \"kubernetes.io/projected/8b881d53-3472-4c0a-99a2-4b0f2b693a07-kube-api-access-4h9hp\") pod \"8b881d53-3472-4c0a-99a2-4b0f2b693a07\" (UID: \"8b881d53-3472-4c0a-99a2-4b0f2b693a07\") " Nov 01 16:28:53.316801 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:53.316773 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b881d53-3472-4c0a-99a2-4b0f2b693a07-maas-api-openshift-ca" pod "8b881d53-3472-4c0a-99a2-4b0f2b693a07" (UID: "8b881d53-3472-4c0a-99a2-4b0f2b693a07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:28:53.327330 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:53.327303 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b881d53-3472-4c0a-99a2-4b0f2b693a07-maas-api-xks-ca" pod "8b881d53-3472-4c0a-99a2-4b0f2b693a07" (UID: "8b881d53-3472-4c0a-99a2-4b0f2b693a07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:28:53.327896 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:53.327871 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b881d53-3472-4c0a-99a2-4b0f2b693a07-kube-api-access-4h9hp" pod "8b881d53-3472-4c0a-99a2-4b0f2b693a07" (UID: "8b881d53-3472-4c0a-99a2-4b0f2b693a07"). InnerVolumeSpecName "kube-api-access-4h9hp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:53.417954 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:53.417880 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b881d53-3472-4c0a-99a2-4b0f2b693a07-maas-api-openshift-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:53.417954 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:53.417915 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b881d53-3472-4c0a-99a2-4b0f2b693a07-maas-api-xks-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:53.417954 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:53.417930 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h9hp\" (UniqueName: \"kubernetes.io/projected/8b881d53-3472-4c0a-99a2-4b0f2b693a07-kube-api-access-4h9hp\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:53.697012 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:53.696929 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr"] Nov 01 16:28:53.697894 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:53.697870 4363 cpu_manager.go:414] "containerMap: removing container" podUID="2d97c2a5-f8a5-4b3f-aae1-2ba5295a7c90" containerName="payload-pre-processing" Nov 01 16:28:53.697894 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:53.697892 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d97c2a5-f8a5-4b3f-aae1-2ba5295a7c90" containerName="payload-pre-processing" Nov 01 16:28:53.698137 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:53.697950 4363 cpu_manager.go:414] "containerMap: removing container" podUID="8b881d53-3472-4c0a-99a2-4b0f2b693a07" containerName="revoke-keys" Nov 01 16:28:53.698137 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:53.697959 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b881d53-3472-4c0a-99a2-4b0f2b693a07" containerName="revoke-keys" Nov 01 16:28:53.698248 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:53.698218 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d97c2a5-f8a5-4b3f-aae1-2ba5295a7c90" containerName="payload-pre-processing" Nov 01 16:28:53.698302 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:53.698248 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b881d53-3472-4c0a-99a2-4b0f2b693a07" containerName="revoke-keys" Nov 01 16:28:53.705081 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:53.703819 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" Nov 01 16:28:53.719130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:53.719105 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-3fd73f-openshift-default-dockercfg-g4zgx\"" Nov 01 16:28:53.729108 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:53.723575 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr"] Nov 01 16:28:53.832060 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:53.831976 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k2sk\" (UniqueName: \"kubernetes.io/projected/c8ea572a-0005-426a-9017-de3331cbcbd7-kube-api-access-6k2sk\") pod \"e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr\" (UID: \"c8ea572a-0005-426a-9017-de3331cbcbd7\") " pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" Nov 01 16:28:53.832060 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:53.832030 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8ea572a-0005-426a-9017-de3331cbcbd7-istio-data\") pod \"e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr\" (UID: \"c8ea572a-0005-426a-9017-de3331cbcbd7\") " pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" Nov 01 16:28:53.832060 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:53.832061 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ea572a-0005-426a-9017-de3331cbcbd7-workload-socket\") pod \"e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr\" (UID: \"c8ea572a-0005-426a-9017-de3331cbcbd7\") " pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" Nov 01 16:28:53.832345 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:53.832112 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8ea572a-0005-426a-9017-de3331cbcbd7-istiod-ca-cert\") pod \"e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr\" (UID: \"c8ea572a-0005-426a-9017-de3331cbcbd7\") " pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" Nov 01 16:28:53.832345 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:53.832202 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8ea572a-0005-426a-9017-de3331cbcbd7-istio-token\") pod \"e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr\" (UID: \"c8ea572a-0005-426a-9017-de3331cbcbd7\") " pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" Nov 01 16:28:53.832345 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:53.832289 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8ea572a-0005-426a-9017-de3331cbcbd7-istio-envoy\") pod \"e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr\" (UID: \"c8ea572a-0005-426a-9017-de3331cbcbd7\") " pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" Nov 01 16:28:53.832345 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:53.832307 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8ea572a-0005-426a-9017-de3331cbcbd7-istio-podinfo\") pod \"e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr\" (UID: \"c8ea572a-0005-426a-9017-de3331cbcbd7\") " pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" Nov 01 16:28:53.832492 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:53.832357 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ea572a-0005-426a-9017-de3331cbcbd7-credential-socket\") pod \"e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr\" (UID: \"c8ea572a-0005-426a-9017-de3331cbcbd7\") " pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" Nov 01 16:28:53.832492 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:53.832381 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8ea572a-0005-426a-9017-de3331cbcbd7-workload-certs\") pod \"e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr\" (UID: \"c8ea572a-0005-426a-9017-de3331cbcbd7\") " pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" Nov 01 16:28:53.837691 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:53.836899 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" Nov 01 16:28:54.228422 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.220540 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca74811d-13fc-4a4c-bfd9-9e787392ffce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca74811d-13fc-4a4c-bfd9-9e787392ffce-workload-socket\") pod \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\" (UID: \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\") " Nov 01 16:28:54.228422 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.220657 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca74811d-13fc-4a4c-bfd9-9e787392ffce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca74811d-13fc-4a4c-bfd9-9e787392ffce-istiod-ca-cert\") pod \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\" (UID: \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\") " Nov 01 16:28:54.228422 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.220755 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca74811d-13fc-4a4c-bfd9-9e787392ffce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca74811d-13fc-4a4c-bfd9-9e787392ffce-istio-envoy\") pod \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\" (UID: \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\") " Nov 01 16:28:54.228422 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.220909 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca74811d-13fc-4a4c-bfd9-9e787392ffce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca74811d-13fc-4a4c-bfd9-9e787392ffce-credential-socket\") pod \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\" (UID: \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\") " Nov 01 16:28:54.228422 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.220985 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca74811d-13fc-4a4c-bfd9-9e787392ffce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca74811d-13fc-4a4c-bfd9-9e787392ffce-istio-podinfo\") pod \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\" (UID: \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\") " Nov 01 16:28:54.228422 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.221035 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca74811d-13fc-4a4c-bfd9-9e787392ffce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca74811d-13fc-4a4c-bfd9-9e787392ffce-istio-data\") pod \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\" (UID: \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\") " Nov 01 16:28:54.228422 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.221151 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca74811d-13fc-4a4c-bfd9-9e787392ffce-kube-api-access-gl88k\" (UniqueName: \"kubernetes.io/projected/ca74811d-13fc-4a4c-bfd9-9e787392ffce-kube-api-access-gl88k\") pod \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\" (UID: \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\") " Nov 01 16:28:54.228422 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.221229 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca74811d-13fc-4a4c-bfd9-9e787392ffce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca74811d-13fc-4a4c-bfd9-9e787392ffce-workload-certs\") pod \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\" (UID: \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\") " Nov 01 16:28:54.228422 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.221260 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca74811d-13fc-4a4c-bfd9-9e787392ffce-istio-token\" (UniqueName: \"kubernetes.io/projected/ca74811d-13fc-4a4c-bfd9-9e787392ffce-istio-token\") pod \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\" (UID: \"ca74811d-13fc-4a4c-bfd9-9e787392ffce\") " Nov 01 16:28:54.228422 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.221562 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k2sk\" (UniqueName: \"kubernetes.io/projected/c8ea572a-0005-426a-9017-de3331cbcbd7-kube-api-access-6k2sk\") pod \"e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr\" (UID: \"c8ea572a-0005-426a-9017-de3331cbcbd7\") " pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" Nov 01 16:28:54.228422 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.221600 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8ea572a-0005-426a-9017-de3331cbcbd7-istio-data\") pod \"e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr\" (UID: \"c8ea572a-0005-426a-9017-de3331cbcbd7\") " pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" Nov 01 16:28:54.228422 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.221642 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ea572a-0005-426a-9017-de3331cbcbd7-workload-socket\") pod \"e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr\" (UID: \"c8ea572a-0005-426a-9017-de3331cbcbd7\") " pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" Nov 01 16:28:54.228422 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.221706 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8ea572a-0005-426a-9017-de3331cbcbd7-istiod-ca-cert\") pod \"e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr\" (UID: \"c8ea572a-0005-426a-9017-de3331cbcbd7\") " pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" Nov 01 16:28:54.228422 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.221856 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8ea572a-0005-426a-9017-de3331cbcbd7-istio-token\") pod \"e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr\" (UID: \"c8ea572a-0005-426a-9017-de3331cbcbd7\") " pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" Nov 01 16:28:54.228422 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.221954 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8ea572a-0005-426a-9017-de3331cbcbd7-istio-envoy\") pod \"e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr\" (UID: \"c8ea572a-0005-426a-9017-de3331cbcbd7\") " pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" Nov 01 16:28:54.228422 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.221982 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8ea572a-0005-426a-9017-de3331cbcbd7-istio-podinfo\") pod \"e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr\" (UID: \"c8ea572a-0005-426a-9017-de3331cbcbd7\") " pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" Nov 01 16:28:54.228422 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.222067 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ea572a-0005-426a-9017-de3331cbcbd7-credential-socket\") pod \"e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr\" (UID: \"c8ea572a-0005-426a-9017-de3331cbcbd7\") " pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" Nov 01 16:28:54.230442 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.230230 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca74811d-13fc-4a4c-bfd9-9e787392ffce-workload-certs" pod "ca74811d-13fc-4a4c-bfd9-9e787392ffce" (UID: "ca74811d-13fc-4a4c-bfd9-9e787392ffce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:54.230765 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.230624 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca74811d-13fc-4a4c-bfd9-9e787392ffce-workload-socket" pod "ca74811d-13fc-4a4c-bfd9-9e787392ffce" (UID: "ca74811d-13fc-4a4c-bfd9-9e787392ffce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:54.233508 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.231264 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca74811d-13fc-4a4c-bfd9-9e787392ffce-istiod-ca-cert" pod "ca74811d-13fc-4a4c-bfd9-9e787392ffce" (UID: "ca74811d-13fc-4a4c-bfd9-9e787392ffce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:28:54.233508 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.231632 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca74811d-13fc-4a4c-bfd9-9e787392ffce-istio-data" pod "ca74811d-13fc-4a4c-bfd9-9e787392ffce" (UID: "ca74811d-13fc-4a4c-bfd9-9e787392ffce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:54.233508 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.232068 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca74811d-13fc-4a4c-bfd9-9e787392ffce-credential-socket" pod "ca74811d-13fc-4a4c-bfd9-9e787392ffce" (UID: "ca74811d-13fc-4a4c-bfd9-9e787392ffce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:54.238272 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.238227 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca74811d-13fc-4a4c-bfd9-9e787392ffce-istio-podinfo" pod "ca74811d-13fc-4a4c-bfd9-9e787392ffce" (UID: "ca74811d-13fc-4a4c-bfd9-9e787392ffce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:28:54.242467 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.242432 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ea572a-0005-426a-9017-de3331cbcbd7-workload-socket\") pod \"e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr\" (UID: \"c8ea572a-0005-426a-9017-de3331cbcbd7\") " pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" Nov 01 16:28:54.243766 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.242841 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8ea572a-0005-426a-9017-de3331cbcbd7-istiod-ca-cert\") pod \"e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr\" (UID: \"c8ea572a-0005-426a-9017-de3331cbcbd7\") " pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" Nov 01 16:28:54.243766 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.243429 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8ea572a-0005-426a-9017-de3331cbcbd7-istio-data\") pod \"e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr\" (UID: \"c8ea572a-0005-426a-9017-de3331cbcbd7\") " pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" Nov 01 16:28:54.243766 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.243729 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8ea572a-0005-426a-9017-de3331cbcbd7-workload-certs\") pod \"e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr\" (UID: \"c8ea572a-0005-426a-9017-de3331cbcbd7\") " pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" Nov 01 16:28:54.244132 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.244106 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca74811d-13fc-4a4c-bfd9-9e787392ffce-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:54.244211 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.244140 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca74811d-13fc-4a4c-bfd9-9e787392ffce-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:54.244211 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.244154 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca74811d-13fc-4a4c-bfd9-9e787392ffce-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:54.244211 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.244168 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca74811d-13fc-4a4c-bfd9-9e787392ffce-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:54.244211 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.244183 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca74811d-13fc-4a4c-bfd9-9e787392ffce-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:54.244211 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.244197 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca74811d-13fc-4a4c-bfd9-9e787392ffce-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:54.244506 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.244466 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8ea572a-0005-426a-9017-de3331cbcbd7-workload-certs\") pod \"e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr\" (UID: \"c8ea572a-0005-426a-9017-de3331cbcbd7\") " pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" Nov 01 16:28:54.244635 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.244605 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca74811d-13fc-4a4c-bfd9-9e787392ffce-istio-token" pod "ca74811d-13fc-4a4c-bfd9-9e787392ffce" (UID: "ca74811d-13fc-4a4c-bfd9-9e787392ffce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:54.250273 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.247247 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ea572a-0005-426a-9017-de3331cbcbd7-credential-socket\") pod \"e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr\" (UID: \"c8ea572a-0005-426a-9017-de3331cbcbd7\") " pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" Nov 01 16:28:54.251327 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.251203 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca74811d-13fc-4a4c-bfd9-9e787392ffce-istio-envoy" pod "ca74811d-13fc-4a4c-bfd9-9e787392ffce" (UID: "ca74811d-13fc-4a4c-bfd9-9e787392ffce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:54.255245 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.252587 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8ea572a-0005-426a-9017-de3331cbcbd7-istio-podinfo\") pod \"e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr\" (UID: \"c8ea572a-0005-426a-9017-de3331cbcbd7\") " pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" Nov 01 16:28:54.256980 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.256178 4363 generic.go:356] "Generic (PLEG): container finished" podID="ca74811d-13fc-4a4c-bfd9-9e787392ffce" containerID="51923b7f22e4e169f57c4d9a5034691b313bee36af92af44172f4704b8d9361d" exitCode=0 Nov 01 16:28:54.256980 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.256403 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" event={"ID":"ca74811d-13fc-4a4c-bfd9-9e787392ffce","Type":"ContainerDied","Data":"51923b7f22e4e169f57c4d9a5034691b313bee36af92af44172f4704b8d9361d"} Nov 01 16:28:54.256980 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.256459 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" event={"ID":"ca74811d-13fc-4a4c-bfd9-9e787392ffce","Type":"ContainerDied","Data":"27fc6c87658b6433dd3dc5398c0fdd7e4d461e43162765efd466035a75d23b19"} Nov 01 16:28:54.256980 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.256492 4363 scope.go:122] "RemoveContainer" containerID="51923b7f22e4e169f57c4d9a5034691b313bee36af92af44172f4704b8d9361d" Nov 01 16:28:54.256980 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.256669 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5" Nov 01 16:28:54.257749 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.257628 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8ea572a-0005-426a-9017-de3331cbcbd7-istio-token\") pod \"e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr\" (UID: \"c8ea572a-0005-426a-9017-de3331cbcbd7\") " pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" Nov 01 16:28:54.273567 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.272565 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8ea572a-0005-426a-9017-de3331cbcbd7-istio-envoy\") pod \"e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr\" (UID: \"c8ea572a-0005-426a-9017-de3331cbcbd7\") " pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" Nov 01 16:28:54.273567 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.272742 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca74811d-13fc-4a4c-bfd9-9e787392ffce-kube-api-access-gl88k" pod "ca74811d-13fc-4a4c-bfd9-9e787392ffce" (UID: "ca74811d-13fc-4a4c-bfd9-9e787392ffce"). InnerVolumeSpecName "kube-api-access-gl88k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:54.273567 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.273207 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k2sk\" (UniqueName: \"kubernetes.io/projected/c8ea572a-0005-426a-9017-de3331cbcbd7-kube-api-access-6k2sk\") pod \"e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr\" (UID: \"c8ea572a-0005-426a-9017-de3331cbcbd7\") " pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" Nov 01 16:28:54.276160 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.276136 4363 generic.go:356] "Generic (PLEG): container finished" podID="1b180ef5-70b3-4bf2-901b-db280a232bde" containerID="82aaead6ae9f4be0f0297f718b4fa0f607322da5fe981cdcb7c21aac92d6e198" exitCode=0 Nov 01 16:28:54.276280 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.276179 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" event={"ID":"1b180ef5-70b3-4bf2-901b-db280a232bde","Type":"ContainerDied","Data":"82aaead6ae9f4be0f0297f718b4fa0f607322da5fe981cdcb7c21aac92d6e198"} Nov 01 16:28:54.286887 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.286854 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6b0e0f-4lpmp" event={"ID":"8b881d53-3472-4c0a-99a2-4b0f2b693a07","Type":"ContainerDied","Data":"a3d681a33060f45915851ad935e1b19e8f3c8eb69593a927df128499b9598a29"} Nov 01 16:28:54.286970 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.286912 4363 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3d681a33060f45915851ad935e1b19e8f3c8eb69593a927df128499b9598a29" Nov 01 16:28:54.287067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.286987 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6b0e0f-4lpmp" Nov 01 16:28:54.294039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.292943 4363 generic.go:356] "Generic (PLEG): container finished" podID="77a6ec04-31e5-4d7e-a829-a4a0704de9db" containerID="fa3edfd3e8a5051dd7849547b19960f645038e9203313e3cc18a81bc2ce8f196" exitCode=0 Nov 01 16:28:54.294039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.293050 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" event={"ID":"77a6ec04-31e5-4d7e-a829-a4a0704de9db","Type":"ContainerDied","Data":"fa3edfd3e8a5051dd7849547b19960f645038e9203313e3cc18a81bc2ce8f196"} Nov 01 16:28:54.314575 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.314543 4363 scope.go:122] "RemoveContainer" containerID="51923b7f22e4e169f57c4d9a5034691b313bee36af92af44172f4704b8d9361d" Nov 01 16:28:54.315460 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:28:54.315429 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51923b7f22e4e169f57c4d9a5034691b313bee36af92af44172f4704b8d9361d\": container with ID starting with 51923b7f22e4e169f57c4d9a5034691b313bee36af92af44172f4704b8d9361d not found: ID does not exist" containerID="51923b7f22e4e169f57c4d9a5034691b313bee36af92af44172f4704b8d9361d" Nov 01 16:28:54.315577 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.315474 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51923b7f22e4e169f57c4d9a5034691b313bee36af92af44172f4704b8d9361d"} err="failed to get container status \"51923b7f22e4e169f57c4d9a5034691b313bee36af92af44172f4704b8d9361d\": rpc error: code = NotFound desc = could not find container \"51923b7f22e4e169f57c4d9a5034691b313bee36af92af44172f4704b8d9361d\": container with ID starting with 51923b7f22e4e169f57c4d9a5034691b313bee36af92af44172f4704b8d9361d not found: ID does not exist" Nov 01 16:28:54.315577 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.315500 4363 scope.go:122] "RemoveContainer" containerID="63409c055dbbf5803b5aac62cd48411547732fa5158dbd6d67f1edcb277d5d0a" Nov 01 16:28:54.346336 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.346185 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca74811d-13fc-4a4c-bfd9-9e787392ffce-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:54.346336 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.346216 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl88k\" (UniqueName: \"kubernetes.io/projected/ca74811d-13fc-4a4c-bfd9-9e787392ffce-kube-api-access-gl88k\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:54.346336 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.346232 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca74811d-13fc-4a4c-bfd9-9e787392ffce-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:54.353309 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.353281 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" Nov 01 16:28:54.375687 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.375659 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" Nov 01 16:28:54.458620 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.447279 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77a6ec04-31e5-4d7e-a829-a4a0704de9db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77a6ec04-31e5-4d7e-a829-a4a0704de9db-istio-data\") pod \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\" (UID: \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\") " Nov 01 16:28:54.458620 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.447550 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77a6ec04-31e5-4d7e-a829-a4a0704de9db-istio-data" pod "77a6ec04-31e5-4d7e-a829-a4a0704de9db" (UID: "77a6ec04-31e5-4d7e-a829-a4a0704de9db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:54.459792 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.459761 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77a6ec04-31e5-4d7e-a829-a4a0704de9db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77a6ec04-31e5-4d7e-a829-a4a0704de9db-workload-socket\") pod \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\" (UID: \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\") " Nov 01 16:28:54.468096 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.468063 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77a6ec04-31e5-4d7e-a829-a4a0704de9db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77a6ec04-31e5-4d7e-a829-a4a0704de9db-istio-podinfo\") pod \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\" (UID: \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\") " Nov 01 16:28:54.474654 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.473468 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77a6ec04-31e5-4d7e-a829-a4a0704de9db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77a6ec04-31e5-4d7e-a829-a4a0704de9db-credential-socket\") pod \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\" (UID: \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\") " Nov 01 16:28:54.476303 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.476088 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77a6ec04-31e5-4d7e-a829-a4a0704de9db-istio-token\" (UniqueName: \"kubernetes.io/projected/77a6ec04-31e5-4d7e-a829-a4a0704de9db-istio-token\") pod \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\" (UID: \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\") " Nov 01 16:28:54.476563 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.476532 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77a6ec04-31e5-4d7e-a829-a4a0704de9db-kube-api-access-twtkw\" (UniqueName: \"kubernetes.io/projected/77a6ec04-31e5-4d7e-a829-a4a0704de9db-kube-api-access-twtkw\") pod \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\" (UID: \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\") " Nov 01 16:28:54.476829 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.476781 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77a6ec04-31e5-4d7e-a829-a4a0704de9db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77a6ec04-31e5-4d7e-a829-a4a0704de9db-istio-envoy\") pod \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\" (UID: \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\") " Nov 01 16:28:54.476973 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.476958 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77a6ec04-31e5-4d7e-a829-a4a0704de9db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77a6ec04-31e5-4d7e-a829-a4a0704de9db-istiod-ca-cert\") pod \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\" (UID: \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\") " Nov 01 16:28:54.477184 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.477169 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77a6ec04-31e5-4d7e-a829-a4a0704de9db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77a6ec04-31e5-4d7e-a829-a4a0704de9db-workload-certs\") pod \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\" (UID: \"77a6ec04-31e5-4d7e-a829-a4a0704de9db\") " Nov 01 16:28:54.499383 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.478317 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77a6ec04-31e5-4d7e-a829-a4a0704de9db-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:54.499639 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.480126 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77a6ec04-31e5-4d7e-a829-a4a0704de9db-workload-socket" pod "77a6ec04-31e5-4d7e-a829-a4a0704de9db" (UID: "77a6ec04-31e5-4d7e-a829-a4a0704de9db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:54.500211 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.500172 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77a6ec04-31e5-4d7e-a829-a4a0704de9db-workload-certs" pod "77a6ec04-31e5-4d7e-a829-a4a0704de9db" (UID: "77a6ec04-31e5-4d7e-a829-a4a0704de9db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:54.500628 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.500584 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77a6ec04-31e5-4d7e-a829-a4a0704de9db-credential-socket" pod "77a6ec04-31e5-4d7e-a829-a4a0704de9db" (UID: "77a6ec04-31e5-4d7e-a829-a4a0704de9db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:54.503596 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.503436 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77a6ec04-31e5-4d7e-a829-a4a0704de9db-istio-envoy" pod "77a6ec04-31e5-4d7e-a829-a4a0704de9db" (UID: "77a6ec04-31e5-4d7e-a829-a4a0704de9db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:54.508691 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.508652 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77a6ec04-31e5-4d7e-a829-a4a0704de9db-istio-token" pod "77a6ec04-31e5-4d7e-a829-a4a0704de9db" (UID: "77a6ec04-31e5-4d7e-a829-a4a0704de9db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:54.527604 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.527192 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77a6ec04-31e5-4d7e-a829-a4a0704de9db-kube-api-access-twtkw" pod "77a6ec04-31e5-4d7e-a829-a4a0704de9db" (UID: "77a6ec04-31e5-4d7e-a829-a4a0704de9db"). InnerVolumeSpecName "kube-api-access-twtkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:54.543072 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.537420 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" Nov 01 16:28:54.544254 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.544221 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77a6ec04-31e5-4d7e-a829-a4a0704de9db-istiod-ca-cert" pod "77a6ec04-31e5-4d7e-a829-a4a0704de9db" (UID: "77a6ec04-31e5-4d7e-a829-a4a0704de9db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:28:54.563921 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.550621 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77a6ec04-31e5-4d7e-a829-a4a0704de9db-istio-podinfo" pod "77a6ec04-31e5-4d7e-a829-a4a0704de9db" (UID: "77a6ec04-31e5-4d7e-a829-a4a0704de9db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:28:54.597156 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.597117 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5"] Nov 01 16:28:54.598894 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.598867 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-5rmp5"] Nov 01 16:28:54.600859 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.600829 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b180ef5-70b3-4bf2-901b-db280a232bde-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b180ef5-70b3-4bf2-901b-db280a232bde-istio-envoy\") pod \"1b180ef5-70b3-4bf2-901b-db280a232bde\" (UID: \"1b180ef5-70b3-4bf2-901b-db280a232bde\") " Nov 01 16:28:54.600996 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.600918 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b180ef5-70b3-4bf2-901b-db280a232bde-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b180ef5-70b3-4bf2-901b-db280a232bde-istio-data\") pod \"1b180ef5-70b3-4bf2-901b-db280a232bde\" (UID: \"1b180ef5-70b3-4bf2-901b-db280a232bde\") " Nov 01 16:28:54.600996 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.600966 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b180ef5-70b3-4bf2-901b-db280a232bde-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b180ef5-70b3-4bf2-901b-db280a232bde-workload-certs\") pod \"1b180ef5-70b3-4bf2-901b-db280a232bde\" (UID: \"1b180ef5-70b3-4bf2-901b-db280a232bde\") " Nov 01 16:28:54.601140 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.601104 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b180ef5-70b3-4bf2-901b-db280a232bde-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b180ef5-70b3-4bf2-901b-db280a232bde-credential-socket\") pod \"1b180ef5-70b3-4bf2-901b-db280a232bde\" (UID: \"1b180ef5-70b3-4bf2-901b-db280a232bde\") " Nov 01 16:28:54.601247 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.601225 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b180ef5-70b3-4bf2-901b-db280a232bde-kube-api-access-vkjlh\" (UniqueName: \"kubernetes.io/projected/1b180ef5-70b3-4bf2-901b-db280a232bde-kube-api-access-vkjlh\") pod \"1b180ef5-70b3-4bf2-901b-db280a232bde\" (UID: \"1b180ef5-70b3-4bf2-901b-db280a232bde\") " Nov 01 16:28:54.601377 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.601308 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b180ef5-70b3-4bf2-901b-db280a232bde-istio-data" pod "1b180ef5-70b3-4bf2-901b-db280a232bde" (UID: "1b180ef5-70b3-4bf2-901b-db280a232bde"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:54.601377 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.601350 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b180ef5-70b3-4bf2-901b-db280a232bde-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b180ef5-70b3-4bf2-901b-db280a232bde-workload-socket\") pod \"1b180ef5-70b3-4bf2-901b-db280a232bde\" (UID: \"1b180ef5-70b3-4bf2-901b-db280a232bde\") " Nov 01 16:28:54.601564 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.601407 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b180ef5-70b3-4bf2-901b-db280a232bde-istio-token\" (UniqueName: \"kubernetes.io/projected/1b180ef5-70b3-4bf2-901b-db280a232bde-istio-token\") pod \"1b180ef5-70b3-4bf2-901b-db280a232bde\" (UID: \"1b180ef5-70b3-4bf2-901b-db280a232bde\") " Nov 01 16:28:54.601564 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.601478 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b180ef5-70b3-4bf2-901b-db280a232bde-workload-certs" pod "1b180ef5-70b3-4bf2-901b-db280a232bde" (UID: "1b180ef5-70b3-4bf2-901b-db280a232bde"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:54.601564 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.601508 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b180ef5-70b3-4bf2-901b-db280a232bde-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b180ef5-70b3-4bf2-901b-db280a232bde-istiod-ca-cert\") pod \"1b180ef5-70b3-4bf2-901b-db280a232bde\" (UID: \"1b180ef5-70b3-4bf2-901b-db280a232bde\") " Nov 01 16:28:54.601564 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.601553 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b180ef5-70b3-4bf2-901b-db280a232bde-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b180ef5-70b3-4bf2-901b-db280a232bde-istio-podinfo\") pod \"1b180ef5-70b3-4bf2-901b-db280a232bde\" (UID: \"1b180ef5-70b3-4bf2-901b-db280a232bde\") " Nov 01 16:28:54.602371 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.602337 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77a6ec04-31e5-4d7e-a829-a4a0704de9db-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:54.602371 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.602364 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77a6ec04-31e5-4d7e-a829-a4a0704de9db-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:54.602597 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.602380 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77a6ec04-31e5-4d7e-a829-a4a0704de9db-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:54.602597 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.602394 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b180ef5-70b3-4bf2-901b-db280a232bde-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:54.602597 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.602407 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b180ef5-70b3-4bf2-901b-db280a232bde-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:54.602597 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.602422 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77a6ec04-31e5-4d7e-a829-a4a0704de9db-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:54.602597 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.602435 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77a6ec04-31e5-4d7e-a829-a4a0704de9db-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:54.602597 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.602451 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77a6ec04-31e5-4d7e-a829-a4a0704de9db-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:54.602597 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.602465 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77a6ec04-31e5-4d7e-a829-a4a0704de9db-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:54.602597 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.602479 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twtkw\" (UniqueName: \"kubernetes.io/projected/77a6ec04-31e5-4d7e-a829-a4a0704de9db-kube-api-access-twtkw\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:54.604106 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.604084 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b180ef5-70b3-4bf2-901b-db280a232bde-istio-envoy" pod "1b180ef5-70b3-4bf2-901b-db280a232bde" (UID: "1b180ef5-70b3-4bf2-901b-db280a232bde"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:54.604445 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.604422 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b180ef5-70b3-4bf2-901b-db280a232bde-kube-api-access-vkjlh" pod "1b180ef5-70b3-4bf2-901b-db280a232bde" (UID: "1b180ef5-70b3-4bf2-901b-db280a232bde"). InnerVolumeSpecName "kube-api-access-vkjlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:54.604722 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.604698 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b180ef5-70b3-4bf2-901b-db280a232bde-credential-socket" pod "1b180ef5-70b3-4bf2-901b-db280a232bde" (UID: "1b180ef5-70b3-4bf2-901b-db280a232bde"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:54.604922 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.604899 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b180ef5-70b3-4bf2-901b-db280a232bde-workload-socket" pod "1b180ef5-70b3-4bf2-901b-db280a232bde" (UID: "1b180ef5-70b3-4bf2-901b-db280a232bde"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:54.605228 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.605206 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b180ef5-70b3-4bf2-901b-db280a232bde-istiod-ca-cert" pod "1b180ef5-70b3-4bf2-901b-db280a232bde" (UID: "1b180ef5-70b3-4bf2-901b-db280a232bde"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:28:54.606208 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.606188 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b180ef5-70b3-4bf2-901b-db280a232bde-istio-token" pod "1b180ef5-70b3-4bf2-901b-db280a232bde" (UID: "1b180ef5-70b3-4bf2-901b-db280a232bde"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:54.607984 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.607959 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b180ef5-70b3-4bf2-901b-db280a232bde-istio-podinfo" pod "1b180ef5-70b3-4bf2-901b-db280a232bde" (UID: "1b180ef5-70b3-4bf2-901b-db280a232bde"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:28:54.704565 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.704529 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b180ef5-70b3-4bf2-901b-db280a232bde-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:54.704565 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.704564 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkjlh\" (UniqueName: \"kubernetes.io/projected/1b180ef5-70b3-4bf2-901b-db280a232bde-kube-api-access-vkjlh\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:54.704849 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.704578 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b180ef5-70b3-4bf2-901b-db280a232bde-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:54.704849 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.704591 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b180ef5-70b3-4bf2-901b-db280a232bde-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:54.704849 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.704604 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b180ef5-70b3-4bf2-901b-db280a232bde-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:54.704849 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.704623 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b180ef5-70b3-4bf2-901b-db280a232bde-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:54.704849 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.704634 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b180ef5-70b3-4bf2-901b-db280a232bde-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:54.712623 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.712262 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr"] Nov 01 16:28:54.717302 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:28:54.717277 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8ea572a_0005_426a_9017_de3331cbcbd7.slice/crio-50fd26e93f89366e49b99a0680857a333869dba52669a2787bff7d34bbd0b062 WatchSource:0}: Error finding container 50fd26e93f89366e49b99a0680857a333869dba52669a2787bff7d34bbd0b062: Status 404 returned error can't find the container with id 50fd26e93f89366e49b99a0680857a333869dba52669a2787bff7d34bbd0b062 Nov 01 16:28:54.721067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.721028 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:28:54.721157 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.721122 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:28:54.721202 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.721162 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:28:54.804463 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:54.804431 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca74811d-13fc-4a4c-bfd9-9e787392ffce" path="/var/lib/kubelet/pods/ca74811d-13fc-4a4c-bfd9-9e787392ffce/volumes" Nov 01 16:28:55.306661 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:55.306629 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" Nov 01 16:28:55.306661 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:55.306651 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b" event={"ID":"1b180ef5-70b3-4bf2-901b-db280a232bde","Type":"ContainerDied","Data":"c44c0544396c5b5a6d6619a9e8b5923d66ff56df09eebce10cd1c52903e97a79"} Nov 01 16:28:55.307312 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:55.306707 4363 scope.go:122] "RemoveContainer" containerID="82aaead6ae9f4be0f0297f718b4fa0f607322da5fe981cdcb7c21aac92d6e198" Nov 01 16:28:55.309304 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:55.309042 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" event={"ID":"c8ea572a-0005-426a-9017-de3331cbcbd7","Type":"ContainerStarted","Data":"4f2463064534c9e5b955d6af551a388d04d3253128b18b6059ce214a47d6f1e6"} Nov 01 16:28:55.309304 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:55.309122 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" event={"ID":"c8ea572a-0005-426a-9017-de3331cbcbd7","Type":"ContainerStarted","Data":"50fd26e93f89366e49b99a0680857a333869dba52669a2787bff7d34bbd0b062"} Nov 01 16:28:55.311801 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:55.311773 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" event={"ID":"77a6ec04-31e5-4d7e-a829-a4a0704de9db","Type":"ContainerDied","Data":"e25152270718b002bd773a8a47ee01071e3e61ace84809ad611edd0032db2dd5"} Nov 01 16:28:55.311999 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:55.311974 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc" Nov 01 16:28:55.322006 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:55.321986 4363 scope.go:122] "RemoveContainer" containerID="fa3edfd3e8a5051dd7849547b19960f645038e9203313e3cc18a81bc2ce8f196" Nov 01 16:28:55.335773 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:55.335733 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" podStartSLOduration=2.335720412 podStartE2EDuration="2.335720412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:28:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:28:55.329493566 +0000 UTC m=+2468.661231229" watchObservedRunningTime="2026-11-01 16:28:55.335720412 +0000 UTC m=+2468.667458062" Nov 01 16:28:55.348120 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:55.348093 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b"] Nov 01 16:28:55.352007 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:55.351989 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-kx72b"] Nov 01 16:28:55.353906 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:55.353887 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" Nov 01 16:28:55.360462 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:55.360439 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" Nov 01 16:28:55.361270 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:55.361250 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc"] Nov 01 16:28:55.365403 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:55.365382 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-cdgzc"] Nov 01 16:28:56.339453 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:56.339425 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" Nov 01 16:28:56.340446 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:56.340423 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" Nov 01 16:28:56.802500 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:56.802465 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b180ef5-70b3-4bf2-901b-db280a232bde" path="/var/lib/kubelet/pods/1b180ef5-70b3-4bf2-901b-db280a232bde/volumes" Nov 01 16:28:56.803185 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:56.803159 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77a6ec04-31e5-4d7e-a829-a4a0704de9db" path="/var/lib/kubelet/pods/77a6ec04-31e5-4d7e-a829-a4a0704de9db/volumes" Nov 01 16:28:57.764926 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:57.764891 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc"] Nov 01 16:28:57.765440 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:28:57.765130 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" podUID="be90dd72-5635-4115-9826-26aaa46f70dc" containerName="istio-proxy" containerID="cri-o://d1453bb6d55de12f27f4019e8f62d38775bd38d3192056af021f06cb55ab74a3" gracePeriod=30 Nov 01 16:29:03.141191 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:03.141168 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" Nov 01 16:29:03.199609 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:03.199456 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be90dd72-5635-4115-9826-26aaa46f70dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be90dd72-5635-4115-9826-26aaa46f70dc-istio-data\") pod \"be90dd72-5635-4115-9826-26aaa46f70dc\" (UID: \"be90dd72-5635-4115-9826-26aaa46f70dc\") " Nov 01 16:29:03.199609 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:03.199542 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be90dd72-5635-4115-9826-26aaa46f70dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be90dd72-5635-4115-9826-26aaa46f70dc-istio-envoy\") pod \"be90dd72-5635-4115-9826-26aaa46f70dc\" (UID: \"be90dd72-5635-4115-9826-26aaa46f70dc\") " Nov 01 16:29:03.199920 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:03.199845 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be90dd72-5635-4115-9826-26aaa46f70dc-istio-data" pod "be90dd72-5635-4115-9826-26aaa46f70dc" (UID: "be90dd72-5635-4115-9826-26aaa46f70dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:03.200445 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:03.200418 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be90dd72-5635-4115-9826-26aaa46f70dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be90dd72-5635-4115-9826-26aaa46f70dc-workload-certs\") pod \"be90dd72-5635-4115-9826-26aaa46f70dc\" (UID: \"be90dd72-5635-4115-9826-26aaa46f70dc\") " Nov 01 16:29:03.200534 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:03.200460 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be90dd72-5635-4115-9826-26aaa46f70dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be90dd72-5635-4115-9826-26aaa46f70dc-credential-socket\") pod \"be90dd72-5635-4115-9826-26aaa46f70dc\" (UID: \"be90dd72-5635-4115-9826-26aaa46f70dc\") " Nov 01 16:29:03.200534 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:03.200511 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be90dd72-5635-4115-9826-26aaa46f70dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be90dd72-5635-4115-9826-26aaa46f70dc-istiod-ca-cert\") pod \"be90dd72-5635-4115-9826-26aaa46f70dc\" (UID: \"be90dd72-5635-4115-9826-26aaa46f70dc\") " Nov 01 16:29:03.200646 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:03.200545 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be90dd72-5635-4115-9826-26aaa46f70dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be90dd72-5635-4115-9826-26aaa46f70dc-workload-socket\") pod \"be90dd72-5635-4115-9826-26aaa46f70dc\" (UID: \"be90dd72-5635-4115-9826-26aaa46f70dc\") " Nov 01 16:29:03.200735 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:03.200720 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be90dd72-5635-4115-9826-26aaa46f70dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be90dd72-5635-4115-9826-26aaa46f70dc-istio-podinfo\") pod \"be90dd72-5635-4115-9826-26aaa46f70dc\" (UID: \"be90dd72-5635-4115-9826-26aaa46f70dc\") " Nov 01 16:29:03.200775 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:03.200763 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be90dd72-5635-4115-9826-26aaa46f70dc-kube-api-access-d5pnv\" (UniqueName: \"kubernetes.io/projected/be90dd72-5635-4115-9826-26aaa46f70dc-kube-api-access-d5pnv\") pod \"be90dd72-5635-4115-9826-26aaa46f70dc\" (UID: \"be90dd72-5635-4115-9826-26aaa46f70dc\") " Nov 01 16:29:03.200911 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:03.200897 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be90dd72-5635-4115-9826-26aaa46f70dc-istio-token\" (UniqueName: \"kubernetes.io/projected/be90dd72-5635-4115-9826-26aaa46f70dc-istio-token\") pod \"be90dd72-5635-4115-9826-26aaa46f70dc\" (UID: \"be90dd72-5635-4115-9826-26aaa46f70dc\") " Nov 01 16:29:03.201828 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:03.201805 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be90dd72-5635-4115-9826-26aaa46f70dc-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:03.202449 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:03.202418 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be90dd72-5635-4115-9826-26aaa46f70dc-istiod-ca-cert" pod "be90dd72-5635-4115-9826-26aaa46f70dc" (UID: "be90dd72-5635-4115-9826-26aaa46f70dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:29:03.202702 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:03.202676 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be90dd72-5635-4115-9826-26aaa46f70dc-workload-certs" pod "be90dd72-5635-4115-9826-26aaa46f70dc" (UID: "be90dd72-5635-4115-9826-26aaa46f70dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:03.202891 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:03.202870 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be90dd72-5635-4115-9826-26aaa46f70dc-credential-socket" pod "be90dd72-5635-4115-9826-26aaa46f70dc" (UID: "be90dd72-5635-4115-9826-26aaa46f70dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:03.203945 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:03.203230 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be90dd72-5635-4115-9826-26aaa46f70dc-istio-envoy" pod "be90dd72-5635-4115-9826-26aaa46f70dc" (UID: "be90dd72-5635-4115-9826-26aaa46f70dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:03.203945 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:03.203589 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be90dd72-5635-4115-9826-26aaa46f70dc-workload-socket" pod "be90dd72-5635-4115-9826-26aaa46f70dc" (UID: "be90dd72-5635-4115-9826-26aaa46f70dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:03.204549 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:03.204525 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be90dd72-5635-4115-9826-26aaa46f70dc-istio-token" pod "be90dd72-5635-4115-9826-26aaa46f70dc" (UID: "be90dd72-5635-4115-9826-26aaa46f70dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:29:03.205215 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:03.205190 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be90dd72-5635-4115-9826-26aaa46f70dc-istio-podinfo" pod "be90dd72-5635-4115-9826-26aaa46f70dc" (UID: "be90dd72-5635-4115-9826-26aaa46f70dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:29:03.206173 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:03.206149 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be90dd72-5635-4115-9826-26aaa46f70dc-kube-api-access-d5pnv" pod "be90dd72-5635-4115-9826-26aaa46f70dc" (UID: "be90dd72-5635-4115-9826-26aaa46f70dc"). InnerVolumeSpecName "kube-api-access-d5pnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:29:03.304057 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:03.303988 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be90dd72-5635-4115-9826-26aaa46f70dc-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:03.304057 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:03.304056 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5pnv\" (UniqueName: \"kubernetes.io/projected/be90dd72-5635-4115-9826-26aaa46f70dc-kube-api-access-d5pnv\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:03.304316 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:03.304072 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be90dd72-5635-4115-9826-26aaa46f70dc-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:03.304316 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:03.304086 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be90dd72-5635-4115-9826-26aaa46f70dc-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:03.304316 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:03.304101 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be90dd72-5635-4115-9826-26aaa46f70dc-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:03.304316 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:03.304115 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be90dd72-5635-4115-9826-26aaa46f70dc-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:03.304316 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:03.304130 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be90dd72-5635-4115-9826-26aaa46f70dc-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:03.304316 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:03.304153 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be90dd72-5635-4115-9826-26aaa46f70dc-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:03.425697 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:03.425618 4363 generic.go:356] "Generic (PLEG): container finished" podID="be90dd72-5635-4115-9826-26aaa46f70dc" containerID="d1453bb6d55de12f27f4019e8f62d38775bd38d3192056af021f06cb55ab74a3" exitCode=0 Nov 01 16:29:03.425697 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:03.425665 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" event={"ID":"be90dd72-5635-4115-9826-26aaa46f70dc","Type":"ContainerDied","Data":"d1453bb6d55de12f27f4019e8f62d38775bd38d3192056af021f06cb55ab74a3"} Nov 01 16:29:03.425882 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:03.425691 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" Nov 01 16:29:03.425882 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:03.425728 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc" event={"ID":"be90dd72-5635-4115-9826-26aaa46f70dc","Type":"ContainerDied","Data":"cf34d5b0ecd84e5fb837332eefe63bc162e607386077d2434e4477c11feeb3b0"} Nov 01 16:29:03.425882 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:03.425755 4363 scope.go:122] "RemoveContainer" containerID="d1453bb6d55de12f27f4019e8f62d38775bd38d3192056af021f06cb55ab74a3" Nov 01 16:29:03.439445 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:03.438325 4363 scope.go:122] "RemoveContainer" containerID="61ea40fc745b3bb1015069215985f45711621cb794c7c73809efd6112fc793f0" Nov 01 16:29:03.449315 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:03.449297 4363 scope.go:122] "RemoveContainer" containerID="d1453bb6d55de12f27f4019e8f62d38775bd38d3192056af021f06cb55ab74a3" Nov 01 16:29:03.449634 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:29:03.449602 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1453bb6d55de12f27f4019e8f62d38775bd38d3192056af021f06cb55ab74a3\": container with ID starting with d1453bb6d55de12f27f4019e8f62d38775bd38d3192056af021f06cb55ab74a3 not found: ID does not exist" containerID="d1453bb6d55de12f27f4019e8f62d38775bd38d3192056af021f06cb55ab74a3" Nov 01 16:29:03.449746 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:03.449642 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1453bb6d55de12f27f4019e8f62d38775bd38d3192056af021f06cb55ab74a3"} err="failed to get container status \"d1453bb6d55de12f27f4019e8f62d38775bd38d3192056af021f06cb55ab74a3\": rpc error: code = NotFound desc = could not find container \"d1453bb6d55de12f27f4019e8f62d38775bd38d3192056af021f06cb55ab74a3\": container with ID starting with d1453bb6d55de12f27f4019e8f62d38775bd38d3192056af021f06cb55ab74a3 not found: ID does not exist" Nov 01 16:29:03.449746 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:03.449660 4363 scope.go:122] "RemoveContainer" containerID="61ea40fc745b3bb1015069215985f45711621cb794c7c73809efd6112fc793f0" Nov 01 16:29:03.449973 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:29:03.449947 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61ea40fc745b3bb1015069215985f45711621cb794c7c73809efd6112fc793f0\": container with ID starting with 61ea40fc745b3bb1015069215985f45711621cb794c7c73809efd6112fc793f0 not found: ID does not exist" containerID="61ea40fc745b3bb1015069215985f45711621cb794c7c73809efd6112fc793f0" Nov 01 16:29:03.450056 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:03.449984 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61ea40fc745b3bb1015069215985f45711621cb794c7c73809efd6112fc793f0"} err="failed to get container status \"61ea40fc745b3bb1015069215985f45711621cb794c7c73809efd6112fc793f0\": rpc error: code = NotFound desc = could not find container \"61ea40fc745b3bb1015069215985f45711621cb794c7c73809efd6112fc793f0\": container with ID starting with 61ea40fc745b3bb1015069215985f45711621cb794c7c73809efd6112fc793f0 not found: ID does not exist" Nov 01 16:29:03.473541 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:03.473517 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc"] Nov 01 16:29:03.487148 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:03.487119 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-104b6e-openshift-default-99f954b75-nrzqc"] Nov 01 16:29:04.801954 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:04.801918 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be90dd72-5635-4115-9826-26aaa46f70dc" path="/var/lib/kubelet/pods/be90dd72-5635-4115-9826-26aaa46f70dc/volumes" Nov 01 16:29:18.316431 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:29:18.316385 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:29:24.112009 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:24.111944 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7321d6-5c5749c946-tmb8z"] Nov 01 16:29:24.113442 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:24.113397 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-7321d6-5c5749c946-tmb8z" podUID="184a13e5-38cc-42b9-a792-dda5dfdc4336" containerName="payload-processing" containerID="cri-o://82eb4761154ee60637c8aaf5ef00f46839316b3a354b29469c05f3a572966cb5" gracePeriod=30 Nov 01 16:29:24.149653 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:24.131614 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-6f78d745c9-xgnqg"] Nov 01 16:29:24.149653 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:24.132267 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-6f78d745c9-xgnqg" podUID="c57093ac-f9f4-4739-bb9c-6b0aef206ba3" containerName="payload-pre-processing" containerID="cri-o://b5a32dc03572943a772673d52e31506791e03db87a179dfbcc8d053e17cb931f" gracePeriod=30 Nov 01 16:29:24.240763 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:24.240725 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b"] Nov 01 16:29:24.240997 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:24.240968 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b" podUID="45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2" containerName="maas-api" containerID="cri-o://ba010ce47ece69eb44f07c3e8f73bf3cc29e0d41cc2dc4077f1c5e3e6e31d26d" gracePeriod=30 Nov 01 16:29:24.806675 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:24.806358 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7321d6-5c5749c946-tmb8z" Nov 01 16:29:24.856456 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:24.856429 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-6f78d745c9-xgnqg" Nov 01 16:29:24.902039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:24.899727 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/184a13e5-38cc-42b9-a792-dda5dfdc4336-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/184a13e5-38cc-42b9-a792-dda5dfdc4336-plugins-config-volume\") pod \"184a13e5-38cc-42b9-a792-dda5dfdc4336\" (UID: \"184a13e5-38cc-42b9-a792-dda5dfdc4336\") " Nov 01 16:29:24.902039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:24.899846 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/184a13e5-38cc-42b9-a792-dda5dfdc4336-kube-api-access-ngkzg\" (UniqueName: \"kubernetes.io/projected/184a13e5-38cc-42b9-a792-dda5dfdc4336-kube-api-access-ngkzg\") pod \"184a13e5-38cc-42b9-a792-dda5dfdc4336\" (UID: \"184a13e5-38cc-42b9-a792-dda5dfdc4336\") " Nov 01 16:29:24.909956 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:24.909908 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/184a13e5-38cc-42b9-a792-dda5dfdc4336-plugins-config-volume" pod "184a13e5-38cc-42b9-a792-dda5dfdc4336" (UID: "184a13e5-38cc-42b9-a792-dda5dfdc4336"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:29:24.930200 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:24.921553 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/184a13e5-38cc-42b9-a792-dda5dfdc4336-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:24.938335 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:24.923452 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/184a13e5-38cc-42b9-a792-dda5dfdc4336-kube-api-access-ngkzg" pod "184a13e5-38cc-42b9-a792-dda5dfdc4336" (UID: "184a13e5-38cc-42b9-a792-dda5dfdc4336"). InnerVolumeSpecName "kube-api-access-ngkzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:29:24.947086 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:24.924066 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b" Nov 01 16:29:24.982435 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:24.982366 4363 generic.go:356] "Generic (PLEG): container finished" podID="45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2" containerID="ba010ce47ece69eb44f07c3e8f73bf3cc29e0d41cc2dc4077f1c5e3e6e31d26d" exitCode=0 Nov 01 16:29:24.982799 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:24.982629 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b" event={"ID":"45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2","Type":"ContainerDied","Data":"ba010ce47ece69eb44f07c3e8f73bf3cc29e0d41cc2dc4077f1c5e3e6e31d26d"} Nov 01 16:29:24.982908 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:24.982839 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b" event={"ID":"45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2","Type":"ContainerDied","Data":"d9dddc497211791ce9c723c6141ce2a18c76db6158c0604ecb1d600b35ed0b77"} Nov 01 16:29:24.982908 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:24.982875 4363 scope.go:122] "RemoveContainer" containerID="ba010ce47ece69eb44f07c3e8f73bf3cc29e0d41cc2dc4077f1c5e3e6e31d26d" Nov 01 16:29:24.983047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:24.982700 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b" Nov 01 16:29:24.989479 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:24.989453 4363 generic.go:356] "Generic (PLEG): container finished" podID="184a13e5-38cc-42b9-a792-dda5dfdc4336" containerID="82eb4761154ee60637c8aaf5ef00f46839316b3a354b29469c05f3a572966cb5" exitCode=0 Nov 01 16:29:24.989889 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:24.989637 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6b0e0f-4lpmp"] Nov 01 16:29:24.990185 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:24.990164 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7321d6-5c5749c946-tmb8z" event={"ID":"184a13e5-38cc-42b9-a792-dda5dfdc4336","Type":"ContainerDied","Data":"82eb4761154ee60637c8aaf5ef00f46839316b3a354b29469c05f3a572966cb5"} Nov 01 16:29:24.990336 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:24.990319 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7321d6-5c5749c946-tmb8z" event={"ID":"184a13e5-38cc-42b9-a792-dda5dfdc4336","Type":"ContainerDied","Data":"700185a256f4ef4cfa619c5c7bd6d09c8f218716cc995369f5a53ef25a5f34c3"} Nov 01 16:29:24.990450 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:24.989743 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7321d6-5c5749c946-tmb8z" Nov 01 16:29:24.996989 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:24.996949 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6b0e0f-4lpmp"] Nov 01 16:29:25.001490 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.001468 4363 generic.go:356] "Generic (PLEG): container finished" podID="c57093ac-f9f4-4739-bb9c-6b0aef206ba3" containerID="b5a32dc03572943a772673d52e31506791e03db87a179dfbcc8d053e17cb931f" exitCode=0 Nov 01 16:29:25.001707 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.001690 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-6f78d745c9-xgnqg" event={"ID":"c57093ac-f9f4-4739-bb9c-6b0aef206ba3","Type":"ContainerDied","Data":"b5a32dc03572943a772673d52e31506791e03db87a179dfbcc8d053e17cb931f"} Nov 01 16:29:25.001846 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.001831 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-6f78d745c9-xgnqg" event={"ID":"c57093ac-f9f4-4739-bb9c-6b0aef206ba3","Type":"ContainerDied","Data":"14a2ced2981c427e01041d811108bfe774d75b158de47583e3d0718532b49abd"} Nov 01 16:29:25.002001 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.001987 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-6f78d745c9-xgnqg" Nov 01 16:29:25.006576 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.006488 4363 scope.go:122] "RemoveContainer" containerID="ba010ce47ece69eb44f07c3e8f73bf3cc29e0d41cc2dc4077f1c5e3e6e31d26d" Nov 01 16:29:25.006839 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:29:25.006808 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba010ce47ece69eb44f07c3e8f73bf3cc29e0d41cc2dc4077f1c5e3e6e31d26d\": container with ID starting with ba010ce47ece69eb44f07c3e8f73bf3cc29e0d41cc2dc4077f1c5e3e6e31d26d not found: ID does not exist" containerID="ba010ce47ece69eb44f07c3e8f73bf3cc29e0d41cc2dc4077f1c5e3e6e31d26d" Nov 01 16:29:25.006908 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.006851 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba010ce47ece69eb44f07c3e8f73bf3cc29e0d41cc2dc4077f1c5e3e6e31d26d"} err="failed to get container status \"ba010ce47ece69eb44f07c3e8f73bf3cc29e0d41cc2dc4077f1c5e3e6e31d26d\": rpc error: code = NotFound desc = could not find container \"ba010ce47ece69eb44f07c3e8f73bf3cc29e0d41cc2dc4077f1c5e3e6e31d26d\": container with ID starting with ba010ce47ece69eb44f07c3e8f73bf3cc29e0d41cc2dc4077f1c5e3e6e31d26d not found: ID does not exist" Nov 01 16:29:25.006908 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.006874 4363 scope.go:122] "RemoveContainer" containerID="82eb4761154ee60637c8aaf5ef00f46839316b3a354b29469c05f3a572966cb5" Nov 01 16:29:25.026473 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.026383 4363 scope.go:122] "RemoveContainer" containerID="82eb4761154ee60637c8aaf5ef00f46839316b3a354b29469c05f3a572966cb5" Nov 01 16:29:25.026936 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:29:25.026909 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82eb4761154ee60637c8aaf5ef00f46839316b3a354b29469c05f3a572966cb5\": container with ID starting with 82eb4761154ee60637c8aaf5ef00f46839316b3a354b29469c05f3a572966cb5 not found: ID does not exist" containerID="82eb4761154ee60637c8aaf5ef00f46839316b3a354b29469c05f3a572966cb5" Nov 01 16:29:25.027087 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.026947 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82eb4761154ee60637c8aaf5ef00f46839316b3a354b29469c05f3a572966cb5"} err="failed to get container status \"82eb4761154ee60637c8aaf5ef00f46839316b3a354b29469c05f3a572966cb5\": rpc error: code = NotFound desc = could not find container \"82eb4761154ee60637c8aaf5ef00f46839316b3a354b29469c05f3a572966cb5\": container with ID starting with 82eb4761154ee60637c8aaf5ef00f46839316b3a354b29469c05f3a572966cb5 not found: ID does not exist" Nov 01 16:29:25.027087 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.026973 4363 scope.go:122] "RemoveContainer" containerID="b5a32dc03572943a772673d52e31506791e03db87a179dfbcc8d053e17cb931f" Nov 01 16:29:25.027226 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.027111 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7321d6-5c5749c946-tmb8z"] Nov 01 16:29:25.031702 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.031677 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7321d6-5c5749c946-tmb8z"] Nov 01 16:29:25.038055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.038011 4363 scope.go:122] "RemoveContainer" containerID="b5a32dc03572943a772673d52e31506791e03db87a179dfbcc8d053e17cb931f" Nov 01 16:29:25.038317 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:29:25.038297 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5a32dc03572943a772673d52e31506791e03db87a179dfbcc8d053e17cb931f\": container with ID starting with b5a32dc03572943a772673d52e31506791e03db87a179dfbcc8d053e17cb931f not found: ID does not exist" containerID="b5a32dc03572943a772673d52e31506791e03db87a179dfbcc8d053e17cb931f" Nov 01 16:29:25.038380 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.038326 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5a32dc03572943a772673d52e31506791e03db87a179dfbcc8d053e17cb931f"} err="failed to get container status \"b5a32dc03572943a772673d52e31506791e03db87a179dfbcc8d053e17cb931f\": rpc error: code = NotFound desc = could not find container \"b5a32dc03572943a772673d52e31506791e03db87a179dfbcc8d053e17cb931f\": container with ID starting with b5a32dc03572943a772673d52e31506791e03db87a179dfbcc8d053e17cb931f not found: ID does not exist" Nov 01 16:29:25.038850 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.038831 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c57093ac-f9f4-4739-bb9c-6b0aef206ba3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c57093ac-f9f4-4739-bb9c-6b0aef206ba3-plugins-config-volume\") pod \"c57093ac-f9f4-4739-bb9c-6b0aef206ba3\" (UID: \"c57093ac-f9f4-4739-bb9c-6b0aef206ba3\") " Nov 01 16:29:25.038908 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.038896 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2-kube-api-access-tjpbw\" (UniqueName: \"kubernetes.io/projected/45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2-kube-api-access-tjpbw\") pod \"45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2\" (UID: \"45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2\") " Nov 01 16:29:25.038960 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.038928 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2-maas-api-tls\") pod \"45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2\" (UID: \"45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2\") " Nov 01 16:29:25.039036 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.038976 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2-metrics-certs\") pod \"45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2\" (UID: \"45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2\") " Nov 01 16:29:25.039446 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.039111 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c57093ac-f9f4-4739-bb9c-6b0aef206ba3-kube-api-access-tn8vv\" (UniqueName: \"kubernetes.io/projected/c57093ac-f9f4-4739-bb9c-6b0aef206ba3-kube-api-access-tn8vv\") pod \"c57093ac-f9f4-4739-bb9c-6b0aef206ba3\" (UID: \"c57093ac-f9f4-4739-bb9c-6b0aef206ba3\") " Nov 01 16:29:25.039446 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.039136 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2-openshift-service-ca\") pod \"45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2\" (UID: \"45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2\") " Nov 01 16:29:25.039446 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.039220 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c57093ac-f9f4-4739-bb9c-6b0aef206ba3-plugins-config-volume" pod "c57093ac-f9f4-4739-bb9c-6b0aef206ba3" (UID: "c57093ac-f9f4-4739-bb9c-6b0aef206ba3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:29:25.039670 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.039648 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2-openshift-service-ca" pod "45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2" (UID: "45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:29:25.040000 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.039982 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngkzg\" (UniqueName: \"kubernetes.io/projected/184a13e5-38cc-42b9-a792-dda5dfdc4336-kube-api-access-ngkzg\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:25.040114 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.040002 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c57093ac-f9f4-4739-bb9c-6b0aef206ba3-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:25.040114 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.040011 4363 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2-openshift-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:25.041461 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.041415 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c57093ac-f9f4-4739-bb9c-6b0aef206ba3-kube-api-access-tn8vv" pod "c57093ac-f9f4-4739-bb9c-6b0aef206ba3" (UID: "c57093ac-f9f4-4739-bb9c-6b0aef206ba3"). InnerVolumeSpecName "kube-api-access-tn8vv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:29:25.041716 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.041692 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2-maas-api-tls" pod "45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2" (UID: "45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:29:25.041921 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.041903 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2-kube-api-access-tjpbw" pod "45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2" (UID: "45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2"). InnerVolumeSpecName "kube-api-access-tjpbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:29:25.042080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.042060 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2-metrics-certs" pod "45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2" (UID: "45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:29:25.142055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.141998 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjpbw\" (UniqueName: \"kubernetes.io/projected/45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2-kube-api-access-tjpbw\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:25.142055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.142045 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2-maas-api-tls\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:25.142055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.142060 4363 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2-metrics-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:25.142552 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.142076 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn8vv\" (UniqueName: \"kubernetes.io/projected/c57093ac-f9f4-4739-bb9c-6b0aef206ba3-kube-api-access-tn8vv\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:25.317274 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.317249 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b"] Nov 01 16:29:25.323380 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.323347 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-6b0e0f-684bc7c8b-7l49b"] Nov 01 16:29:25.339797 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.339763 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-6f78d745c9-xgnqg"] Nov 01 16:29:25.344508 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.344482 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7321d6-6f78d745c9-xgnqg"] Nov 01 16:29:25.455437 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.455404 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11450cde-vvjzv"] Nov 01 16:29:25.456475 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.456457 4363 cpu_manager.go:414] "containerMap: removing container" podUID="45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2" containerName="maas-api" Nov 01 16:29:25.456578 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.456477 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2" containerName="maas-api" Nov 01 16:29:25.456578 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.456489 4363 cpu_manager.go:414] "containerMap: removing container" podUID="1b180ef5-70b3-4bf2-901b-db280a232bde" containerName="istio-proxy" Nov 01 16:29:25.456578 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.456497 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b180ef5-70b3-4bf2-901b-db280a232bde" containerName="istio-proxy" Nov 01 16:29:25.456578 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.456522 4363 cpu_manager.go:414] "containerMap: removing container" podUID="184a13e5-38cc-42b9-a792-dda5dfdc4336" containerName="payload-processing" Nov 01 16:29:25.456578 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.456530 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="184a13e5-38cc-42b9-a792-dda5dfdc4336" containerName="payload-processing" Nov 01 16:29:25.456578 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.456544 4363 cpu_manager.go:414] "containerMap: removing container" podUID="ca74811d-13fc-4a4c-bfd9-9e787392ffce" containerName="istio-proxy" Nov 01 16:29:25.456578 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.456552 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca74811d-13fc-4a4c-bfd9-9e787392ffce" containerName="istio-proxy" Nov 01 16:29:25.456578 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.456579 4363 cpu_manager.go:414] "containerMap: removing container" podUID="be90dd72-5635-4115-9826-26aaa46f70dc" containerName="istio-proxy" Nov 01 16:29:25.456990 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.456587 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be90dd72-5635-4115-9826-26aaa46f70dc" containerName="istio-proxy" Nov 01 16:29:25.456990 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.456608 4363 cpu_manager.go:414] "containerMap: removing container" podUID="77a6ec04-31e5-4d7e-a829-a4a0704de9db" containerName="istio-proxy" Nov 01 16:29:25.456990 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.456621 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77a6ec04-31e5-4d7e-a829-a4a0704de9db" containerName="istio-proxy" Nov 01 16:29:25.456990 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.456636 4363 cpu_manager.go:414] "containerMap: removing container" podUID="c57093ac-f9f4-4739-bb9c-6b0aef206ba3" containerName="payload-pre-processing" Nov 01 16:29:25.456990 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.456644 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c57093ac-f9f4-4739-bb9c-6b0aef206ba3" containerName="payload-pre-processing" Nov 01 16:29:25.456990 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.456804 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="c57093ac-f9f4-4739-bb9c-6b0aef206ba3" containerName="payload-pre-processing" Nov 01 16:29:25.456990 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.456821 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="be90dd72-5635-4115-9826-26aaa46f70dc" containerName="istio-proxy" Nov 01 16:29:25.456990 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.456835 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b180ef5-70b3-4bf2-901b-db280a232bde" containerName="istio-proxy" Nov 01 16:29:25.456990 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.456844 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2" containerName="maas-api" Nov 01 16:29:25.456990 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.456854 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b180ef5-70b3-4bf2-901b-db280a232bde" containerName="istio-proxy" Nov 01 16:29:25.456990 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.456864 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="184a13e5-38cc-42b9-a792-dda5dfdc4336" containerName="payload-processing" Nov 01 16:29:25.456990 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.456874 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca74811d-13fc-4a4c-bfd9-9e787392ffce" containerName="istio-proxy" Nov 01 16:29:25.456990 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.456887 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="77a6ec04-31e5-4d7e-a829-a4a0704de9db" containerName="istio-proxy" Nov 01 16:29:25.466333 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.462692 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11450cde-vvjzv" Nov 01 16:29:25.466333 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.466253 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11450cde-vvjzv"] Nov 01 16:29:25.466491 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.466418 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:29:25.466578 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.466557 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:29:25.649336 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.649300 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08a13416-5e1c-4bc0-a4ea-023e4960bc2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11450cde-vvjzv\" (UID: \"08a13416-5e1c-4bc0-a4ea-023e4960bc2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11450cde-vvjzv" Nov 01 16:29:25.649634 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.649609 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08a13416-5e1c-4bc0-a4ea-023e4960bc2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11450cde-vvjzv\" (UID: \"08a13416-5e1c-4bc0-a4ea-023e4960bc2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11450cde-vvjzv" Nov 01 16:29:25.649728 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.649665 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52lgs\" (UniqueName: \"kubernetes.io/projected/08a13416-5e1c-4bc0-a4ea-023e4960bc2f-kube-api-access-52lgs\") pod \"maas-api-revoke-sub-11450cde-vvjzv\" (UID: \"08a13416-5e1c-4bc0-a4ea-023e4960bc2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11450cde-vvjzv" Nov 01 16:29:25.751647 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.751614 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08a13416-5e1c-4bc0-a4ea-023e4960bc2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11450cde-vvjzv\" (UID: \"08a13416-5e1c-4bc0-a4ea-023e4960bc2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11450cde-vvjzv" Nov 01 16:29:25.751853 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.751830 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08a13416-5e1c-4bc0-a4ea-023e4960bc2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11450cde-vvjzv\" (UID: \"08a13416-5e1c-4bc0-a4ea-023e4960bc2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11450cde-vvjzv" Nov 01 16:29:25.751926 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.751864 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52lgs\" (UniqueName: \"kubernetes.io/projected/08a13416-5e1c-4bc0-a4ea-023e4960bc2f-kube-api-access-52lgs\") pod \"maas-api-revoke-sub-11450cde-vvjzv\" (UID: \"08a13416-5e1c-4bc0-a4ea-023e4960bc2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11450cde-vvjzv" Nov 01 16:29:25.752647 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.752611 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08a13416-5e1c-4bc0-a4ea-023e4960bc2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11450cde-vvjzv\" (UID: \"08a13416-5e1c-4bc0-a4ea-023e4960bc2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11450cde-vvjzv" Nov 01 16:29:25.754353 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.754321 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08a13416-5e1c-4bc0-a4ea-023e4960bc2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11450cde-vvjzv\" (UID: \"08a13416-5e1c-4bc0-a4ea-023e4960bc2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11450cde-vvjzv" Nov 01 16:29:25.759383 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.759353 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52lgs\" (UniqueName: \"kubernetes.io/projected/08a13416-5e1c-4bc0-a4ea-023e4960bc2f-kube-api-access-52lgs\") pod \"maas-api-revoke-sub-11450cde-vvjzv\" (UID: \"08a13416-5e1c-4bc0-a4ea-023e4960bc2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11450cde-vvjzv" Nov 01 16:29:25.781172 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:25.781146 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11450cde-vvjzv" Nov 01 16:29:26.097088 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:26.097055 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11450cde-vvjzv"] Nov 01 16:29:26.097210 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:29:26.097081 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08a13416_5e1c_4bc0_a4ea_023e4960bc2f.slice/crio-7caf6db8146052b1ef045b86c38c96aca4d255cd409b1f1caa89cce686961622 WatchSource:0}: Error finding container 7caf6db8146052b1ef045b86c38c96aca4d255cd409b1f1caa89cce686961622: Status 404 returned error can't find the container with id 7caf6db8146052b1ef045b86c38c96aca4d255cd409b1f1caa89cce686961622 Nov 01 16:29:26.962091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:26.955870 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="184a13e5-38cc-42b9-a792-dda5dfdc4336" path="/var/lib/kubelet/pods/184a13e5-38cc-42b9-a792-dda5dfdc4336/volumes" Nov 01 16:29:26.989039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:26.967925 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2" path="/var/lib/kubelet/pods/45637fe1-f7c2-4ce9-aee1-79d93fc3d2b2/volumes" Nov 01 16:29:26.989039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:26.968699 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b881d53-3472-4c0a-99a2-4b0f2b693a07" path="/var/lib/kubelet/pods/8b881d53-3472-4c0a-99a2-4b0f2b693a07/volumes" Nov 01 16:29:26.989039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:26.969275 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c57093ac-f9f4-4739-bb9c-6b0aef206ba3" path="/var/lib/kubelet/pods/c57093ac-f9f4-4739-bb9c-6b0aef206ba3/volumes" Nov 01 16:29:27.094655 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:27.093867 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11450cde-vvjzv" event={"ID":"08a13416-5e1c-4bc0-a4ea-023e4960bc2f","Type":"ContainerStarted","Data":"135a09e4b651f948a2cb35aac342ba4828b33c07358a429fccdf08a44569003a"} Nov 01 16:29:27.094655 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:27.093934 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11450cde-vvjzv" event={"ID":"08a13416-5e1c-4bc0-a4ea-023e4960bc2f","Type":"ContainerStarted","Data":"7caf6db8146052b1ef045b86c38c96aca4d255cd409b1f1caa89cce686961622"} Nov 01 16:29:27.115148 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:27.115042 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11450cde-vvjzv" podStartSLOduration=2.115004085 podStartE2EDuration="2.115004085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:29:27.109415477 +0000 UTC m=+2500.441153128" watchObservedRunningTime="2026-11-01 16:29:27.115004085 +0000 UTC m=+2500.446741738" Nov 01 16:29:27.163723 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:29:27.163694 4363 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08a13416_5e1c_4bc0_a4ea_023e4960bc2f.slice/crio-135a09e4b651f948a2cb35aac342ba4828b33c07358a429fccdf08a44569003a.scope\": RecentStats: unable to find data in memory cache]" Nov 01 16:29:28.106527 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:28.106481 4363 generic.go:356] "Generic (PLEG): container finished" podID="08a13416-5e1c-4bc0-a4ea-023e4960bc2f" containerID="135a09e4b651f948a2cb35aac342ba4828b33c07358a429fccdf08a44569003a" exitCode=0 Nov 01 16:29:28.106923 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:28.106558 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11450cde-vvjzv" event={"ID":"08a13416-5e1c-4bc0-a4ea-023e4960bc2f","Type":"ContainerDied","Data":"135a09e4b651f948a2cb35aac342ba4828b33c07358a429fccdf08a44569003a"} Nov 01 16:29:29.353859 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:29.353829 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11450cde-vvjzv" Nov 01 16:29:29.520661 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:29.520561 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08a13416-5e1c-4bc0-a4ea-023e4960bc2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08a13416-5e1c-4bc0-a4ea-023e4960bc2f-maas-api-xks-ca\") pod \"08a13416-5e1c-4bc0-a4ea-023e4960bc2f\" (UID: \"08a13416-5e1c-4bc0-a4ea-023e4960bc2f\") " Nov 01 16:29:29.520828 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:29.520720 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08a13416-5e1c-4bc0-a4ea-023e4960bc2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08a13416-5e1c-4bc0-a4ea-023e4960bc2f-maas-api-openshift-ca\") pod \"08a13416-5e1c-4bc0-a4ea-023e4960bc2f\" (UID: \"08a13416-5e1c-4bc0-a4ea-023e4960bc2f\") " Nov 01 16:29:29.520828 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:29.520763 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08a13416-5e1c-4bc0-a4ea-023e4960bc2f-kube-api-access-52lgs\" (UniqueName: \"kubernetes.io/projected/08a13416-5e1c-4bc0-a4ea-023e4960bc2f-kube-api-access-52lgs\") pod \"08a13416-5e1c-4bc0-a4ea-023e4960bc2f\" (UID: \"08a13416-5e1c-4bc0-a4ea-023e4960bc2f\") " Nov 01 16:29:29.521238 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:29.521197 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08a13416-5e1c-4bc0-a4ea-023e4960bc2f-maas-api-openshift-ca" pod "08a13416-5e1c-4bc0-a4ea-023e4960bc2f" (UID: "08a13416-5e1c-4bc0-a4ea-023e4960bc2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:29:29.521600 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:29.521581 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08a13416-5e1c-4bc0-a4ea-023e4960bc2f-maas-api-openshift-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:29.522858 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:29.522833 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08a13416-5e1c-4bc0-a4ea-023e4960bc2f-maas-api-xks-ca" pod "08a13416-5e1c-4bc0-a4ea-023e4960bc2f" (UID: "08a13416-5e1c-4bc0-a4ea-023e4960bc2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:29:29.523087 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:29.523069 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08a13416-5e1c-4bc0-a4ea-023e4960bc2f-kube-api-access-52lgs" pod "08a13416-5e1c-4bc0-a4ea-023e4960bc2f" (UID: "08a13416-5e1c-4bc0-a4ea-023e4960bc2f"). InnerVolumeSpecName "kube-api-access-52lgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:29:29.623228 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:29.623196 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52lgs\" (UniqueName: \"kubernetes.io/projected/08a13416-5e1c-4bc0-a4ea-023e4960bc2f-kube-api-access-52lgs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:29.623228 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:29.623225 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08a13416-5e1c-4bc0-a4ea-023e4960bc2f-maas-api-xks-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:30.141418 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:30.141387 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11450cde-vvjzv" Nov 01 16:29:30.141607 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:30.141417 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11450cde-vvjzv" event={"ID":"08a13416-5e1c-4bc0-a4ea-023e4960bc2f","Type":"ContainerDied","Data":"7caf6db8146052b1ef045b86c38c96aca4d255cd409b1f1caa89cce686961622"} Nov 01 16:29:30.141607 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:30.141507 4363 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7caf6db8146052b1ef045b86c38c96aca4d255cd409b1f1caa89cce686961622" Nov 01 16:29:32.515037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:32.514687 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq"] Nov 01 16:29:32.515037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:32.514936 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" podUID="742547a4-fbd7-40d3-a7a3-86fc91fd7cc3" containerName="istio-proxy" containerID="cri-o://5dc6bfe92933edec7e596d52fece68e868ea7ca66853a0220f2a18209a7ec8ff" gracePeriod=30 Nov 01 16:29:33.093995 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:33.093963 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5e2d03-pw25b"] Nov 01 16:29:33.094892 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:33.094867 4363 cpu_manager.go:414] "containerMap: removing container" podUID="1b180ef5-70b3-4bf2-901b-db280a232bde" containerName="istio-proxy" Nov 01 16:29:33.094892 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:33.094888 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b180ef5-70b3-4bf2-901b-db280a232bde" containerName="istio-proxy" Nov 01 16:29:33.095131 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:33.094917 4363 cpu_manager.go:414] "containerMap: removing container" podUID="08a13416-5e1c-4bc0-a4ea-023e4960bc2f" containerName="revoke-keys" Nov 01 16:29:33.095131 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:33.094923 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08a13416-5e1c-4bc0-a4ea-023e4960bc2f" containerName="revoke-keys" Nov 01 16:29:33.095131 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:33.094932 4363 cpu_manager.go:414] "containerMap: removing container" podUID="be90dd72-5635-4115-9826-26aaa46f70dc" containerName="istio-proxy" Nov 01 16:29:33.095131 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:33.094938 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be90dd72-5635-4115-9826-26aaa46f70dc" containerName="istio-proxy" Nov 01 16:29:33.095131 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:33.095055 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="08a13416-5e1c-4bc0-a4ea-023e4960bc2f" containerName="revoke-keys" Nov 01 16:29:33.095131 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:33.095066 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="be90dd72-5635-4115-9826-26aaa46f70dc" containerName="istio-proxy" Nov 01 16:29:33.099007 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:33.098983 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5e2d03-pw25b" Nov 01 16:29:33.106816 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:33.106757 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:29:33.107201 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:33.107159 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:29:33.109519 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:33.109495 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5e2d03-pw25b"] Nov 01 16:29:33.207261 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:33.207168 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvjrw\" (UniqueName: \"kubernetes.io/projected/eb7ee4b9-3e65-496f-b221-b6d29c10fa97-kube-api-access-bvjrw\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5e2d03-pw25b\" (UID: \"eb7ee4b9-3e65-496f-b221-b6d29c10fa97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5e2d03-pw25b" Nov 01 16:29:33.207784 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:33.207756 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb7ee4b9-3e65-496f-b221-b6d29c10fa97-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5e2d03-pw25b\" (UID: \"eb7ee4b9-3e65-496f-b221-b6d29c10fa97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5e2d03-pw25b" Nov 01 16:29:33.208750 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:33.208723 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb7ee4b9-3e65-496f-b221-b6d29c10fa97-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5e2d03-pw25b\" (UID: \"eb7ee4b9-3e65-496f-b221-b6d29c10fa97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5e2d03-pw25b" Nov 01 16:29:33.342264 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:33.342223 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb7ee4b9-3e65-496f-b221-b6d29c10fa97-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5e2d03-pw25b\" (UID: \"eb7ee4b9-3e65-496f-b221-b6d29c10fa97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5e2d03-pw25b" Nov 01 16:29:33.353809 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:33.353716 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvjrw\" (UniqueName: \"kubernetes.io/projected/eb7ee4b9-3e65-496f-b221-b6d29c10fa97-kube-api-access-bvjrw\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5e2d03-pw25b\" (UID: \"eb7ee4b9-3e65-496f-b221-b6d29c10fa97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5e2d03-pw25b" Nov 01 16:29:33.357165 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:33.354235 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb7ee4b9-3e65-496f-b221-b6d29c10fa97-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5e2d03-pw25b\" (UID: \"eb7ee4b9-3e65-496f-b221-b6d29c10fa97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5e2d03-pw25b" Nov 01 16:29:33.359538 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:33.358615 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb7ee4b9-3e65-496f-b221-b6d29c10fa97-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5e2d03-pw25b\" (UID: \"eb7ee4b9-3e65-496f-b221-b6d29c10fa97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5e2d03-pw25b" Nov 01 16:29:33.375485 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:33.375452 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb7ee4b9-3e65-496f-b221-b6d29c10fa97-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5e2d03-pw25b\" (UID: \"eb7ee4b9-3e65-496f-b221-b6d29c10fa97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5e2d03-pw25b" Nov 01 16:29:33.388965 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:33.388930 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvjrw\" (UniqueName: \"kubernetes.io/projected/eb7ee4b9-3e65-496f-b221-b6d29c10fa97-kube-api-access-bvjrw\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5e2d03-pw25b\" (UID: \"eb7ee4b9-3e65-496f-b221-b6d29c10fa97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5e2d03-pw25b" Nov 01 16:29:33.424672 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:33.424631 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5e2d03-pw25b" Nov 01 16:29:33.744986 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:29:33.744942 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb7ee4b9_3e65_496f_b221_b6d29c10fa97.slice/crio-28ff783bd3da0f7aee45b08f39452dac4dbc611917e1eceae884aa5b39c8678b WatchSource:0}: Error finding container 28ff783bd3da0f7aee45b08f39452dac4dbc611917e1eceae884aa5b39c8678b: Status 404 returned error can't find the container with id 28ff783bd3da0f7aee45b08f39452dac4dbc611917e1eceae884aa5b39c8678b Nov 01 16:29:33.745343 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:33.745196 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5e2d03-pw25b"] Nov 01 16:29:34.306417 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:34.306343 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5e2d03-pw25b" event={"ID":"eb7ee4b9-3e65-496f-b221-b6d29c10fa97","Type":"ContainerStarted","Data":"528285b11d82c8fe43d425d5e388473abb8f2782f0e0bc3e00567d2bfc460c71"} Nov 01 16:29:34.306417 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:34.306410 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5e2d03-pw25b" event={"ID":"eb7ee4b9-3e65-496f-b221-b6d29c10fa97","Type":"ContainerStarted","Data":"28ff783bd3da0f7aee45b08f39452dac4dbc611917e1eceae884aa5b39c8678b"} Nov 01 16:29:34.324872 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:34.324830 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5e2d03-pw25b" podStartSLOduration=1.324817611 podStartE2EDuration="1.324817611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:29:34.320887035 +0000 UTC m=+2507.652624685" watchObservedRunningTime="2026-11-01 16:29:34.324817611 +0000 UTC m=+2507.656555261" Nov 01 16:29:35.316597 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:35.316497 4363 generic.go:356] "Generic (PLEG): container finished" podID="eb7ee4b9-3e65-496f-b221-b6d29c10fa97" containerID="528285b11d82c8fe43d425d5e388473abb8f2782f0e0bc3e00567d2bfc460c71" exitCode=0 Nov 01 16:29:35.316597 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:35.316547 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5e2d03-pw25b" event={"ID":"eb7ee4b9-3e65-496f-b221-b6d29c10fa97","Type":"ContainerDied","Data":"528285b11d82c8fe43d425d5e388473abb8f2782f0e0bc3e00567d2bfc460c71"} Nov 01 16:29:36.513569 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:36.513539 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5e2d03-pw25b" Nov 01 16:29:36.620861 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:36.620834 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb7ee4b9-3e65-496f-b221-b6d29c10fa97-kube-api-access-bvjrw\" (UniqueName: \"kubernetes.io/projected/eb7ee4b9-3e65-496f-b221-b6d29c10fa97-kube-api-access-bvjrw\") pod \"eb7ee4b9-3e65-496f-b221-b6d29c10fa97\" (UID: \"eb7ee4b9-3e65-496f-b221-b6d29c10fa97\") " Nov 01 16:29:36.621111 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:36.620951 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb7ee4b9-3e65-496f-b221-b6d29c10fa97-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb7ee4b9-3e65-496f-b221-b6d29c10fa97-maas-api-openshift-ca\") pod \"eb7ee4b9-3e65-496f-b221-b6d29c10fa97\" (UID: \"eb7ee4b9-3e65-496f-b221-b6d29c10fa97\") " Nov 01 16:29:36.621111 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:36.620991 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb7ee4b9-3e65-496f-b221-b6d29c10fa97-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb7ee4b9-3e65-496f-b221-b6d29c10fa97-maas-api-xks-ca\") pod \"eb7ee4b9-3e65-496f-b221-b6d29c10fa97\" (UID: \"eb7ee4b9-3e65-496f-b221-b6d29c10fa97\") " Nov 01 16:29:36.621327 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:36.621306 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb7ee4b9-3e65-496f-b221-b6d29c10fa97-maas-api-openshift-ca" pod "eb7ee4b9-3e65-496f-b221-b6d29c10fa97" (UID: "eb7ee4b9-3e65-496f-b221-b6d29c10fa97"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:29:36.621656 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:36.621639 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb7ee4b9-3e65-496f-b221-b6d29c10fa97-maas-api-openshift-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:36.623097 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:36.623067 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb7ee4b9-3e65-496f-b221-b6d29c10fa97-kube-api-access-bvjrw" pod "eb7ee4b9-3e65-496f-b221-b6d29c10fa97" (UID: "eb7ee4b9-3e65-496f-b221-b6d29c10fa97"). InnerVolumeSpecName "kube-api-access-bvjrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:29:36.623236 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:36.623071 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb7ee4b9-3e65-496f-b221-b6d29c10fa97-maas-api-xks-ca" pod "eb7ee4b9-3e65-496f-b221-b6d29c10fa97" (UID: "eb7ee4b9-3e65-496f-b221-b6d29c10fa97"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:29:36.722737 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:36.722698 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvjrw\" (UniqueName: \"kubernetes.io/projected/eb7ee4b9-3e65-496f-b221-b6d29c10fa97-kube-api-access-bvjrw\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:36.722737 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:36.722734 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb7ee4b9-3e65-496f-b221-b6d29c10fa97-maas-api-xks-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:37.338432 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:37.338400 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5e2d03-pw25b" Nov 01 16:29:37.338635 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:37.338438 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5e2d03-pw25b" event={"ID":"eb7ee4b9-3e65-496f-b221-b6d29c10fa97","Type":"ContainerDied","Data":"28ff783bd3da0f7aee45b08f39452dac4dbc611917e1eceae884aa5b39c8678b"} Nov 01 16:29:37.338635 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:37.338497 4363 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28ff783bd3da0f7aee45b08f39452dac4dbc611917e1eceae884aa5b39c8678b" Nov 01 16:29:37.850551 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:37.850526 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" Nov 01 16:29:37.956398 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:37.956320 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-kube-api-access-wf5dk\" (UniqueName: \"kubernetes.io/projected/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-kube-api-access-wf5dk\") pod \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\" (UID: \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\") " Nov 01 16:29:37.956562 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:37.956415 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-istiod-ca-cert\") pod \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\" (UID: \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\") " Nov 01 16:29:37.956562 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:37.956460 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-istio-podinfo\") pod \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\" (UID: \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\") " Nov 01 16:29:37.956562 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:37.956498 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-workload-certs\") pod \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\" (UID: \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\") " Nov 01 16:29:37.956757 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:37.956561 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-istio-data\") pod \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\" (UID: \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\") " Nov 01 16:29:37.956757 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:37.956630 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-workload-socket\") pod \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\" (UID: \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\") " Nov 01 16:29:37.956757 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:37.956675 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-istio-envoy\") pod \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\" (UID: \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\") " Nov 01 16:29:37.956757 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:37.956720 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-credential-socket\") pod \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\" (UID: \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\") " Nov 01 16:29:37.956994 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:37.956812 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-istio-token\" (UniqueName: \"kubernetes.io/projected/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-istio-token\") pod \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\" (UID: \"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3\") " Nov 01 16:29:37.956994 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:37.956841 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-workload-certs" pod "742547a4-fbd7-40d3-a7a3-86fc91fd7cc3" (UID: "742547a4-fbd7-40d3-a7a3-86fc91fd7cc3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:37.956994 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:37.956891 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-istiod-ca-cert" pod "742547a4-fbd7-40d3-a7a3-86fc91fd7cc3" (UID: "742547a4-fbd7-40d3-a7a3-86fc91fd7cc3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:29:37.957174 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:37.957001 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-workload-socket" pod "742547a4-fbd7-40d3-a7a3-86fc91fd7cc3" (UID: "742547a4-fbd7-40d3-a7a3-86fc91fd7cc3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:37.957174 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:37.957133 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-istio-data" pod "742547a4-fbd7-40d3-a7a3-86fc91fd7cc3" (UID: "742547a4-fbd7-40d3-a7a3-86fc91fd7cc3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:37.957174 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:37.957143 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-credential-socket" pod "742547a4-fbd7-40d3-a7a3-86fc91fd7cc3" (UID: "742547a4-fbd7-40d3-a7a3-86fc91fd7cc3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:37.957686 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:37.957655 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:37.957809 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:37.957688 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:37.957809 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:37.957705 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:37.957809 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:37.957720 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:37.957809 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:37.957735 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:37.959089 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:37.959062 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-kube-api-access-wf5dk" pod "742547a4-fbd7-40d3-a7a3-86fc91fd7cc3" (UID: "742547a4-fbd7-40d3-a7a3-86fc91fd7cc3"). InnerVolumeSpecName "kube-api-access-wf5dk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:29:37.959213 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:37.959105 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-istio-podinfo" pod "742547a4-fbd7-40d3-a7a3-86fc91fd7cc3" (UID: "742547a4-fbd7-40d3-a7a3-86fc91fd7cc3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:29:37.959213 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:37.959105 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-istio-token" pod "742547a4-fbd7-40d3-a7a3-86fc91fd7cc3" (UID: "742547a4-fbd7-40d3-a7a3-86fc91fd7cc3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:29:37.959424 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:37.959400 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-istio-envoy" pod "742547a4-fbd7-40d3-a7a3-86fc91fd7cc3" (UID: "742547a4-fbd7-40d3-a7a3-86fc91fd7cc3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:38.058747 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:38.058715 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf5dk\" (UniqueName: \"kubernetes.io/projected/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-kube-api-access-wf5dk\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:38.058747 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:38.058744 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:38.058938 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:38.058755 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:38.058938 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:38.058764 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:38.360483 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:38.360438 4363 generic.go:356] "Generic (PLEG): container finished" podID="742547a4-fbd7-40d3-a7a3-86fc91fd7cc3" containerID="5dc6bfe92933edec7e596d52fece68e868ea7ca66853a0220f2a18209a7ec8ff" exitCode=0 Nov 01 16:29:38.360703 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:38.360572 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" event={"ID":"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3","Type":"ContainerDied","Data":"5dc6bfe92933edec7e596d52fece68e868ea7ca66853a0220f2a18209a7ec8ff"} Nov 01 16:29:38.360703 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:38.360667 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" event={"ID":"742547a4-fbd7-40d3-a7a3-86fc91fd7cc3","Type":"ContainerDied","Data":"3259ffae98371937cf65c77d2581fdbf86aeb4dc1310cac0c6e090ffbeafb5c4"} Nov 01 16:29:38.360703 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:38.360700 4363 scope.go:122] "RemoveContainer" containerID="5dc6bfe92933edec7e596d52fece68e868ea7ca66853a0220f2a18209a7ec8ff" Nov 01 16:29:38.360890 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:38.360873 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq" Nov 01 16:29:38.381175 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:38.381151 4363 scope.go:122] "RemoveContainer" containerID="2d271a9c18d6ff4575a67d700a8b05bbd8356e297a3de9b6e16758d5ff5242e2" Nov 01 16:29:38.395466 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:38.395438 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq"] Nov 01 16:29:38.396783 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:38.396759 4363 scope.go:122] "RemoveContainer" containerID="5dc6bfe92933edec7e596d52fece68e868ea7ca66853a0220f2a18209a7ec8ff" Nov 01 16:29:38.397202 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:29:38.397173 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dc6bfe92933edec7e596d52fece68e868ea7ca66853a0220f2a18209a7ec8ff\": container with ID starting with 5dc6bfe92933edec7e596d52fece68e868ea7ca66853a0220f2a18209a7ec8ff not found: ID does not exist" containerID="5dc6bfe92933edec7e596d52fece68e868ea7ca66853a0220f2a18209a7ec8ff" Nov 01 16:29:38.397328 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:38.397214 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dc6bfe92933edec7e596d52fece68e868ea7ca66853a0220f2a18209a7ec8ff"} err="failed to get container status \"5dc6bfe92933edec7e596d52fece68e868ea7ca66853a0220f2a18209a7ec8ff\": rpc error: code = NotFound desc = could not find container \"5dc6bfe92933edec7e596d52fece68e868ea7ca66853a0220f2a18209a7ec8ff\": container with ID starting with 5dc6bfe92933edec7e596d52fece68e868ea7ca66853a0220f2a18209a7ec8ff not found: ID does not exist" Nov 01 16:29:38.397328 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:38.397241 4363 scope.go:122] "RemoveContainer" containerID="2d271a9c18d6ff4575a67d700a8b05bbd8356e297a3de9b6e16758d5ff5242e2" Nov 01 16:29:38.397554 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:29:38.397529 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d271a9c18d6ff4575a67d700a8b05bbd8356e297a3de9b6e16758d5ff5242e2\": container with ID starting with 2d271a9c18d6ff4575a67d700a8b05bbd8356e297a3de9b6e16758d5ff5242e2 not found: ID does not exist" containerID="2d271a9c18d6ff4575a67d700a8b05bbd8356e297a3de9b6e16758d5ff5242e2" Nov 01 16:29:38.397639 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:38.397562 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d271a9c18d6ff4575a67d700a8b05bbd8356e297a3de9b6e16758d5ff5242e2"} err="failed to get container status \"2d271a9c18d6ff4575a67d700a8b05bbd8356e297a3de9b6e16758d5ff5242e2\": rpc error: code = NotFound desc = could not find container \"2d271a9c18d6ff4575a67d700a8b05bbd8356e297a3de9b6e16758d5ff5242e2\": container with ID starting with 2d271a9c18d6ff4575a67d700a8b05bbd8356e297a3de9b6e16758d5ff5242e2 not found: ID does not exist" Nov 01 16:29:38.400363 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:38.400342 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-7321d6-openshift-default-65fdb447-gv9jq"] Nov 01 16:29:38.843204 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:38.843177 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="742547a4-fbd7-40d3-a7a3-86fc91fd7cc3" path="/var/lib/kubelet/pods/742547a4-fbd7-40d3-a7a3-86fc91fd7cc3/volumes" Nov 01 16:29:43.137681 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:43.137643 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp"] Nov 01 16:29:43.361056 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:43.351118 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5e2d03-cbd9bb6f9-bbprx"] Nov 01 16:29:43.401837 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:43.399177 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5e2d03-757556fddc-j9gph"] Nov 01 16:29:44.212766 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:44.212731 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl"] Nov 01 16:29:44.213973 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:44.213946 4363 cpu_manager.go:414] "containerMap: removing container" podUID="eb7ee4b9-3e65-496f-b221-b6d29c10fa97" containerName="revoke-keys" Nov 01 16:29:44.213973 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:44.213972 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb7ee4b9-3e65-496f-b221-b6d29c10fa97" containerName="revoke-keys" Nov 01 16:29:44.214178 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:44.214002 4363 cpu_manager.go:414] "containerMap: removing container" podUID="742547a4-fbd7-40d3-a7a3-86fc91fd7cc3" containerName="istio-proxy" Nov 01 16:29:44.214178 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:44.214009 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="742547a4-fbd7-40d3-a7a3-86fc91fd7cc3" containerName="istio-proxy" Nov 01 16:29:44.214178 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:44.214072 4363 cpu_manager.go:414] "containerMap: removing container" podUID="742547a4-fbd7-40d3-a7a3-86fc91fd7cc3" containerName="istio-proxy" Nov 01 16:29:44.214178 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:44.214080 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="742547a4-fbd7-40d3-a7a3-86fc91fd7cc3" containerName="istio-proxy" Nov 01 16:29:44.214383 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:44.214277 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb7ee4b9-3e65-496f-b221-b6d29c10fa97" containerName="revoke-keys" Nov 01 16:29:44.214383 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:44.214293 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="742547a4-fbd7-40d3-a7a3-86fc91fd7cc3" containerName="istio-proxy" Nov 01 16:29:44.214383 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:44.214312 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="742547a4-fbd7-40d3-a7a3-86fc91fd7cc3" containerName="istio-proxy" Nov 01 16:29:44.219736 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:44.219714 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" Nov 01 16:29:44.226316 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:44.226289 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-028c46-openshift-default-dockercfg-8z222\"" Nov 01 16:29:44.231195 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:44.231171 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl"] Nov 01 16:29:44.369038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:44.367291 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64dcf\" (UniqueName: \"kubernetes.io/projected/f3655831-2619-45f1-9da3-5bbe7dca87f4-kube-api-access-64dcf\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl\" (UID: \"f3655831-2619-45f1-9da3-5bbe7dca87f4\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" Nov 01 16:29:44.369038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:44.367364 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3655831-2619-45f1-9da3-5bbe7dca87f4-workload-socket\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl\" (UID: \"f3655831-2619-45f1-9da3-5bbe7dca87f4\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" Nov 01 16:29:44.369038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:44.367479 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3655831-2619-45f1-9da3-5bbe7dca87f4-istio-podinfo\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl\" (UID: \"f3655831-2619-45f1-9da3-5bbe7dca87f4\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" Nov 01 16:29:44.369038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:44.367514 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3655831-2619-45f1-9da3-5bbe7dca87f4-istio-envoy\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl\" (UID: \"f3655831-2619-45f1-9da3-5bbe7dca87f4\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" Nov 01 16:29:44.369038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:44.367605 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3655831-2619-45f1-9da3-5bbe7dca87f4-istio-data\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl\" (UID: \"f3655831-2619-45f1-9da3-5bbe7dca87f4\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" Nov 01 16:29:44.369038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:44.367632 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3655831-2619-45f1-9da3-5bbe7dca87f4-workload-certs\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl\" (UID: \"f3655831-2619-45f1-9da3-5bbe7dca87f4\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" Nov 01 16:29:44.369038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:44.367698 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3655831-2619-45f1-9da3-5bbe7dca87f4-istio-token\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl\" (UID: \"f3655831-2619-45f1-9da3-5bbe7dca87f4\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" Nov 01 16:29:44.369038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:44.367735 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3655831-2619-45f1-9da3-5bbe7dca87f4-istiod-ca-cert\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl\" (UID: \"f3655831-2619-45f1-9da3-5bbe7dca87f4\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" Nov 01 16:29:44.369038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:44.367837 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3655831-2619-45f1-9da3-5bbe7dca87f4-credential-socket\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl\" (UID: \"f3655831-2619-45f1-9da3-5bbe7dca87f4\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" Nov 01 16:29:44.516091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:44.515293 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3655831-2619-45f1-9da3-5bbe7dca87f4-istiod-ca-cert\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl\" (UID: \"f3655831-2619-45f1-9da3-5bbe7dca87f4\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" Nov 01 16:29:44.516091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:44.515417 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3655831-2619-45f1-9da3-5bbe7dca87f4-credential-socket\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl\" (UID: \"f3655831-2619-45f1-9da3-5bbe7dca87f4\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" Nov 01 16:29:44.516091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:44.515524 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64dcf\" (UniqueName: \"kubernetes.io/projected/f3655831-2619-45f1-9da3-5bbe7dca87f4-kube-api-access-64dcf\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl\" (UID: \"f3655831-2619-45f1-9da3-5bbe7dca87f4\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" Nov 01 16:29:44.516091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:44.515572 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3655831-2619-45f1-9da3-5bbe7dca87f4-workload-socket\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl\" (UID: \"f3655831-2619-45f1-9da3-5bbe7dca87f4\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" Nov 01 16:29:44.516091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:44.515690 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3655831-2619-45f1-9da3-5bbe7dca87f4-istio-podinfo\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl\" (UID: \"f3655831-2619-45f1-9da3-5bbe7dca87f4\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" Nov 01 16:29:44.516091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:44.515730 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3655831-2619-45f1-9da3-5bbe7dca87f4-istio-envoy\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl\" (UID: \"f3655831-2619-45f1-9da3-5bbe7dca87f4\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" Nov 01 16:29:44.516091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:44.515825 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3655831-2619-45f1-9da3-5bbe7dca87f4-istio-data\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl\" (UID: \"f3655831-2619-45f1-9da3-5bbe7dca87f4\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" Nov 01 16:29:44.516091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:44.515856 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3655831-2619-45f1-9da3-5bbe7dca87f4-workload-certs\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl\" (UID: \"f3655831-2619-45f1-9da3-5bbe7dca87f4\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" Nov 01 16:29:44.516091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:44.515927 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3655831-2619-45f1-9da3-5bbe7dca87f4-istio-token\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl\" (UID: \"f3655831-2619-45f1-9da3-5bbe7dca87f4\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" Nov 01 16:29:44.536037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:44.517194 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3655831-2619-45f1-9da3-5bbe7dca87f4-istiod-ca-cert\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl\" (UID: \"f3655831-2619-45f1-9da3-5bbe7dca87f4\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" Nov 01 16:29:44.536037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:44.517496 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3655831-2619-45f1-9da3-5bbe7dca87f4-credential-socket\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl\" (UID: \"f3655831-2619-45f1-9da3-5bbe7dca87f4\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" Nov 01 16:29:44.536037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:44.517989 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3655831-2619-45f1-9da3-5bbe7dca87f4-workload-socket\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl\" (UID: \"f3655831-2619-45f1-9da3-5bbe7dca87f4\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" Nov 01 16:29:44.547092 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:44.544325 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3655831-2619-45f1-9da3-5bbe7dca87f4-workload-certs\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl\" (UID: \"f3655831-2619-45f1-9da3-5bbe7dca87f4\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" Nov 01 16:29:44.563031 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:44.561554 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3655831-2619-45f1-9da3-5bbe7dca87f4-istio-podinfo\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl\" (UID: \"f3655831-2619-45f1-9da3-5bbe7dca87f4\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" Nov 01 16:29:44.563031 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:44.561834 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3655831-2619-45f1-9da3-5bbe7dca87f4-istio-data\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl\" (UID: \"f3655831-2619-45f1-9da3-5bbe7dca87f4\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" Nov 01 16:29:44.582073 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:44.578979 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64dcf\" (UniqueName: \"kubernetes.io/projected/f3655831-2619-45f1-9da3-5bbe7dca87f4-kube-api-access-64dcf\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl\" (UID: \"f3655831-2619-45f1-9da3-5bbe7dca87f4\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" Nov 01 16:29:44.614046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:44.597778 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3655831-2619-45f1-9da3-5bbe7dca87f4-istio-token\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl\" (UID: \"f3655831-2619-45f1-9da3-5bbe7dca87f4\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" Nov 01 16:29:44.634656 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:44.630650 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3655831-2619-45f1-9da3-5bbe7dca87f4-istio-envoy\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl\" (UID: \"f3655831-2619-45f1-9da3-5bbe7dca87f4\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" Nov 01 16:29:44.843819 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:44.843791 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" Nov 01 16:29:45.166295 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:45.166267 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl"] Nov 01 16:29:45.166896 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:29:45.166873 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3655831_2619_45f1_9da3_5bbe7dca87f4.slice/crio-42bbde4f019d77e774ed377fb56efa8906f57dfb498a4ef45c5ae967729ab5e3 WatchSource:0}: Error finding container 42bbde4f019d77e774ed377fb56efa8906f57dfb498a4ef45c5ae967729ab5e3: Status 404 returned error can't find the container with id 42bbde4f019d77e774ed377fb56efa8906f57dfb498a4ef45c5ae967729ab5e3 Nov 01 16:29:45.169786 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:45.169744 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:29:45.169904 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:45.169830 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:29:45.169904 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:45.169873 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:29:45.627958 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:45.627920 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" event={"ID":"f3655831-2619-45f1-9da3-5bbe7dca87f4","Type":"ContainerStarted","Data":"1c053fc029571cf2587d21ea99f73ad2fdc6c84e9249f49578275898ea8d521e"} Nov 01 16:29:45.628388 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:45.627988 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" event={"ID":"f3655831-2619-45f1-9da3-5bbe7dca87f4","Type":"ContainerStarted","Data":"42bbde4f019d77e774ed377fb56efa8906f57dfb498a4ef45c5ae967729ab5e3"} Nov 01 16:29:45.649311 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:45.649259 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" podStartSLOduration=1.64924655 podStartE2EDuration="1.64924655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:29:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:29:45.64545396 +0000 UTC m=+2518.977191611" watchObservedRunningTime="2026-11-01 16:29:45.64924655 +0000 UTC m=+2518.980984203" Nov 01 16:29:45.845540 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:45.845114 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" Nov 01 16:29:46.851048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:46.850985 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" Nov 01 16:29:47.646855 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:47.646820 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" Nov 01 16:29:47.647695 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:47.647674 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" Nov 01 16:29:48.312998 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:29:48.312952 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:29:48.352157 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:48.352108 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5e2d03-cbd9bb6f9-bbprx" podUID="96c20c16-b48c-40b5-853c-edddd1225652" containerName="payload-processing" containerID="cri-o://5657a369b4632c5bc020f3a2169db572af33e2980861b959df88957d23a9f99d" gracePeriod=25 Nov 01 16:29:48.421945 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:48.421898 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5e2d03-757556fddc-j9gph" podUID="57dbe32d-78fb-4168-a5f8-5fa0a38f2e2b" containerName="payload-pre-processing" containerID="cri-o://44fdf0f6ebd17a0dd4f8dbc5a4b72bb1afe67bb9be6ce8ecd1f6f08822eaccf9" gracePeriod=25 Nov 01 16:29:49.052071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:49.052040 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-9c5644f64-xrw75"] Nov 01 16:29:49.057212 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:49.057185 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-9c5644f64-xrw75" Nov 01 16:29:49.071695 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:49.071663 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-9c5644f64-xrw75"] Nov 01 16:29:49.159831 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:49.159796 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3fd73f\"" Nov 01 16:29:49.160114 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:49.159960 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3fd73f-dockercfg-n5xdq\"" Nov 01 16:29:49.212288 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:49.212257 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d60400a5-9792-41ec-90c2-f1e76bee301b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3fd73f-9c5644f64-xrw75\" (UID: \"d60400a5-9792-41ec-90c2-f1e76bee301b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-9c5644f64-xrw75" Nov 01 16:29:49.212456 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:49.212329 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v9cg\" (UniqueName: \"kubernetes.io/projected/d60400a5-9792-41ec-90c2-f1e76bee301b-kube-api-access-6v9cg\") pod \"payload-processing-e2e-ipp-a-3fd73f-9c5644f64-xrw75\" (UID: \"d60400a5-9792-41ec-90c2-f1e76bee301b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-9c5644f64-xrw75" Nov 01 16:29:49.315209 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:49.315115 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d60400a5-9792-41ec-90c2-f1e76bee301b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3fd73f-9c5644f64-xrw75\" (UID: \"d60400a5-9792-41ec-90c2-f1e76bee301b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-9c5644f64-xrw75" Nov 01 16:29:49.316962 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:49.316185 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d60400a5-9792-41ec-90c2-f1e76bee301b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3fd73f-9c5644f64-xrw75\" (UID: \"d60400a5-9792-41ec-90c2-f1e76bee301b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-9c5644f64-xrw75" Nov 01 16:29:49.317420 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:49.317383 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v9cg\" (UniqueName: \"kubernetes.io/projected/d60400a5-9792-41ec-90c2-f1e76bee301b-kube-api-access-6v9cg\") pod \"payload-processing-e2e-ipp-a-3fd73f-9c5644f64-xrw75\" (UID: \"d60400a5-9792-41ec-90c2-f1e76bee301b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-9c5644f64-xrw75" Nov 01 16:29:49.336067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:49.336038 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v9cg\" (UniqueName: \"kubernetes.io/projected/d60400a5-9792-41ec-90c2-f1e76bee301b-kube-api-access-6v9cg\") pod \"payload-processing-e2e-ipp-a-3fd73f-9c5644f64-xrw75\" (UID: \"d60400a5-9792-41ec-90c2-f1e76bee301b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-9c5644f64-xrw75" Nov 01 16:29:49.451644 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:49.451608 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-9c5644f64-xrw75" Nov 01 16:29:50.010418 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:50.010380 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-9c5644f64-xrw75"] Nov 01 16:29:50.011193 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:29:50.011167 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd60400a5_9792_41ec_90c2_f1e76bee301b.slice/crio-aa66c5f735fc82cdfec95e70fef6434e6276464214d69e6f3803ee93eb097f41 WatchSource:0}: Error finding container aa66c5f735fc82cdfec95e70fef6434e6276464214d69e6f3803ee93eb097f41: Status 404 returned error can't find the container with id aa66c5f735fc82cdfec95e70fef6434e6276464214d69e6f3803ee93eb097f41 Nov 01 16:29:50.696252 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:50.696200 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-9c5644f64-xrw75" event={"ID":"d60400a5-9792-41ec-90c2-f1e76bee301b","Type":"ContainerStarted","Data":"2366b7adb832230aaabbaf349f114b32073a6512c4bd3e0253a1cf6ed09cdc8d"} Nov 01 16:29:50.696717 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:50.696279 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-9c5644f64-xrw75" event={"ID":"d60400a5-9792-41ec-90c2-f1e76bee301b","Type":"ContainerStarted","Data":"aa66c5f735fc82cdfec95e70fef6434e6276464214d69e6f3803ee93eb097f41"} Nov 01 16:29:50.696717 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:50.696311 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-9c5644f64-xrw75" Nov 01 16:29:50.717114 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:50.717071 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-9c5644f64-xrw75" podStartSLOduration=1.717056629 podStartE2EDuration="1.717056629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:29:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:29:50.711319642 +0000 UTC m=+2524.043057292" watchObservedRunningTime="2026-11-01 16:29:50.717056629 +0000 UTC m=+2524.048794280" Nov 01 16:29:54.811911 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:54.811863 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5e2d03-cbd9bb6f9-bbprx" podUID="96c20c16-b48c-40b5-853c-edddd1225652" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:29:54.818882 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:54.818834 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5e2d03-757556fddc-j9gph" podUID="57dbe32d-78fb-4168-a5f8-5fa0a38f2e2b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:29:57.346209 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:57.346171 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5e2d03-86f8cf979c-xd2rs"] Nov 01 16:29:57.346847 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:57.346410 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5e2d03-86f8cf979c-xd2rs" podUID="0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18" containerName="maas-api" containerID="cri-o://b23b7d28d16d49e60b83cb8498611689b32f6c1190f9fbe0a85ae63c67d15945" gracePeriod=30 Nov 01 16:29:57.382962 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:29:57.382931 4363 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e9b36f1_f3d1_4efa_a9fe_317d57f0fe18.slice/crio-b23b7d28d16d49e60b83cb8498611689b32f6c1190f9fbe0a85ae63c67d15945.scope\": RecentStats: unable to find data in memory cache]" Nov 01 16:29:57.943888 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:57.943860 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5e2d03-pw25b"] Nov 01 16:29:57.944225 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:57.944202 4363 generic.go:356] "Generic (PLEG): container finished" podID="0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18" containerID="b23b7d28d16d49e60b83cb8498611689b32f6c1190f9fbe0a85ae63c67d15945" exitCode=0 Nov 01 16:29:57.944449 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:57.944350 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5e2d03-86f8cf979c-xd2rs" event={"ID":"0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18","Type":"ContainerDied","Data":"b23b7d28d16d49e60b83cb8498611689b32f6c1190f9fbe0a85ae63c67d15945"} Nov 01 16:29:57.981040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:57.975503 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5e2d03-86f8cf979c-xd2rs" Nov 01 16:29:58.014701 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:58.014671 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5e2d03-pw25b"] Nov 01 16:29:58.036857 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:58.036824 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18-openshift-service-ca\") pod \"0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18\" (UID: \"0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18\") " Nov 01 16:29:58.037090 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:58.036890 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18-kube-api-access-xmfr6\" (UniqueName: \"kubernetes.io/projected/0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18-kube-api-access-xmfr6\") pod \"0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18\" (UID: \"0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18\") " Nov 01 16:29:58.037090 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:58.036984 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18-maas-api-tls\") pod \"0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18\" (UID: \"0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18\") " Nov 01 16:29:58.037179 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:58.037160 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18-metrics-certs\") pod \"0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18\" (UID: \"0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18\") " Nov 01 16:29:58.037256 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:58.037230 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18-openshift-service-ca" pod "0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18" (UID: "0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:29:58.038210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:58.037801 4363 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18-openshift-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:58.039399 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:58.039372 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18-metrics-certs" pod "0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18" (UID: "0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:29:58.039504 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:58.039393 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18-maas-api-tls" pod "0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18" (UID: "0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:29:58.039774 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:58.039759 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18-kube-api-access-xmfr6" pod "0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18" (UID: "0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18"). InnerVolumeSpecName "kube-api-access-xmfr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:29:58.139440 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:58.139401 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmfr6\" (UniqueName: \"kubernetes.io/projected/0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18-kube-api-access-xmfr6\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:58.139440 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:58.139434 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18-maas-api-tls\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:58.139440 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:58.139443 4363 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18-metrics-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:58.167767 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:58.167612 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp" podUID="f8ca3021-8cbd-4cfc-aab7-33bb656f4c01" containerName="main" containerID="cri-o://1f83e3929c29391804dfd6954867e2dd6cdf5ab8f08d70e2ee812dd3f5a51446" gracePeriod=45 Nov 01 16:29:58.465900 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:58.465866 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp" Nov 01 16:29:58.546200 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:58.546167 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8ca3021-8cbd-4cfc-aab7-33bb656f4c01-tls-certs\" (UniqueName: \"kubernetes.io/secret/f8ca3021-8cbd-4cfc-aab7-33bb656f4c01-tls-certs\") pod \"f8ca3021-8cbd-4cfc-aab7-33bb656f4c01\" (UID: \"f8ca3021-8cbd-4cfc-aab7-33bb656f4c01\") " Nov 01 16:29:58.546403 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:58.546322 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8ca3021-8cbd-4cfc-aab7-33bb656f4c01-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8ca3021-8cbd-4cfc-aab7-33bb656f4c01-tmp-dir\") pod \"f8ca3021-8cbd-4cfc-aab7-33bb656f4c01\" (UID: \"f8ca3021-8cbd-4cfc-aab7-33bb656f4c01\") " Nov 01 16:29:58.546759 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:58.546401 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8ca3021-8cbd-4cfc-aab7-33bb656f4c01-home\" (UniqueName: \"kubernetes.io/empty-dir/f8ca3021-8cbd-4cfc-aab7-33bb656f4c01-home\") pod \"f8ca3021-8cbd-4cfc-aab7-33bb656f4c01\" (UID: \"f8ca3021-8cbd-4cfc-aab7-33bb656f4c01\") " Nov 01 16:29:58.546759 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:58.546456 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8ca3021-8cbd-4cfc-aab7-33bb656f4c01-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8ca3021-8cbd-4cfc-aab7-33bb656f4c01-dshm\") pod \"f8ca3021-8cbd-4cfc-aab7-33bb656f4c01\" (UID: \"f8ca3021-8cbd-4cfc-aab7-33bb656f4c01\") " Nov 01 16:29:58.546759 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:58.546508 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8ca3021-8cbd-4cfc-aab7-33bb656f4c01-kube-api-access-g66mz\" (UniqueName: \"kubernetes.io/projected/f8ca3021-8cbd-4cfc-aab7-33bb656f4c01-kube-api-access-g66mz\") pod \"f8ca3021-8cbd-4cfc-aab7-33bb656f4c01\" (UID: \"f8ca3021-8cbd-4cfc-aab7-33bb656f4c01\") " Nov 01 16:29:58.546759 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:58.546563 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8ca3021-8cbd-4cfc-aab7-33bb656f4c01-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8ca3021-8cbd-4cfc-aab7-33bb656f4c01-model-cache\") pod \"f8ca3021-8cbd-4cfc-aab7-33bb656f4c01\" (UID: \"f8ca3021-8cbd-4cfc-aab7-33bb656f4c01\") " Nov 01 16:29:58.548858 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:58.546618 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ca3021-8cbd-4cfc-aab7-33bb656f4c01-tmp-dir" pod "f8ca3021-8cbd-4cfc-aab7-33bb656f4c01" (UID: "f8ca3021-8cbd-4cfc-aab7-33bb656f4c01"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:58.548858 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:58.546645 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ca3021-8cbd-4cfc-aab7-33bb656f4c01-home" pod "f8ca3021-8cbd-4cfc-aab7-33bb656f4c01" (UID: "f8ca3021-8cbd-4cfc-aab7-33bb656f4c01"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:58.548858 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:58.547141 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ca3021-8cbd-4cfc-aab7-33bb656f4c01-model-cache" pod "f8ca3021-8cbd-4cfc-aab7-33bb656f4c01" (UID: "f8ca3021-8cbd-4cfc-aab7-33bb656f4c01"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:58.550131 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:58.549714 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8ca3021-8cbd-4cfc-aab7-33bb656f4c01-tls-certs" pod "f8ca3021-8cbd-4cfc-aab7-33bb656f4c01" (UID: "f8ca3021-8cbd-4cfc-aab7-33bb656f4c01"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:29:58.554029 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:58.552525 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8ca3021-8cbd-4cfc-aab7-33bb656f4c01-kube-api-access-g66mz" pod "f8ca3021-8cbd-4cfc-aab7-33bb656f4c01" (UID: "f8ca3021-8cbd-4cfc-aab7-33bb656f4c01"). InnerVolumeSpecName "kube-api-access-g66mz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:29:58.554029 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:58.552678 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ca3021-8cbd-4cfc-aab7-33bb656f4c01-dshm" pod "f8ca3021-8cbd-4cfc-aab7-33bb656f4c01" (UID: "f8ca3021-8cbd-4cfc-aab7-33bb656f4c01"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:58.649215 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:58.649185 4363 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8ca3021-8cbd-4cfc-aab7-33bb656f4c01-tmp-dir\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:58.649215 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:58.649212 4363 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8ca3021-8cbd-4cfc-aab7-33bb656f4c01-home\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:58.649215 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:58.649220 4363 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8ca3021-8cbd-4cfc-aab7-33bb656f4c01-dshm\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:58.649468 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:58.649228 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g66mz\" (UniqueName: \"kubernetes.io/projected/f8ca3021-8cbd-4cfc-aab7-33bb656f4c01-kube-api-access-g66mz\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:58.649468 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:58.649239 4363 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8ca3021-8cbd-4cfc-aab7-33bb656f4c01-model-cache\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:58.649468 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:58.649269 4363 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8ca3021-8cbd-4cfc-aab7-33bb656f4c01-tls-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:58.801996 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:58.801912 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb7ee4b9-3e65-496f-b221-b6d29c10fa97" path="/var/lib/kubelet/pods/eb7ee4b9-3e65-496f-b221-b6d29c10fa97/volumes" Nov 01 16:29:58.985885 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:58.985847 4363 generic.go:356] "Generic (PLEG): container finished" podID="f8ca3021-8cbd-4cfc-aab7-33bb656f4c01" containerID="1f83e3929c29391804dfd6954867e2dd6cdf5ab8f08d70e2ee812dd3f5a51446" exitCode=0 Nov 01 16:29:58.986130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:58.985999 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp" event={"ID":"f8ca3021-8cbd-4cfc-aab7-33bb656f4c01","Type":"ContainerDied","Data":"1f83e3929c29391804dfd6954867e2dd6cdf5ab8f08d70e2ee812dd3f5a51446"} Nov 01 16:29:58.986130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:58.986084 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp" event={"ID":"f8ca3021-8cbd-4cfc-aab7-33bb656f4c01","Type":"ContainerDied","Data":"8916bb946ec624f1b7024d6451ca0cacbd7d9d12fcdef0eccb8dd2d19e6d3b40"} Nov 01 16:29:58.986130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:58.986107 4363 scope.go:122] "RemoveContainer" containerID="1f83e3929c29391804dfd6954867e2dd6cdf5ab8f08d70e2ee812dd3f5a51446" Nov 01 16:29:58.986345 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:58.986270 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp" Nov 01 16:29:58.989589 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:58.989558 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5e2d03-86f8cf979c-xd2rs" event={"ID":"0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18","Type":"ContainerDied","Data":"19acf69af6482201b16ac4de8a634f70646839fa91db5790849b58dad484682b"} Nov 01 16:29:58.989708 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:58.989661 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5e2d03-86f8cf979c-xd2rs" Nov 01 16:29:59.006828 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:59.006802 4363 scope.go:122] "RemoveContainer" containerID="1f83e3929c29391804dfd6954867e2dd6cdf5ab8f08d70e2ee812dd3f5a51446" Nov 01 16:29:59.007195 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:29:59.007170 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f83e3929c29391804dfd6954867e2dd6cdf5ab8f08d70e2ee812dd3f5a51446\": container with ID starting with 1f83e3929c29391804dfd6954867e2dd6cdf5ab8f08d70e2ee812dd3f5a51446 not found: ID does not exist" containerID="1f83e3929c29391804dfd6954867e2dd6cdf5ab8f08d70e2ee812dd3f5a51446" Nov 01 16:29:59.007267 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:59.007206 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f83e3929c29391804dfd6954867e2dd6cdf5ab8f08d70e2ee812dd3f5a51446"} err="failed to get container status \"1f83e3929c29391804dfd6954867e2dd6cdf5ab8f08d70e2ee812dd3f5a51446\": rpc error: code = NotFound desc = could not find container \"1f83e3929c29391804dfd6954867e2dd6cdf5ab8f08d70e2ee812dd3f5a51446\": container with ID starting with 1f83e3929c29391804dfd6954867e2dd6cdf5ab8f08d70e2ee812dd3f5a51446 not found: ID does not exist" Nov 01 16:29:59.007267 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:59.007229 4363 scope.go:122] "RemoveContainer" containerID="b23b7d28d16d49e60b83cb8498611689b32f6c1190f9fbe0a85ae63c67d15945" Nov 01 16:29:59.016780 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:59.016751 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp"] Nov 01 16:29:59.023736 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:59.023711 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp"] Nov 01 16:29:59.034852 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:59.034827 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5e2d03-86f8cf979c-xd2rs"] Nov 01 16:29:59.042695 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:59.042662 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5e2d03-86f8cf979c-xd2rs"] Nov 01 16:29:59.453089 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:29:59.453002 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-c8537c/auth-test-model-c8537c-kserve-5859bdfb6f-kblvp" podUID="f8ca3021-8cbd-4cfc-aab7-33bb656f4c01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.225:8000/ready\": context deadline exceeded" Nov 01 16:30:00.139265 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.139228 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2b94b5-29892510-9n2h2"] Nov 01 16:30:00.139967 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.139952 4363 cpu_manager.go:414] "containerMap: removing container" podUID="0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18" containerName="maas-api" Nov 01 16:30:00.140010 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.139968 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18" containerName="maas-api" Nov 01 16:30:00.140010 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.139986 4363 cpu_manager.go:414] "containerMap: removing container" podUID="f8ca3021-8cbd-4cfc-aab7-33bb656f4c01" containerName="main" Nov 01 16:30:00.140010 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.139991 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8ca3021-8cbd-4cfc-aab7-33bb656f4c01" containerName="main" Nov 01 16:30:00.140161 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.140151 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18" containerName="maas-api" Nov 01 16:30:00.140202 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.140163 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8ca3021-8cbd-4cfc-aab7-33bb656f4c01" containerName="main" Nov 01 16:30:00.145344 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.145319 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2b94b5-29892510-9n2h2" Nov 01 16:30:00.149118 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.149094 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5ccd7b-29892510-9v7dk"] Nov 01 16:30:00.154278 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.154250 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:30:00.155334 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.155316 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c8537c-29892510-6c55f"] Nov 01 16:30:00.155987 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.155967 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5ccd7b-29892510-9v7dk" Nov 01 16:30:00.160874 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.160853 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2b94b5-29892510-9n2h2"] Nov 01 16:30:00.164068 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.161224 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c8537c-29892510-6c55f" Nov 01 16:30:00.167766 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.167716 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c8537c-29892510-6c55f"] Nov 01 16:30:00.176254 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.174894 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5ccd7b-29892510-9v7dk"] Nov 01 16:30:00.189129 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.189102 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892510-t259f"] Nov 01 16:30:00.191241 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.191197 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbzq9\" (UniqueName: \"kubernetes.io/projected/975c2d1d-45a8-4439-923e-59f70d69ea04-kube-api-access-fbzq9\") pod \"maas-api-key-cleanup-e2e-ipp-b-2b94b5-29892510-9n2h2\" (UID: \"975c2d1d-45a8-4439-923e-59f70d69ea04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2b94b5-29892510-9n2h2" Nov 01 16:30:00.191379 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.191288 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89cbf816-3afe-455a-8e90-5606573018b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5ccd7b-29892510-9v7dk\" (UID: \"89cbf816-3afe-455a-8e90-5606573018b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5ccd7b-29892510-9v7dk" Nov 01 16:30:00.191477 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.191460 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/975c2d1d-45a8-4439-923e-59f70d69ea04-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-2b94b5-29892510-9n2h2\" (UID: \"975c2d1d-45a8-4439-923e-59f70d69ea04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2b94b5-29892510-9n2h2" Nov 01 16:30:00.191668 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.191651 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l282\" (UniqueName: \"kubernetes.io/projected/89cbf816-3afe-455a-8e90-5606573018b0-kube-api-access-4l282\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5ccd7b-29892510-9v7dk\" (UID: \"89cbf816-3afe-455a-8e90-5606573018b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5ccd7b-29892510-9v7dk" Nov 01 16:30:00.199842 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.199819 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892510-t259f" Nov 01 16:30:00.214821 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.213973 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892510-947xq"] Nov 01 16:30:00.229175 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.222503 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892510-947xq" Nov 01 16:30:00.232565 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.232531 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892510-t259f"] Nov 01 16:30:00.237545 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.237523 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892510-947xq"] Nov 01 16:30:00.249728 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.249701 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892510-zjcr9"] Nov 01 16:30:00.259287 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.257988 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892510-n7htt"] Nov 01 16:30:00.259287 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.258097 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892510-zjcr9" Nov 01 16:30:00.273176 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.273151 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892510-zjcr9"] Nov 01 16:30:00.273291 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.273270 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892510-n7htt" Nov 01 16:30:00.277923 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.277901 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892510-cg9fh"] Nov 01 16:30:00.286308 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.285102 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892510-cg9fh" Nov 01 16:30:00.295237 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.293622 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/975c2d1d-45a8-4439-923e-59f70d69ea04-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-2b94b5-29892510-9n2h2\" (UID: \"975c2d1d-45a8-4439-923e-59f70d69ea04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2b94b5-29892510-9n2h2" Nov 01 16:30:00.295237 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.293758 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l282\" (UniqueName: \"kubernetes.io/projected/89cbf816-3afe-455a-8e90-5606573018b0-kube-api-access-4l282\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5ccd7b-29892510-9v7dk\" (UID: \"89cbf816-3afe-455a-8e90-5606573018b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5ccd7b-29892510-9v7dk" Nov 01 16:30:00.295237 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.293801 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t8pv\" (UniqueName: \"kubernetes.io/projected/07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d-kube-api-access-8t8pv\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892510-947xq\" (UID: \"07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892510-947xq" Nov 01 16:30:00.295237 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.293872 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-c8537c-29892510-6c55f\" (UID: \"bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c8537c-29892510-6c55f" Nov 01 16:30:00.295237 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.293903 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpjcm\" (UniqueName: \"kubernetes.io/projected/bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68-kube-api-access-jpjcm\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-c8537c-29892510-6c55f\" (UID: \"bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c8537c-29892510-6c55f" Nov 01 16:30:00.295237 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.293938 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsbp2\" (UniqueName: \"kubernetes.io/projected/954da573-4dff-4a25-a180-17fe21a92841-kube-api-access-wsbp2\") pod \"maas-api-key-cleanup-29892510-zjcr9\" (UID: \"954da573-4dff-4a25-a180-17fe21a92841\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892510-zjcr9" Nov 01 16:30:00.295237 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.293987 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbzq9\" (UniqueName: \"kubernetes.io/projected/975c2d1d-45a8-4439-923e-59f70d69ea04-kube-api-access-fbzq9\") pod \"maas-api-key-cleanup-e2e-ipp-b-2b94b5-29892510-9n2h2\" (UID: \"975c2d1d-45a8-4439-923e-59f70d69ea04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2b94b5-29892510-9n2h2" Nov 01 16:30:00.295237 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.294085 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89cbf816-3afe-455a-8e90-5606573018b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5ccd7b-29892510-9v7dk\" (UID: \"89cbf816-3afe-455a-8e90-5606573018b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5ccd7b-29892510-9v7dk" Nov 01 16:30:00.295237 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.294782 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/975c2d1d-45a8-4439-923e-59f70d69ea04-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-2b94b5-29892510-9n2h2\" (UID: \"975c2d1d-45a8-4439-923e-59f70d69ea04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2b94b5-29892510-9n2h2" Nov 01 16:30:00.295237 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.294879 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/954da573-4dff-4a25-a180-17fe21a92841-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892510-zjcr9\" (UID: \"954da573-4dff-4a25-a180-17fe21a92841\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892510-zjcr9" Nov 01 16:30:00.295237 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.294928 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa25d700-0231-4655-a067-4d637ea4c5c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892510-t259f\" (UID: \"aa25d700-0231-4655-a067-4d637ea4c5c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892510-t259f" Nov 01 16:30:00.295237 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.295008 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w72j4\" (UniqueName: \"kubernetes.io/projected/aa25d700-0231-4655-a067-4d637ea4c5c0-kube-api-access-w72j4\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892510-t259f\" (UID: \"aa25d700-0231-4655-a067-4d637ea4c5c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892510-t259f" Nov 01 16:30:00.295237 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.295100 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892510-947xq\" (UID: \"07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892510-947xq" Nov 01 16:30:00.295237 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.295099 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89cbf816-3afe-455a-8e90-5606573018b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5ccd7b-29892510-9v7dk\" (UID: \"89cbf816-3afe-455a-8e90-5606573018b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5ccd7b-29892510-9v7dk" Nov 01 16:30:00.296928 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.296908 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fd73f-29892510-768f5"] Nov 01 16:30:00.309027 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.308877 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fd73f-29892510-768f5" Nov 01 16:30:00.309774 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.309357 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892510-cg9fh"] Nov 01 16:30:00.314434 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.310327 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892510-n7htt"] Nov 01 16:30:00.314434 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.311853 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbzq9\" (UniqueName: \"kubernetes.io/projected/975c2d1d-45a8-4439-923e-59f70d69ea04-kube-api-access-fbzq9\") pod \"maas-api-key-cleanup-e2e-ipp-b-2b94b5-29892510-9n2h2\" (UID: \"975c2d1d-45a8-4439-923e-59f70d69ea04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2b94b5-29892510-9n2h2" Nov 01 16:30:00.318066 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.317294 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l282\" (UniqueName: \"kubernetes.io/projected/89cbf816-3afe-455a-8e90-5606573018b0-kube-api-access-4l282\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5ccd7b-29892510-9v7dk\" (UID: \"89cbf816-3afe-455a-8e90-5606573018b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5ccd7b-29892510-9v7dk" Nov 01 16:30:00.323575 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.323553 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fd73f-29892510-768f5"] Nov 01 16:30:00.397178 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.397108 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1817f9a5-677f-43b6-a8d0-b80a92870530-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-3fd73f-29892510-768f5\" (UID: \"1817f9a5-677f-43b6-a8d0-b80a92870530\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fd73f-29892510-768f5" Nov 01 16:30:00.397360 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.397180 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t8pv\" (UniqueName: \"kubernetes.io/projected/07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d-kube-api-access-8t8pv\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892510-947xq\" (UID: \"07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892510-947xq" Nov 01 16:30:00.397360 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.397251 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w85dl\" (UniqueName: \"kubernetes.io/projected/cfb264a2-0df1-4b83-bd6d-a355f75a2aca-kube-api-access-w85dl\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892510-cg9fh\" (UID: \"cfb264a2-0df1-4b83-bd6d-a355f75a2aca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892510-cg9fh" Nov 01 16:30:00.397633 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.397379 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-c8537c-29892510-6c55f\" (UID: \"bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c8537c-29892510-6c55f" Nov 01 16:30:00.397633 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.397416 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpjcm\" (UniqueName: \"kubernetes.io/projected/bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68-kube-api-access-jpjcm\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-c8537c-29892510-6c55f\" (UID: \"bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c8537c-29892510-6c55f" Nov 01 16:30:00.397633 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.397448 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsbp2\" (UniqueName: \"kubernetes.io/projected/954da573-4dff-4a25-a180-17fe21a92841-kube-api-access-wsbp2\") pod \"maas-api-key-cleanup-29892510-zjcr9\" (UID: \"954da573-4dff-4a25-a180-17fe21a92841\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892510-zjcr9" Nov 01 16:30:00.397633 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.397499 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvvkl\" (UniqueName: \"kubernetes.io/projected/4e98f9e8-2337-406a-807c-d20dc90a35a2-kube-api-access-xvvkl\") pod \"maas-api-key-cleanup-e2e-worker-w6-43fac8-29892510-n7htt\" (UID: \"4e98f9e8-2337-406a-807c-d20dc90a35a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892510-n7htt" Nov 01 16:30:00.397633 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.397567 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p8bl\" (UniqueName: \"kubernetes.io/projected/1817f9a5-677f-43b6-a8d0-b80a92870530-kube-api-access-7p8bl\") pod \"maas-api-key-cleanup-e2e-ipp-a-3fd73f-29892510-768f5\" (UID: \"1817f9a5-677f-43b6-a8d0-b80a92870530\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fd73f-29892510-768f5" Nov 01 16:30:00.397633 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.397609 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/954da573-4dff-4a25-a180-17fe21a92841-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892510-zjcr9\" (UID: \"954da573-4dff-4a25-a180-17fe21a92841\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892510-zjcr9" Nov 01 16:30:00.398140 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.397642 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa25d700-0231-4655-a067-4d637ea4c5c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892510-t259f\" (UID: \"aa25d700-0231-4655-a067-4d637ea4c5c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892510-t259f" Nov 01 16:30:00.398140 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.397842 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w72j4\" (UniqueName: \"kubernetes.io/projected/aa25d700-0231-4655-a067-4d637ea4c5c0-kube-api-access-w72j4\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892510-t259f\" (UID: \"aa25d700-0231-4655-a067-4d637ea4c5c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892510-t259f" Nov 01 16:30:00.398140 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.397900 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e98f9e8-2337-406a-807c-d20dc90a35a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-43fac8-29892510-n7htt\" (UID: \"4e98f9e8-2337-406a-807c-d20dc90a35a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892510-n7htt" Nov 01 16:30:00.398140 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.397951 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892510-947xq\" (UID: \"07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892510-947xq" Nov 01 16:30:00.398140 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.398105 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb264a2-0df1-4b83-bd6d-a355f75a2aca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892510-cg9fh\" (UID: \"cfb264a2-0df1-4b83-bd6d-a355f75a2aca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892510-cg9fh" Nov 01 16:30:00.398402 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.398194 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-c8537c-29892510-6c55f\" (UID: \"bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c8537c-29892510-6c55f" Nov 01 16:30:00.398402 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.398365 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/954da573-4dff-4a25-a180-17fe21a92841-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892510-zjcr9\" (UID: \"954da573-4dff-4a25-a180-17fe21a92841\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892510-zjcr9" Nov 01 16:30:00.398982 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.398964 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892510-947xq\" (UID: \"07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892510-947xq" Nov 01 16:30:00.399081 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.398987 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa25d700-0231-4655-a067-4d637ea4c5c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892510-t259f\" (UID: \"aa25d700-0231-4655-a067-4d637ea4c5c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892510-t259f" Nov 01 16:30:00.406503 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.406479 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t8pv\" (UniqueName: \"kubernetes.io/projected/07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d-kube-api-access-8t8pv\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892510-947xq\" (UID: \"07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892510-947xq" Nov 01 16:30:00.406620 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.406513 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpjcm\" (UniqueName: \"kubernetes.io/projected/bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68-kube-api-access-jpjcm\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-c8537c-29892510-6c55f\" (UID: \"bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c8537c-29892510-6c55f" Nov 01 16:30:00.406770 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.406748 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsbp2\" (UniqueName: \"kubernetes.io/projected/954da573-4dff-4a25-a180-17fe21a92841-kube-api-access-wsbp2\") pod \"maas-api-key-cleanup-29892510-zjcr9\" (UID: \"954da573-4dff-4a25-a180-17fe21a92841\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892510-zjcr9" Nov 01 16:30:00.407304 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.407283 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w72j4\" (UniqueName: \"kubernetes.io/projected/aa25d700-0231-4655-a067-4d637ea4c5c0-kube-api-access-w72j4\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892510-t259f\" (UID: \"aa25d700-0231-4655-a067-4d637ea4c5c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892510-t259f" Nov 01 16:30:00.464219 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.464196 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2b94b5-29892510-9n2h2" Nov 01 16:30:00.488788 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.488763 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5ccd7b-29892510-9v7dk" Nov 01 16:30:00.500011 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.499965 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb264a2-0df1-4b83-bd6d-a355f75a2aca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892510-cg9fh\" (UID: \"cfb264a2-0df1-4b83-bd6d-a355f75a2aca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892510-cg9fh" Nov 01 16:30:00.500185 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.500092 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1817f9a5-677f-43b6-a8d0-b80a92870530-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-3fd73f-29892510-768f5\" (UID: \"1817f9a5-677f-43b6-a8d0-b80a92870530\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fd73f-29892510-768f5" Nov 01 16:30:00.500185 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.500170 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w85dl\" (UniqueName: \"kubernetes.io/projected/cfb264a2-0df1-4b83-bd6d-a355f75a2aca-kube-api-access-w85dl\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892510-cg9fh\" (UID: \"cfb264a2-0df1-4b83-bd6d-a355f75a2aca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892510-cg9fh" Nov 01 16:30:00.500341 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.500277 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvvkl\" (UniqueName: \"kubernetes.io/projected/4e98f9e8-2337-406a-807c-d20dc90a35a2-kube-api-access-xvvkl\") pod \"maas-api-key-cleanup-e2e-worker-w6-43fac8-29892510-n7htt\" (UID: \"4e98f9e8-2337-406a-807c-d20dc90a35a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892510-n7htt" Nov 01 16:30:00.500404 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.500357 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p8bl\" (UniqueName: \"kubernetes.io/projected/1817f9a5-677f-43b6-a8d0-b80a92870530-kube-api-access-7p8bl\") pod \"maas-api-key-cleanup-e2e-ipp-a-3fd73f-29892510-768f5\" (UID: \"1817f9a5-677f-43b6-a8d0-b80a92870530\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fd73f-29892510-768f5" Nov 01 16:30:00.500583 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.500559 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e98f9e8-2337-406a-807c-d20dc90a35a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-43fac8-29892510-n7htt\" (UID: \"4e98f9e8-2337-406a-807c-d20dc90a35a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892510-n7htt" Nov 01 16:30:00.501702 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.500896 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1817f9a5-677f-43b6-a8d0-b80a92870530-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-3fd73f-29892510-768f5\" (UID: \"1817f9a5-677f-43b6-a8d0-b80a92870530\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fd73f-29892510-768f5" Nov 01 16:30:00.501702 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.500901 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb264a2-0df1-4b83-bd6d-a355f75a2aca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892510-cg9fh\" (UID: \"cfb264a2-0df1-4b83-bd6d-a355f75a2aca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892510-cg9fh" Nov 01 16:30:00.501702 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.501554 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e98f9e8-2337-406a-807c-d20dc90a35a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-43fac8-29892510-n7htt\" (UID: \"4e98f9e8-2337-406a-807c-d20dc90a35a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892510-n7htt" Nov 01 16:30:00.505439 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.505416 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c8537c-29892510-6c55f" Nov 01 16:30:00.513075 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.510843 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p8bl\" (UniqueName: \"kubernetes.io/projected/1817f9a5-677f-43b6-a8d0-b80a92870530-kube-api-access-7p8bl\") pod \"maas-api-key-cleanup-e2e-ipp-a-3fd73f-29892510-768f5\" (UID: \"1817f9a5-677f-43b6-a8d0-b80a92870530\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fd73f-29892510-768f5" Nov 01 16:30:00.514478 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.514319 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvvkl\" (UniqueName: \"kubernetes.io/projected/4e98f9e8-2337-406a-807c-d20dc90a35a2-kube-api-access-xvvkl\") pod \"maas-api-key-cleanup-e2e-worker-w6-43fac8-29892510-n7htt\" (UID: \"4e98f9e8-2337-406a-807c-d20dc90a35a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892510-n7htt" Nov 01 16:30:00.515249 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.515198 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w85dl\" (UniqueName: \"kubernetes.io/projected/cfb264a2-0df1-4b83-bd6d-a355f75a2aca-kube-api-access-w85dl\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892510-cg9fh\" (UID: \"cfb264a2-0df1-4b83-bd6d-a355f75a2aca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892510-cg9fh" Nov 01 16:30:00.535127 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.532780 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892510-t259f" Nov 01 16:30:00.551328 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.551071 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892510-947xq" Nov 01 16:30:00.588876 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.588716 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892510-zjcr9" Nov 01 16:30:00.611228 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.610413 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892510-n7htt" Nov 01 16:30:00.715110 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.714399 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892510-cg9fh" Nov 01 16:30:00.728090 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.728056 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fd73f-29892510-768f5" Nov 01 16:30:00.821860 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.821824 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18" path="/var/lib/kubelet/pods/0e9b36f1-f3d1-4efa-a9fe-317d57f0fe18/volumes" Nov 01 16:30:00.822846 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.822820 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8ca3021-8cbd-4cfc-aab7-33bb656f4c01" path="/var/lib/kubelet/pods/f8ca3021-8cbd-4cfc-aab7-33bb656f4c01/volumes" Nov 01 16:30:00.964334 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:00.964288 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2b94b5-29892510-9n2h2"] Nov 01 16:30:01.015294 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:01.015269 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5ccd7b-29892510-9v7dk"] Nov 01 16:30:01.015897 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:30:01.015875 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89cbf816_3afe_455a_8e90_5606573018b0.slice/crio-27d79b5cf0af22eb737704085c51e5204d909528517d17f1cd3966802eb380a8 WatchSource:0}: Error finding container 27d79b5cf0af22eb737704085c51e5204d909528517d17f1cd3966802eb380a8: Status 404 returned error can't find the container with id 27d79b5cf0af22eb737704085c51e5204d909528517d17f1cd3966802eb380a8 Nov 01 16:30:01.026462 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:01.026431 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2b94b5-29892510-9n2h2" event={"ID":"975c2d1d-45a8-4439-923e-59f70d69ea04","Type":"ContainerStarted","Data":"a18ba1bcb3a65bb076ca84b0c6ab6d2ab6cd8cbc786de9a9de7438c07a262cf7"} Nov 01 16:30:01.117794 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:01.117764 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c8537c-29892510-6c55f"] Nov 01 16:30:01.235195 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:30:01.235169 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa25d700_0231_4655_a067_4d637ea4c5c0.slice/crio-3aa4c52fa7004be9dbe1ed60dc671eb85f07ec9b7bbd486f3ca55db8b721ee4a WatchSource:0}: Error finding container 3aa4c52fa7004be9dbe1ed60dc671eb85f07ec9b7bbd486f3ca55db8b721ee4a: Status 404 returned error can't find the container with id 3aa4c52fa7004be9dbe1ed60dc671eb85f07ec9b7bbd486f3ca55db8b721ee4a Nov 01 16:30:01.235712 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:01.235615 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892510-t259f"] Nov 01 16:30:01.265689 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:01.265658 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892510-zjcr9"] Nov 01 16:30:01.271166 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:01.271138 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892510-n7htt"] Nov 01 16:30:01.271958 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:30:01.271927 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod954da573_4dff_4a25_a180_17fe21a92841.slice/crio-a9e54d3e09fd8e79ed9fcbd5679f4ac69858299185741216f7d5562a3c095871 WatchSource:0}: Error finding container a9e54d3e09fd8e79ed9fcbd5679f4ac69858299185741216f7d5562a3c095871: Status 404 returned error can't find the container with id a9e54d3e09fd8e79ed9fcbd5679f4ac69858299185741216f7d5562a3c095871 Nov 01 16:30:01.272157 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:30:01.272142 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e98f9e8_2337_406a_807c_d20dc90a35a2.slice/crio-f29d1886523ea21b1a407d53caec7acc4206e022c7bab32453cde5db7a425ea2 WatchSource:0}: Error finding container f29d1886523ea21b1a407d53caec7acc4206e022c7bab32453cde5db7a425ea2: Status 404 returned error can't find the container with id f29d1886523ea21b1a407d53caec7acc4206e022c7bab32453cde5db7a425ea2 Nov 01 16:30:01.272822 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:30:01.272801 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07c858bd_2ffa_4bbb_8123_5fdb4dc5bd0d.slice/crio-846341ba43997e46302d5884a09d0ef8f4d8c59231847c92f7a8f6a69943acfc WatchSource:0}: Error finding container 846341ba43997e46302d5884a09d0ef8f4d8c59231847c92f7a8f6a69943acfc: Status 404 returned error can't find the container with id 846341ba43997e46302d5884a09d0ef8f4d8c59231847c92f7a8f6a69943acfc Nov 01 16:30:01.276661 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:01.276637 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892510-947xq"] Nov 01 16:30:01.344091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:01.344051 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892510-cg9fh"] Nov 01 16:30:01.357328 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:01.357306 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fd73f-29892510-768f5"] Nov 01 16:30:01.384629 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:30:01.384567 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfb264a2_0df1_4b83_bd6d_a355f75a2aca.slice/crio-c093a8ccd7bf648104a27f227503586e8b2ffdc63ccc762e3abae6a8ae1aeaad WatchSource:0}: Error finding container c093a8ccd7bf648104a27f227503586e8b2ffdc63ccc762e3abae6a8ae1aeaad: Status 404 returned error can't find the container with id c093a8ccd7bf648104a27f227503586e8b2ffdc63ccc762e3abae6a8ae1aeaad Nov 01 16:30:01.481560 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:30:01.481196 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1817f9a5_677f_43b6_a8d0_b80a92870530.slice/crio-c94d1eefc15d5d633e5c2749b8f96174ddc920fae4fc6ad2e93053c8fbaa6ba6 WatchSource:0}: Error finding container c94d1eefc15d5d633e5c2749b8f96174ddc920fae4fc6ad2e93053c8fbaa6ba6: Status 404 returned error can't find the container with id c94d1eefc15d5d633e5c2749b8f96174ddc920fae4fc6ad2e93053c8fbaa6ba6 Nov 01 16:30:01.709614 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:01.709589 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-9c5644f64-xrw75" Nov 01 16:30:02.036898 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:02.036866 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fd73f-29892510-768f5" event={"ID":"1817f9a5-677f-43b6-a8d0-b80a92870530","Type":"ContainerStarted","Data":"c94d1eefc15d5d633e5c2749b8f96174ddc920fae4fc6ad2e93053c8fbaa6ba6"} Nov 01 16:30:02.038297 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:02.038268 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892510-947xq" event={"ID":"07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d","Type":"ContainerStarted","Data":"846341ba43997e46302d5884a09d0ef8f4d8c59231847c92f7a8f6a69943acfc"} Nov 01 16:30:02.039608 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:02.039586 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892510-n7htt" event={"ID":"4e98f9e8-2337-406a-807c-d20dc90a35a2","Type":"ContainerStarted","Data":"f29d1886523ea21b1a407d53caec7acc4206e022c7bab32453cde5db7a425ea2"} Nov 01 16:30:02.041329 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:02.041271 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5ccd7b-29892510-9v7dk" event={"ID":"89cbf816-3afe-455a-8e90-5606573018b0","Type":"ContainerStarted","Data":"5e0c4892e1a1704e14ef654aafae3d13838c42babfd4c7cb5a1d24c899e16777"} Nov 01 16:30:02.041329 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:02.041307 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5ccd7b-29892510-9v7dk" event={"ID":"89cbf816-3afe-455a-8e90-5606573018b0","Type":"ContainerStarted","Data":"27d79b5cf0af22eb737704085c51e5204d909528517d17f1cd3966802eb380a8"} Nov 01 16:30:02.043052 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:02.043003 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2b94b5-29892510-9n2h2" event={"ID":"975c2d1d-45a8-4439-923e-59f70d69ea04","Type":"ContainerStarted","Data":"e88c53f88aeb0daa6562973c9d033e2577d3ed6ad273a3fc00e1bb3fdc874c43"} Nov 01 16:30:02.044740 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:02.044719 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892510-t259f" event={"ID":"aa25d700-0231-4655-a067-4d637ea4c5c0","Type":"ContainerStarted","Data":"0b7672e3430f3189ab56b6925799cfc38d213b8648091ed4af253b5f21f37193"} Nov 01 16:30:02.044826 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:02.044765 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892510-t259f" event={"ID":"aa25d700-0231-4655-a067-4d637ea4c5c0","Type":"ContainerStarted","Data":"3aa4c52fa7004be9dbe1ed60dc671eb85f07ec9b7bbd486f3ca55db8b721ee4a"} Nov 01 16:30:02.046281 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:02.046260 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c8537c-29892510-6c55f" event={"ID":"bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68","Type":"ContainerStarted","Data":"34cfb6d48e723811b8ba1b913cad6b56424dfc40d93f421d73034987f985dbc8"} Nov 01 16:30:02.046386 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:02.046305 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c8537c-29892510-6c55f" event={"ID":"bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68","Type":"ContainerStarted","Data":"70b05a37cdaa3d596cf267dc378711be0fe08997a5c4c6fed2fd0c11bfa37d16"} Nov 01 16:30:02.047461 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:02.047441 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892510-zjcr9" event={"ID":"954da573-4dff-4a25-a180-17fe21a92841","Type":"ContainerStarted","Data":"a9e54d3e09fd8e79ed9fcbd5679f4ac69858299185741216f7d5562a3c095871"} Nov 01 16:30:02.048764 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:02.048741 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892510-cg9fh" event={"ID":"cfb264a2-0df1-4b83-bd6d-a355f75a2aca","Type":"ContainerStarted","Data":"c093a8ccd7bf648104a27f227503586e8b2ffdc63ccc762e3abae6a8ae1aeaad"} Nov 01 16:30:02.074967 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:02.074926 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5ccd7b-29892510-9v7dk" podStartSLOduration=2.074913732 podStartE2EDuration="2.074913732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:30:02.072973377 +0000 UTC m=+2535.404711027" watchObservedRunningTime="2026-11-01 16:30:02.074913732 +0000 UTC m=+2535.406651381" Nov 01 16:30:02.104109 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:02.104070 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2b94b5-29892510-9n2h2" podStartSLOduration=2.104052974 podStartE2EDuration="2.104052974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:30:02.100493825 +0000 UTC m=+2535.432231475" watchObservedRunningTime="2026-11-01 16:30:02.104052974 +0000 UTC m=+2535.435790624" Nov 01 16:30:02.134723 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:02.134675 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892510-t259f" podStartSLOduration=2.13466233 podStartE2EDuration="2.13466233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:30:02.132302702 +0000 UTC m=+2535.464040351" watchObservedRunningTime="2026-11-01 16:30:02.13466233 +0000 UTC m=+2535.466399979" Nov 01 16:30:02.162419 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:02.162376 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c8537c-29892510-6c55f" podStartSLOduration=2.162363642 podStartE2EDuration="2.162363642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:30:02.158404765 +0000 UTC m=+2535.490142415" watchObservedRunningTime="2026-11-01 16:30:02.162363642 +0000 UTC m=+2535.494101307" Nov 01 16:30:03.059435 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:03.059394 4363 generic.go:356] "Generic (PLEG): container finished" podID="89cbf816-3afe-455a-8e90-5606573018b0" containerID="5e0c4892e1a1704e14ef654aafae3d13838c42babfd4c7cb5a1d24c899e16777" exitCode=60 Nov 01 16:30:03.059888 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:03.059432 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5ccd7b-29892510-9v7dk" event={"ID":"89cbf816-3afe-455a-8e90-5606573018b0","Type":"ContainerDied","Data":"5e0c4892e1a1704e14ef654aafae3d13838c42babfd4c7cb5a1d24c899e16777"} Nov 01 16:30:03.059951 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:03.059901 4363 scope.go:122] "RemoveContainer" containerID="5e0c4892e1a1704e14ef654aafae3d13838c42babfd4c7cb5a1d24c899e16777" Nov 01 16:30:03.061310 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:03.061283 4363 generic.go:356] "Generic (PLEG): container finished" podID="975c2d1d-45a8-4439-923e-59f70d69ea04" containerID="e88c53f88aeb0daa6562973c9d033e2577d3ed6ad273a3fc00e1bb3fdc874c43" exitCode=60 Nov 01 16:30:03.061455 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:03.061310 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2b94b5-29892510-9n2h2" event={"ID":"975c2d1d-45a8-4439-923e-59f70d69ea04","Type":"ContainerDied","Data":"e88c53f88aeb0daa6562973c9d033e2577d3ed6ad273a3fc00e1bb3fdc874c43"} Nov 01 16:30:03.061652 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:03.061634 4363 scope.go:122] "RemoveContainer" containerID="e88c53f88aeb0daa6562973c9d033e2577d3ed6ad273a3fc00e1bb3fdc874c43" Nov 01 16:30:03.063671 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:03.063655 4363 generic.go:356] "Generic (PLEG): container finished" podID="aa25d700-0231-4655-a067-4d637ea4c5c0" containerID="0b7672e3430f3189ab56b6925799cfc38d213b8648091ed4af253b5f21f37193" exitCode=60 Nov 01 16:30:03.063776 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:03.063737 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892510-t259f" event={"ID":"aa25d700-0231-4655-a067-4d637ea4c5c0","Type":"ContainerDied","Data":"0b7672e3430f3189ab56b6925799cfc38d213b8648091ed4af253b5f21f37193"} Nov 01 16:30:03.063985 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:03.063971 4363 scope.go:122] "RemoveContainer" containerID="0b7672e3430f3189ab56b6925799cfc38d213b8648091ed4af253b5f21f37193" Nov 01 16:30:03.065730 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:03.065479 4363 generic.go:356] "Generic (PLEG): container finished" podID="bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68" containerID="34cfb6d48e723811b8ba1b913cad6b56424dfc40d93f421d73034987f985dbc8" exitCode=60 Nov 01 16:30:03.065730 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:03.065526 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c8537c-29892510-6c55f" event={"ID":"bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68","Type":"ContainerDied","Data":"34cfb6d48e723811b8ba1b913cad6b56424dfc40d93f421d73034987f985dbc8"} Nov 01 16:30:03.065822 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:03.065760 4363 scope.go:122] "RemoveContainer" containerID="34cfb6d48e723811b8ba1b913cad6b56424dfc40d93f421d73034987f985dbc8" Nov 01 16:30:03.067339 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:03.067317 4363 generic.go:356] "Generic (PLEG): container finished" podID="954da573-4dff-4a25-a180-17fe21a92841" containerID="ae7135c6d634bf56fc92d8da0658ea1bd11c73ed765cb95f8e1478e25cb62fe5" exitCode=60 Nov 01 16:30:03.067423 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:03.067395 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892510-zjcr9" event={"ID":"954da573-4dff-4a25-a180-17fe21a92841","Type":"ContainerDied","Data":"ae7135c6d634bf56fc92d8da0658ea1bd11c73ed765cb95f8e1478e25cb62fe5"} Nov 01 16:30:03.067576 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:03.067561 4363 scope.go:122] "RemoveContainer" containerID="ae7135c6d634bf56fc92d8da0658ea1bd11c73ed765cb95f8e1478e25cb62fe5" Nov 01 16:30:03.068994 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:03.068977 4363 generic.go:356] "Generic (PLEG): container finished" podID="cfb264a2-0df1-4b83-bd6d-a355f75a2aca" containerID="a99aa9565721c430148264b896ca057ae648eba77cb3592f73928ef5539d2722" exitCode=60 Nov 01 16:30:03.069092 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:03.069042 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892510-cg9fh" event={"ID":"cfb264a2-0df1-4b83-bd6d-a355f75a2aca","Type":"ContainerDied","Data":"a99aa9565721c430148264b896ca057ae648eba77cb3592f73928ef5539d2722"} Nov 01 16:30:03.069283 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:03.069269 4363 scope.go:122] "RemoveContainer" containerID="a99aa9565721c430148264b896ca057ae648eba77cb3592f73928ef5539d2722" Nov 01 16:30:03.070674 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:03.070657 4363 generic.go:356] "Generic (PLEG): container finished" podID="1817f9a5-677f-43b6-a8d0-b80a92870530" containerID="967fb6ce1d9d12b9bcdb9259bd42fddbdf936f8c025b2d9b12d2c6881b772fe8" exitCode=60 Nov 01 16:30:03.070784 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:03.070682 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fd73f-29892510-768f5" event={"ID":"1817f9a5-677f-43b6-a8d0-b80a92870530","Type":"ContainerDied","Data":"967fb6ce1d9d12b9bcdb9259bd42fddbdf936f8c025b2d9b12d2c6881b772fe8"} Nov 01 16:30:03.070943 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:03.070926 4363 scope.go:122] "RemoveContainer" containerID="967fb6ce1d9d12b9bcdb9259bd42fddbdf936f8c025b2d9b12d2c6881b772fe8" Nov 01 16:30:03.072319 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:03.072302 4363 generic.go:356] "Generic (PLEG): container finished" podID="07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d" containerID="d776f75544a18656410da0c1bd684f79f50945ad04657180c6ba15ab5eec7a65" exitCode=60 Nov 01 16:30:03.072397 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:03.072381 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892510-947xq" event={"ID":"07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d","Type":"ContainerDied","Data":"d776f75544a18656410da0c1bd684f79f50945ad04657180c6ba15ab5eec7a65"} Nov 01 16:30:03.072583 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:03.072565 4363 scope.go:122] "RemoveContainer" containerID="d776f75544a18656410da0c1bd684f79f50945ad04657180c6ba15ab5eec7a65" Nov 01 16:30:03.083582 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:03.082740 4363 generic.go:356] "Generic (PLEG): container finished" podID="4e98f9e8-2337-406a-807c-d20dc90a35a2" containerID="00223708f30a70ecb8784574da778512d0950bbe7add56ab87f63a64f384d10e" exitCode=60 Nov 01 16:30:03.083582 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:03.082792 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892510-n7htt" event={"ID":"4e98f9e8-2337-406a-807c-d20dc90a35a2","Type":"ContainerDied","Data":"00223708f30a70ecb8784574da778512d0950bbe7add56ab87f63a64f384d10e"} Nov 01 16:30:03.083582 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:03.083274 4363 scope.go:122] "RemoveContainer" containerID="00223708f30a70ecb8784574da778512d0950bbe7add56ab87f63a64f384d10e" Nov 01 16:30:03.535926 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:03.535899 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-l5qnm_dae26a45-6a59-4c32-a565-704e50c5f271/payload-processing/0.log" Nov 01 16:30:03.700793 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:03.700774 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-l5qnm_dae26a45-6a59-4c32-a565-704e50c5f271/payload-processing/0.log" Nov 01 16:30:04.101425 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:04.101393 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892510-cg9fh" event={"ID":"cfb264a2-0df1-4b83-bd6d-a355f75a2aca","Type":"ContainerStarted","Data":"a26c9dc5112f36ec783326824458696b1fe7f1cc659a5cafd81e5d3fbece4164"} Nov 01 16:30:04.103532 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:04.103500 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fd73f-29892510-768f5" event={"ID":"1817f9a5-677f-43b6-a8d0-b80a92870530","Type":"ContainerStarted","Data":"bf093ad2ce9878574e5cb34b04b3a631b39f51041f8501514b7501096882428d"} Nov 01 16:30:04.106561 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:04.106535 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892510-947xq" event={"ID":"07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d","Type":"ContainerStarted","Data":"6eb76660363a99a1621ceeb8af832c612399469e04b8d10c88c61ad16b9885f0"} Nov 01 16:30:04.108479 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:04.108453 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892510-n7htt" event={"ID":"4e98f9e8-2337-406a-807c-d20dc90a35a2","Type":"ContainerStarted","Data":"f8aa6f0f37be9228c6a938c8d18ed9a8609fb8eb38fb58fbed58ad8cb1de36a6"} Nov 01 16:30:04.110532 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:04.110509 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5ccd7b-29892510-9v7dk" event={"ID":"89cbf816-3afe-455a-8e90-5606573018b0","Type":"ContainerStarted","Data":"62634e1cdc0137521d9aae0e262e834c1b7895b0e3535cce0de8d8dcc23774c2"} Nov 01 16:30:04.112462 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:04.112438 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2b94b5-29892510-9n2h2" event={"ID":"975c2d1d-45a8-4439-923e-59f70d69ea04","Type":"ContainerStarted","Data":"4af16d31c7a66cefc30fa843d1a2c8e0ed0aa0503cfbd9989794f4ca018c7db0"} Nov 01 16:30:04.114961 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:04.114651 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892510-t259f" event={"ID":"aa25d700-0231-4655-a067-4d637ea4c5c0","Type":"ContainerStarted","Data":"ba46e08edd70040f475720e9fa239126d5f5b81fdfc280dedd2609e003acfc71"} Nov 01 16:30:04.119253 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:04.116725 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c8537c-29892510-6c55f" event={"ID":"bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68","Type":"ContainerStarted","Data":"4534b751b43106ea725cec2c13527b2e5d6975b9d981b557fba81ad950db324a"} Nov 01 16:30:04.119602 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:04.119582 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892510-zjcr9" event={"ID":"954da573-4dff-4a25-a180-17fe21a92841","Type":"ContainerStarted","Data":"0b13f038fd7dc35b1ffd3dacdbb92d7033f23cb56dc4197b776e632fad47c9fd"} Nov 01 16:30:04.120728 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:04.120690 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892510-cg9fh" podStartSLOduration=4.120677106 podStartE2EDuration="4.120677106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:30:04.116977686 +0000 UTC m=+2537.448715336" watchObservedRunningTime="2026-11-01 16:30:04.120677106 +0000 UTC m=+2537.452414756" Nov 01 16:30:04.155214 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:04.155171 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892510-n7htt" podStartSLOduration=4.15515748 podStartE2EDuration="4.15515748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:30:04.149411637 +0000 UTC m=+2537.481149287" watchObservedRunningTime="2026-11-01 16:30:04.15515748 +0000 UTC m=+2537.486895160" Nov 01 16:30:04.248431 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:04.248367 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fd73f-29892510-768f5" podStartSLOduration=4.248346302 podStartE2EDuration="4.248346302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:30:04.239679994 +0000 UTC m=+2537.571417648" watchObservedRunningTime="2026-11-01 16:30:04.248346302 +0000 UTC m=+2537.580083958" Nov 01 16:30:04.261523 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:04.261451 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892510-947xq" podStartSLOduration=4.261431448 podStartE2EDuration="4.261431448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:30:04.257573617 +0000 UTC m=+2537.589311267" watchObservedRunningTime="2026-11-01 16:30:04.261431448 +0000 UTC m=+2537.593169102" Nov 01 16:30:04.309361 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:04.309307 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892510-zjcr9" podStartSLOduration=4.309294178 podStartE2EDuration="4.309294178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:30:04.306995216 +0000 UTC m=+2537.638732867" watchObservedRunningTime="2026-11-01 16:30:04.309294178 +0000 UTC m=+2537.641031827" Nov 01 16:30:04.813231 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:04.813188 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5e2d03-cbd9bb6f9-bbprx" podUID="96c20c16-b48c-40b5-853c-edddd1225652" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:30:04.819269 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:04.819233 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5e2d03-757556fddc-j9gph" podUID="57dbe32d-78fb-4168-a5f8-5fa0a38f2e2b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:30:05.131991 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.131951 4363 generic.go:356] "Generic (PLEG): container finished" podID="07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d" containerID="6eb76660363a99a1621ceeb8af832c612399469e04b8d10c88c61ad16b9885f0" exitCode=60 Nov 01 16:30:05.132439 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.132047 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892510-947xq" event={"ID":"07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d","Type":"ContainerDied","Data":"6eb76660363a99a1621ceeb8af832c612399469e04b8d10c88c61ad16b9885f0"} Nov 01 16:30:05.132439 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.132143 4363 scope.go:122] "RemoveContainer" containerID="d776f75544a18656410da0c1bd684f79f50945ad04657180c6ba15ab5eec7a65" Nov 01 16:30:05.132632 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.132598 4363 scope.go:122] "RemoveContainer" containerID="6eb76660363a99a1621ceeb8af832c612399469e04b8d10c88c61ad16b9885f0" Nov 01 16:30:05.133109 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:05.133086 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892510-947xq_odh-ai-gateway-infra(07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892510-947xq" podUID="07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d" Nov 01 16:30:05.134324 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.134302 4363 generic.go:356] "Generic (PLEG): container finished" podID="4e98f9e8-2337-406a-807c-d20dc90a35a2" containerID="f8aa6f0f37be9228c6a938c8d18ed9a8609fb8eb38fb58fbed58ad8cb1de36a6" exitCode=60 Nov 01 16:30:05.134461 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.134377 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892510-n7htt" event={"ID":"4e98f9e8-2337-406a-807c-d20dc90a35a2","Type":"ContainerDied","Data":"f8aa6f0f37be9228c6a938c8d18ed9a8609fb8eb38fb58fbed58ad8cb1de36a6"} Nov 01 16:30:05.134725 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.134705 4363 scope.go:122] "RemoveContainer" containerID="f8aa6f0f37be9228c6a938c8d18ed9a8609fb8eb38fb58fbed58ad8cb1de36a6" Nov 01 16:30:05.135131 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:05.135110 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-43fac8-29892510-n7htt_odh-ai-gateway-infra(4e98f9e8-2337-406a-807c-d20dc90a35a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892510-n7htt" podUID="4e98f9e8-2337-406a-807c-d20dc90a35a2" Nov 01 16:30:05.136296 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.136276 4363 generic.go:356] "Generic (PLEG): container finished" podID="89cbf816-3afe-455a-8e90-5606573018b0" containerID="62634e1cdc0137521d9aae0e262e834c1b7895b0e3535cce0de8d8dcc23774c2" exitCode=60 Nov 01 16:30:05.136382 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.136355 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5ccd7b-29892510-9v7dk" event={"ID":"89cbf816-3afe-455a-8e90-5606573018b0","Type":"ContainerDied","Data":"62634e1cdc0137521d9aae0e262e834c1b7895b0e3535cce0de8d8dcc23774c2"} Nov 01 16:30:05.136600 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.136581 4363 scope.go:122] "RemoveContainer" containerID="62634e1cdc0137521d9aae0e262e834c1b7895b0e3535cce0de8d8dcc23774c2" Nov 01 16:30:05.136872 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:05.136850 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-5ccd7b-29892510-9v7dk_odh-ai-gateway-infra(89cbf816-3afe-455a-8e90-5606573018b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5ccd7b-29892510-9v7dk" podUID="89cbf816-3afe-455a-8e90-5606573018b0" Nov 01 16:30:05.140207 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.140188 4363 generic.go:356] "Generic (PLEG): container finished" podID="975c2d1d-45a8-4439-923e-59f70d69ea04" containerID="4af16d31c7a66cefc30fa843d1a2c8e0ed0aa0503cfbd9989794f4ca018c7db0" exitCode=60 Nov 01 16:30:05.140321 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.140277 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2b94b5-29892510-9n2h2" event={"ID":"975c2d1d-45a8-4439-923e-59f70d69ea04","Type":"ContainerDied","Data":"4af16d31c7a66cefc30fa843d1a2c8e0ed0aa0503cfbd9989794f4ca018c7db0"} Nov 01 16:30:05.140505 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.140488 4363 scope.go:122] "RemoveContainer" containerID="4af16d31c7a66cefc30fa843d1a2c8e0ed0aa0503cfbd9989794f4ca018c7db0" Nov 01 16:30:05.140798 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:05.140775 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-2b94b5-29892510-9n2h2_odh-ai-gateway-infra(975c2d1d-45a8-4439-923e-59f70d69ea04)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2b94b5-29892510-9n2h2" podUID="975c2d1d-45a8-4439-923e-59f70d69ea04" Nov 01 16:30:05.142438 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.142413 4363 generic.go:356] "Generic (PLEG): container finished" podID="aa25d700-0231-4655-a067-4d637ea4c5c0" containerID="ba46e08edd70040f475720e9fa239126d5f5b81fdfc280dedd2609e003acfc71" exitCode=60 Nov 01 16:30:05.142559 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.142499 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892510-t259f" event={"ID":"aa25d700-0231-4655-a067-4d637ea4c5c0","Type":"ContainerDied","Data":"ba46e08edd70040f475720e9fa239126d5f5b81fdfc280dedd2609e003acfc71"} Nov 01 16:30:05.142707 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.142691 4363 scope.go:122] "RemoveContainer" containerID="ba46e08edd70040f475720e9fa239126d5f5b81fdfc280dedd2609e003acfc71" Nov 01 16:30:05.142984 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:05.142957 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892510-t259f_odh-ai-gateway-infra(aa25d700-0231-4655-a067-4d637ea4c5c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892510-t259f" podUID="aa25d700-0231-4655-a067-4d637ea4c5c0" Nov 01 16:30:05.144587 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.144567 4363 generic.go:356] "Generic (PLEG): container finished" podID="bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68" containerID="4534b751b43106ea725cec2c13527b2e5d6975b9d981b557fba81ad950db324a" exitCode=60 Nov 01 16:30:05.144717 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.144678 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c8537c-29892510-6c55f" event={"ID":"bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68","Type":"ContainerDied","Data":"4534b751b43106ea725cec2c13527b2e5d6975b9d981b557fba81ad950db324a"} Nov 01 16:30:05.144866 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.144847 4363 scope.go:122] "RemoveContainer" containerID="4534b751b43106ea725cec2c13527b2e5d6975b9d981b557fba81ad950db324a" Nov 01 16:30:05.145226 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:05.145198 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-c8537c-29892510-6c55f_odh-ai-gateway-infra(bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c8537c-29892510-6c55f" podUID="bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68" Nov 01 16:30:05.146764 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.146744 4363 generic.go:356] "Generic (PLEG): container finished" podID="954da573-4dff-4a25-a180-17fe21a92841" containerID="0b13f038fd7dc35b1ffd3dacdbb92d7033f23cb56dc4197b776e632fad47c9fd" exitCode=60 Nov 01 16:30:05.146861 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.146769 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892510-zjcr9" event={"ID":"954da573-4dff-4a25-a180-17fe21a92841","Type":"ContainerDied","Data":"0b13f038fd7dc35b1ffd3dacdbb92d7033f23cb56dc4197b776e632fad47c9fd"} Nov 01 16:30:05.147142 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.147118 4363 scope.go:122] "RemoveContainer" containerID="0b13f038fd7dc35b1ffd3dacdbb92d7033f23cb56dc4197b776e632fad47c9fd" Nov 01 16:30:05.147459 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:05.147435 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892510-zjcr9_odh-ai-gateway-infra(954da573-4dff-4a25-a180-17fe21a92841)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892510-zjcr9" podUID="954da573-4dff-4a25-a180-17fe21a92841" Nov 01 16:30:05.148666 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.148647 4363 scope.go:122] "RemoveContainer" containerID="00223708f30a70ecb8784574da778512d0950bbe7add56ab87f63a64f384d10e" Nov 01 16:30:05.148889 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.148871 4363 generic.go:356] "Generic (PLEG): container finished" podID="cfb264a2-0df1-4b83-bd6d-a355f75a2aca" containerID="a26c9dc5112f36ec783326824458696b1fe7f1cc659a5cafd81e5d3fbece4164" exitCode=60 Nov 01 16:30:05.148961 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.148944 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892510-cg9fh" event={"ID":"cfb264a2-0df1-4b83-bd6d-a355f75a2aca","Type":"ContainerDied","Data":"a26c9dc5112f36ec783326824458696b1fe7f1cc659a5cafd81e5d3fbece4164"} Nov 01 16:30:05.149300 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.149161 4363 scope.go:122] "RemoveContainer" containerID="a26c9dc5112f36ec783326824458696b1fe7f1cc659a5cafd81e5d3fbece4164" Nov 01 16:30:05.149729 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:05.149472 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892510-cg9fh_odh-ai-gateway-infra(cfb264a2-0df1-4b83-bd6d-a355f75a2aca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892510-cg9fh" podUID="cfb264a2-0df1-4b83-bd6d-a355f75a2aca" Nov 01 16:30:05.152004 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.151981 4363 generic.go:356] "Generic (PLEG): container finished" podID="1817f9a5-677f-43b6-a8d0-b80a92870530" containerID="bf093ad2ce9878574e5cb34b04b3a631b39f51041f8501514b7501096882428d" exitCode=60 Nov 01 16:30:05.152173 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.152049 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fd73f-29892510-768f5" event={"ID":"1817f9a5-677f-43b6-a8d0-b80a92870530","Type":"ContainerDied","Data":"bf093ad2ce9878574e5cb34b04b3a631b39f51041f8501514b7501096882428d"} Nov 01 16:30:05.152333 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.152319 4363 scope.go:122] "RemoveContainer" containerID="bf093ad2ce9878574e5cb34b04b3a631b39f51041f8501514b7501096882428d" Nov 01 16:30:05.152607 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:05.152587 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-3fd73f-29892510-768f5_odh-ai-gateway-infra(1817f9a5-677f-43b6-a8d0-b80a92870530)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fd73f-29892510-768f5" podUID="1817f9a5-677f-43b6-a8d0-b80a92870530" Nov 01 16:30:05.165304 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.165288 4363 scope.go:122] "RemoveContainer" containerID="5e0c4892e1a1704e14ef654aafae3d13838c42babfd4c7cb5a1d24c899e16777" Nov 01 16:30:05.180091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.180064 4363 scope.go:122] "RemoveContainer" containerID="e88c53f88aeb0daa6562973c9d033e2577d3ed6ad273a3fc00e1bb3fdc874c43" Nov 01 16:30:05.196688 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.196664 4363 scope.go:122] "RemoveContainer" containerID="0b7672e3430f3189ab56b6925799cfc38d213b8648091ed4af253b5f21f37193" Nov 01 16:30:05.213420 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.213397 4363 scope.go:122] "RemoveContainer" containerID="34cfb6d48e723811b8ba1b913cad6b56424dfc40d93f421d73034987f985dbc8" Nov 01 16:30:05.229080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.229059 4363 scope.go:122] "RemoveContainer" containerID="ae7135c6d634bf56fc92d8da0658ea1bd11c73ed765cb95f8e1478e25cb62fe5" Nov 01 16:30:05.243513 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.243496 4363 scope.go:122] "RemoveContainer" containerID="a99aa9565721c430148264b896ca057ae648eba77cb3592f73928ef5539d2722" Nov 01 16:30:05.258153 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.258132 4363 scope.go:122] "RemoveContainer" containerID="967fb6ce1d9d12b9bcdb9259bd42fddbdf936f8c025b2d9b12d2c6881b772fe8" Nov 01 16:30:05.397983 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.393184 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4"] Nov 01 16:30:05.403778 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.403750 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" Nov 01 16:30:05.418614 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.418442 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-6b6cfc-openshift-default-dockercfg-52f5v\"" Nov 01 16:30:05.420377 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.420352 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4"] Nov 01 16:30:05.582053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.508455 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69113cc0-84e0-4de6-8078-aa79ad5cad66-credential-socket\") pod \"e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4\" (UID: \"69113cc0-84e0-4de6-8078-aa79ad5cad66\") " pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" Nov 01 16:30:05.582053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.508504 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69113cc0-84e0-4de6-8078-aa79ad5cad66-istio-data\") pod \"e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4\" (UID: \"69113cc0-84e0-4de6-8078-aa79ad5cad66\") " pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" Nov 01 16:30:05.582053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.508544 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69113cc0-84e0-4de6-8078-aa79ad5cad66-istio-envoy\") pod \"e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4\" (UID: \"69113cc0-84e0-4de6-8078-aa79ad5cad66\") " pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" Nov 01 16:30:05.582053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.508688 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69113cc0-84e0-4de6-8078-aa79ad5cad66-istio-podinfo\") pod \"e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4\" (UID: \"69113cc0-84e0-4de6-8078-aa79ad5cad66\") " pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" Nov 01 16:30:05.582053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.508720 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69113cc0-84e0-4de6-8078-aa79ad5cad66-istio-token\") pod \"e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4\" (UID: \"69113cc0-84e0-4de6-8078-aa79ad5cad66\") " pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" Nov 01 16:30:05.582053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.508784 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69113cc0-84e0-4de6-8078-aa79ad5cad66-workload-socket\") pod \"e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4\" (UID: \"69113cc0-84e0-4de6-8078-aa79ad5cad66\") " pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" Nov 01 16:30:05.582053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.508903 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69113cc0-84e0-4de6-8078-aa79ad5cad66-workload-certs\") pod \"e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4\" (UID: \"69113cc0-84e0-4de6-8078-aa79ad5cad66\") " pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" Nov 01 16:30:05.582053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.508965 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69113cc0-84e0-4de6-8078-aa79ad5cad66-istiod-ca-cert\") pod \"e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4\" (UID: \"69113cc0-84e0-4de6-8078-aa79ad5cad66\") " pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" Nov 01 16:30:05.582053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.509052 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vr8n\" (UniqueName: \"kubernetes.io/projected/69113cc0-84e0-4de6-8078-aa79ad5cad66-kube-api-access-7vr8n\") pod \"e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4\" (UID: \"69113cc0-84e0-4de6-8078-aa79ad5cad66\") " pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" Nov 01 16:30:05.630038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.627558 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69113cc0-84e0-4de6-8078-aa79ad5cad66-credential-socket\") pod \"e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4\" (UID: \"69113cc0-84e0-4de6-8078-aa79ad5cad66\") " pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" Nov 01 16:30:05.630038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.627602 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69113cc0-84e0-4de6-8078-aa79ad5cad66-istio-data\") pod \"e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4\" (UID: \"69113cc0-84e0-4de6-8078-aa79ad5cad66\") " pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" Nov 01 16:30:05.630038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.627638 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69113cc0-84e0-4de6-8078-aa79ad5cad66-istio-envoy\") pod \"e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4\" (UID: \"69113cc0-84e0-4de6-8078-aa79ad5cad66\") " pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" Nov 01 16:30:05.630038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.627734 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69113cc0-84e0-4de6-8078-aa79ad5cad66-istio-podinfo\") pod \"e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4\" (UID: \"69113cc0-84e0-4de6-8078-aa79ad5cad66\") " pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" Nov 01 16:30:05.630038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.627764 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69113cc0-84e0-4de6-8078-aa79ad5cad66-istio-token\") pod \"e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4\" (UID: \"69113cc0-84e0-4de6-8078-aa79ad5cad66\") " pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" Nov 01 16:30:05.630038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.627814 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69113cc0-84e0-4de6-8078-aa79ad5cad66-workload-socket\") pod \"e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4\" (UID: \"69113cc0-84e0-4de6-8078-aa79ad5cad66\") " pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" Nov 01 16:30:05.630038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.627902 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69113cc0-84e0-4de6-8078-aa79ad5cad66-workload-certs\") pod \"e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4\" (UID: \"69113cc0-84e0-4de6-8078-aa79ad5cad66\") " pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" Nov 01 16:30:05.630038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.627956 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69113cc0-84e0-4de6-8078-aa79ad5cad66-istiod-ca-cert\") pod \"e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4\" (UID: \"69113cc0-84e0-4de6-8078-aa79ad5cad66\") " pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" Nov 01 16:30:05.630038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.628036 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vr8n\" (UniqueName: \"kubernetes.io/projected/69113cc0-84e0-4de6-8078-aa79ad5cad66-kube-api-access-7vr8n\") pod \"e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4\" (UID: \"69113cc0-84e0-4de6-8078-aa79ad5cad66\") " pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" Nov 01 16:30:05.630038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.628751 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69113cc0-84e0-4de6-8078-aa79ad5cad66-credential-socket\") pod \"e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4\" (UID: \"69113cc0-84e0-4de6-8078-aa79ad5cad66\") " pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" Nov 01 16:30:05.680575 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.672440 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69113cc0-84e0-4de6-8078-aa79ad5cad66-workload-certs\") pod \"e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4\" (UID: \"69113cc0-84e0-4de6-8078-aa79ad5cad66\") " pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" Nov 01 16:30:05.680575 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.679230 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69113cc0-84e0-4de6-8078-aa79ad5cad66-workload-socket\") pod \"e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4\" (UID: \"69113cc0-84e0-4de6-8078-aa79ad5cad66\") " pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" Nov 01 16:30:05.680575 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.679885 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69113cc0-84e0-4de6-8078-aa79ad5cad66-istiod-ca-cert\") pod \"e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4\" (UID: \"69113cc0-84e0-4de6-8078-aa79ad5cad66\") " pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" Nov 01 16:30:05.691037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.689181 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69113cc0-84e0-4de6-8078-aa79ad5cad66-istio-podinfo\") pod \"e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4\" (UID: \"69113cc0-84e0-4de6-8078-aa79ad5cad66\") " pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" Nov 01 16:30:05.719918 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.718045 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69113cc0-84e0-4de6-8078-aa79ad5cad66-istio-envoy\") pod \"e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4\" (UID: \"69113cc0-84e0-4de6-8078-aa79ad5cad66\") " pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" Nov 01 16:30:05.724377 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.724005 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69113cc0-84e0-4de6-8078-aa79ad5cad66-istio-token\") pod \"e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4\" (UID: \"69113cc0-84e0-4de6-8078-aa79ad5cad66\") " pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" Nov 01 16:30:05.767692 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.756857 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vr8n\" (UniqueName: \"kubernetes.io/projected/69113cc0-84e0-4de6-8078-aa79ad5cad66-kube-api-access-7vr8n\") pod \"e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4\" (UID: \"69113cc0-84e0-4de6-8078-aa79ad5cad66\") " pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" Nov 01 16:30:05.821144 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:05.629009 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69113cc0-84e0-4de6-8078-aa79ad5cad66-istio-data\") pod \"e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4\" (UID: \"69113cc0-84e0-4de6-8078-aa79ad5cad66\") " pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" Nov 01 16:30:06.046152 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:06.040537 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" Nov 01 16:30:06.227535 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:06.227494 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz"] Nov 01 16:30:06.227984 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:06.227768 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" podUID="19456502-09eb-4dc7-8877-395f1b5e7218" containerName="istio-proxy" containerID="cri-o://78c2dd5df3b02f6d6dd5d479cbb2392b759b43038a569cc0550ee78de1e5ee13" gracePeriod=30 Nov 01 16:30:06.279190 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:06.253789 4363 scope.go:122] "RemoveContainer" containerID="0b13f038fd7dc35b1ffd3dacdbb92d7033f23cb56dc4197b776e632fad47c9fd" Nov 01 16:30:06.279190 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:06.254332 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892510-zjcr9_odh-ai-gateway-infra(954da573-4dff-4a25-a180-17fe21a92841)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892510-zjcr9" podUID="954da573-4dff-4a25-a180-17fe21a92841" Nov 01 16:30:06.287550 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:06.286984 4363 scope.go:122] "RemoveContainer" containerID="a26c9dc5112f36ec783326824458696b1fe7f1cc659a5cafd81e5d3fbece4164" Nov 01 16:30:06.287550 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:06.287286 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892510-cg9fh_odh-ai-gateway-infra(cfb264a2-0df1-4b83-bd6d-a355f75a2aca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892510-cg9fh" podUID="cfb264a2-0df1-4b83-bd6d-a355f75a2aca" Nov 01 16:30:06.289929 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:06.289654 4363 scope.go:122] "RemoveContainer" containerID="bf093ad2ce9878574e5cb34b04b3a631b39f51041f8501514b7501096882428d" Nov 01 16:30:06.289929 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:06.289881 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-3fd73f-29892510-768f5_odh-ai-gateway-infra(1817f9a5-677f-43b6-a8d0-b80a92870530)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fd73f-29892510-768f5" podUID="1817f9a5-677f-43b6-a8d0-b80a92870530" Nov 01 16:30:06.292336 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:06.292167 4363 scope.go:122] "RemoveContainer" containerID="6eb76660363a99a1621ceeb8af832c612399469e04b8d10c88c61ad16b9885f0" Nov 01 16:30:06.292677 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:06.292628 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892510-947xq_odh-ai-gateway-infra(07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892510-947xq" podUID="07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d" Nov 01 16:30:06.500043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:06.421742 4363 scope.go:122] "RemoveContainer" containerID="f8aa6f0f37be9228c6a938c8d18ed9a8609fb8eb38fb58fbed58ad8cb1de36a6" Nov 01 16:30:06.500043 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:06.422190 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-43fac8-29892510-n7htt_odh-ai-gateway-infra(4e98f9e8-2337-406a-807c-d20dc90a35a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892510-n7htt" podUID="4e98f9e8-2337-406a-807c-d20dc90a35a2" Nov 01 16:30:06.652308 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:06.652268 4363 scope.go:122] "RemoveContainer" containerID="62634e1cdc0137521d9aae0e262e834c1b7895b0e3535cce0de8d8dcc23774c2" Nov 01 16:30:06.652941 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:06.652914 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-5ccd7b-29892510-9v7dk_odh-ai-gateway-infra(89cbf816-3afe-455a-8e90-5606573018b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5ccd7b-29892510-9v7dk" podUID="89cbf816-3afe-455a-8e90-5606573018b0" Nov 01 16:30:06.656494 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:06.656471 4363 scope.go:122] "RemoveContainer" containerID="4af16d31c7a66cefc30fa843d1a2c8e0ed0aa0503cfbd9989794f4ca018c7db0" Nov 01 16:30:06.657010 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:06.656985 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-2b94b5-29892510-9n2h2_odh-ai-gateway-infra(975c2d1d-45a8-4439-923e-59f70d69ea04)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2b94b5-29892510-9n2h2" podUID="975c2d1d-45a8-4439-923e-59f70d69ea04" Nov 01 16:30:06.660799 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:06.659926 4363 scope.go:122] "RemoveContainer" containerID="ba46e08edd70040f475720e9fa239126d5f5b81fdfc280dedd2609e003acfc71" Nov 01 16:30:06.660799 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:06.660467 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892510-t259f_odh-ai-gateway-infra(aa25d700-0231-4655-a067-4d637ea4c5c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892510-t259f" podUID="aa25d700-0231-4655-a067-4d637ea4c5c0" Nov 01 16:30:06.690764 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:06.690279 4363 scope.go:122] "RemoveContainer" containerID="4534b751b43106ea725cec2c13527b2e5d6975b9d981b557fba81ad950db324a" Nov 01 16:30:06.690764 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:06.690707 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-c8537c-29892510-6c55f_odh-ai-gateway-infra(bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c8537c-29892510-6c55f" podUID="bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68" Nov 01 16:30:06.702039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:06.701410 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d121-k6r77"] Nov 01 16:30:06.710252 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:06.710228 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4"] Nov 01 16:30:06.710429 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:06.710321 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d121-k6r77" Nov 01 16:30:06.714283 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:06.714113 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d121-k6r77"] Nov 01 16:30:06.717915 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:06.717890 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:30:06.726160 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:06.726128 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:30:06.726275 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:06.726205 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:30:06.726275 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:06.726253 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:30:06.735966 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:06.735941 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6097f042-a435-451a-8fba-818eeef14f43-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c8d121-k6r77\" (UID: \"6097f042-a435-451a-8fba-818eeef14f43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d121-k6r77" Nov 01 16:30:06.736180 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:06.736158 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6097f042-a435-451a-8fba-818eeef14f43-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c8d121-k6r77\" (UID: \"6097f042-a435-451a-8fba-818eeef14f43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d121-k6r77" Nov 01 16:30:06.736370 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:06.736354 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5svl4\" (UniqueName: \"kubernetes.io/projected/6097f042-a435-451a-8fba-818eeef14f43-kube-api-access-5svl4\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c8d121-k6r77\" (UID: \"6097f042-a435-451a-8fba-818eeef14f43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d121-k6r77" Nov 01 16:30:06.837666 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:06.837631 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6097f042-a435-451a-8fba-818eeef14f43-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c8d121-k6r77\" (UID: \"6097f042-a435-451a-8fba-818eeef14f43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d121-k6r77" Nov 01 16:30:06.837793 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:06.837748 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6097f042-a435-451a-8fba-818eeef14f43-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c8d121-k6r77\" (UID: \"6097f042-a435-451a-8fba-818eeef14f43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d121-k6r77" Nov 01 16:30:06.837985 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:06.837962 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5svl4\" (UniqueName: \"kubernetes.io/projected/6097f042-a435-451a-8fba-818eeef14f43-kube-api-access-5svl4\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c8d121-k6r77\" (UID: \"6097f042-a435-451a-8fba-818eeef14f43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d121-k6r77" Nov 01 16:30:06.838404 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:06.838386 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6097f042-a435-451a-8fba-818eeef14f43-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c8d121-k6r77\" (UID: \"6097f042-a435-451a-8fba-818eeef14f43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d121-k6r77" Nov 01 16:30:06.840000 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:06.839975 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6097f042-a435-451a-8fba-818eeef14f43-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c8d121-k6r77\" (UID: \"6097f042-a435-451a-8fba-818eeef14f43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d121-k6r77" Nov 01 16:30:06.845048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:06.845008 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5svl4\" (UniqueName: \"kubernetes.io/projected/6097f042-a435-451a-8fba-818eeef14f43-kube-api-access-5svl4\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c8d121-k6r77\" (UID: \"6097f042-a435-451a-8fba-818eeef14f43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d121-k6r77" Nov 01 16:30:07.051718 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:07.051635 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d121-k6r77" Nov 01 16:30:07.513330 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:07.496055 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d121-k6r77"] Nov 01 16:30:07.701726 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:07.701690 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d121-k6r77" event={"ID":"6097f042-a435-451a-8fba-818eeef14f43","Type":"ContainerStarted","Data":"715ffd15f8b50383597573af7b8192677265d3059568be68fcea3404b05c3ddd"} Nov 01 16:30:07.703891 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:07.703861 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" event={"ID":"69113cc0-84e0-4de6-8078-aa79ad5cad66","Type":"ContainerStarted","Data":"23fa7d7a5213c3f856678ea3caab08f627ff8f5dd3bc2456e2f45811ed5a5ab0"} Nov 01 16:30:07.704102 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:07.703908 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" event={"ID":"69113cc0-84e0-4de6-8078-aa79ad5cad66","Type":"ContainerStarted","Data":"0cb077b09086d13c7753b36c9464843d37947b08f8b8623cb177fd51f6ebeef5"} Nov 01 16:30:07.750095 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:07.747945 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" podStartSLOduration=2.747920068 podStartE2EDuration="2.747920068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:30:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:30:07.741400588 +0000 UTC m=+2541.073138252" watchObservedRunningTime="2026-11-01 16:30:07.747920068 +0000 UTC m=+2541.079657720" Nov 01 16:30:08.041583 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:08.041488 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" Nov 01 16:30:08.047171 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:08.047146 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" Nov 01 16:30:08.738701 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:08.738652 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d121-k6r77" event={"ID":"6097f042-a435-451a-8fba-818eeef14f43","Type":"ContainerStarted","Data":"c5559907747a696a36e741066171b1c3ee79328c4c0ee47ab89fec29e0cb0ca9"} Nov 01 16:30:08.742283 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:08.742256 4363 generic.go:356] "Generic (PLEG): container finished" podID="96c20c16-b48c-40b5-853c-edddd1225652" containerID="5657a369b4632c5bc020f3a2169db572af33e2980861b959df88957d23a9f99d" exitCode=0 Nov 01 16:30:08.742462 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:08.742442 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5e2d03-cbd9bb6f9-bbprx" event={"ID":"96c20c16-b48c-40b5-853c-edddd1225652","Type":"ContainerDied","Data":"5657a369b4632c5bc020f3a2169db572af33e2980861b959df88957d23a9f99d"} Nov 01 16:30:08.746064 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:08.745705 4363 generic.go:356] "Generic (PLEG): container finished" podID="57dbe32d-78fb-4168-a5f8-5fa0a38f2e2b" containerID="44fdf0f6ebd17a0dd4f8dbc5a4b72bb1afe67bb9be6ce8ecd1f6f08822eaccf9" exitCode=0 Nov 01 16:30:08.746185 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:08.746130 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5e2d03-757556fddc-j9gph" event={"ID":"57dbe32d-78fb-4168-a5f8-5fa0a38f2e2b","Type":"ContainerDied","Data":"44fdf0f6ebd17a0dd4f8dbc5a4b72bb1afe67bb9be6ce8ecd1f6f08822eaccf9"} Nov 01 16:30:08.746257 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:08.746200 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" Nov 01 16:30:08.747596 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:08.747569 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" Nov 01 16:30:08.763184 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:08.762246 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d121-k6r77" podStartSLOduration=2.762229164 podStartE2EDuration="2.762229164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:30:08.756352786 +0000 UTC m=+2542.088090437" watchObservedRunningTime="2026-11-01 16:30:08.762229164 +0000 UTC m=+2542.093966815" Nov 01 16:30:08.770760 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:08.770743 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5e2d03-cbd9bb6f9-bbprx" Nov 01 16:30:08.827160 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:08.826069 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96c20c16-b48c-40b5-853c-edddd1225652-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96c20c16-b48c-40b5-853c-edddd1225652-plugins-config-volume\") pod \"96c20c16-b48c-40b5-853c-edddd1225652\" (UID: \"96c20c16-b48c-40b5-853c-edddd1225652\") " Nov 01 16:30:08.827160 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:08.826333 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96c20c16-b48c-40b5-853c-edddd1225652-kube-api-access-59skl\" (UniqueName: \"kubernetes.io/projected/96c20c16-b48c-40b5-853c-edddd1225652-kube-api-access-59skl\") pod \"96c20c16-b48c-40b5-853c-edddd1225652\" (UID: \"96c20c16-b48c-40b5-853c-edddd1225652\") " Nov 01 16:30:08.828893 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:08.828866 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96c20c16-b48c-40b5-853c-edddd1225652-plugins-config-volume" pod "96c20c16-b48c-40b5-853c-edddd1225652" (UID: "96c20c16-b48c-40b5-853c-edddd1225652"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:30:08.834453 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:08.834407 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96c20c16-b48c-40b5-853c-edddd1225652-kube-api-access-59skl" pod "96c20c16-b48c-40b5-853c-edddd1225652" (UID: "96c20c16-b48c-40b5-853c-edddd1225652"). InnerVolumeSpecName "kube-api-access-59skl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:08.935227 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:08.935186 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59skl\" (UniqueName: \"kubernetes.io/projected/96c20c16-b48c-40b5-853c-edddd1225652-kube-api-access-59skl\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:08.935227 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:08.935228 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96c20c16-b48c-40b5-853c-edddd1225652-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:08.956419 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:08.956389 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5e2d03-757556fddc-j9gph" Nov 01 16:30:09.036718 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:09.036681 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57dbe32d-78fb-4168-a5f8-5fa0a38f2e2b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57dbe32d-78fb-4168-a5f8-5fa0a38f2e2b-plugins-config-volume\") pod \"57dbe32d-78fb-4168-a5f8-5fa0a38f2e2b\" (UID: \"57dbe32d-78fb-4168-a5f8-5fa0a38f2e2b\") " Nov 01 16:30:09.036923 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:09.036774 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57dbe32d-78fb-4168-a5f8-5fa0a38f2e2b-kube-api-access-4l5k8\" (UniqueName: \"kubernetes.io/projected/57dbe32d-78fb-4168-a5f8-5fa0a38f2e2b-kube-api-access-4l5k8\") pod \"57dbe32d-78fb-4168-a5f8-5fa0a38f2e2b\" (UID: \"57dbe32d-78fb-4168-a5f8-5fa0a38f2e2b\") " Nov 01 16:30:09.037172 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:09.037140 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57dbe32d-78fb-4168-a5f8-5fa0a38f2e2b-plugins-config-volume" pod "57dbe32d-78fb-4168-a5f8-5fa0a38f2e2b" (UID: "57dbe32d-78fb-4168-a5f8-5fa0a38f2e2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:30:09.037662 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:09.037639 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57dbe32d-78fb-4168-a5f8-5fa0a38f2e2b-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:09.039396 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:09.039371 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57dbe32d-78fb-4168-a5f8-5fa0a38f2e2b-kube-api-access-4l5k8" pod "57dbe32d-78fb-4168-a5f8-5fa0a38f2e2b" (UID: "57dbe32d-78fb-4168-a5f8-5fa0a38f2e2b"). InnerVolumeSpecName "kube-api-access-4l5k8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:09.139483 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:09.139401 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l5k8\" (UniqueName: \"kubernetes.io/projected/57dbe32d-78fb-4168-a5f8-5fa0a38f2e2b-kube-api-access-4l5k8\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:09.755499 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:09.755459 4363 generic.go:356] "Generic (PLEG): container finished" podID="6097f042-a435-451a-8fba-818eeef14f43" containerID="c5559907747a696a36e741066171b1c3ee79328c4c0ee47ab89fec29e0cb0ca9" exitCode=0 Nov 01 16:30:09.755966 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:09.755553 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d121-k6r77" event={"ID":"6097f042-a435-451a-8fba-818eeef14f43","Type":"ContainerDied","Data":"c5559907747a696a36e741066171b1c3ee79328c4c0ee47ab89fec29e0cb0ca9"} Nov 01 16:30:09.757182 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:09.757164 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5e2d03-cbd9bb6f9-bbprx" Nov 01 16:30:09.757263 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:09.757207 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5e2d03-cbd9bb6f9-bbprx" event={"ID":"96c20c16-b48c-40b5-853c-edddd1225652","Type":"ContainerDied","Data":"7db87934577925cfa89c63f8165023624730a14d82f86d43aa15e62dc0b26d29"} Nov 01 16:30:09.757263 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:09.757253 4363 scope.go:122] "RemoveContainer" containerID="5657a369b4632c5bc020f3a2169db572af33e2980861b959df88957d23a9f99d" Nov 01 16:30:09.758657 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:09.758642 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5e2d03-757556fddc-j9gph" Nov 01 16:30:09.758770 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:09.758699 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5e2d03-757556fddc-j9gph" event={"ID":"57dbe32d-78fb-4168-a5f8-5fa0a38f2e2b","Type":"ContainerDied","Data":"7860d62f3148186e78814fd0206af3ecdda28b241dfd2e9f930fe83660c88f3e"} Nov 01 16:30:09.845630 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:09.845602 4363 scope.go:122] "RemoveContainer" containerID="44fdf0f6ebd17a0dd4f8dbc5a4b72bb1afe67bb9be6ce8ecd1f6f08822eaccf9" Nov 01 16:30:09.846344 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:09.846313 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5e2d03-cbd9bb6f9-bbprx"] Nov 01 16:30:09.853707 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:09.853673 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5e2d03-cbd9bb6f9-bbprx"] Nov 01 16:30:09.858242 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:09.858216 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5e2d03-757556fddc-j9gph"] Nov 01 16:30:09.864226 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:09.863774 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5e2d03-757556fddc-j9gph"] Nov 01 16:30:10.878854 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:10.877238 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57dbe32d-78fb-4168-a5f8-5fa0a38f2e2b" path="/var/lib/kubelet/pods/57dbe32d-78fb-4168-a5f8-5fa0a38f2e2b/volumes" Nov 01 16:30:10.878854 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:10.877970 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96c20c16-b48c-40b5-853c-edddd1225652" path="/var/lib/kubelet/pods/96c20c16-b48c-40b5-853c-edddd1225652/volumes" Nov 01 16:30:11.137207 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:11.137175 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d121-k6r77" Nov 01 16:30:11.222150 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:11.222106 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-6f8b7f8fdc-qfwn9"] Nov 01 16:30:11.223441 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:11.223402 4363 cpu_manager.go:414] "containerMap: removing container" podUID="57dbe32d-78fb-4168-a5f8-5fa0a38f2e2b" containerName="payload-pre-processing" Nov 01 16:30:11.224563 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:11.224533 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57dbe32d-78fb-4168-a5f8-5fa0a38f2e2b" containerName="payload-pre-processing" Nov 01 16:30:11.224684 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:11.224631 4363 cpu_manager.go:414] "containerMap: removing container" podUID="6097f042-a435-451a-8fba-818eeef14f43" containerName="revoke-keys" Nov 01 16:30:11.224684 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:11.224642 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6097f042-a435-451a-8fba-818eeef14f43" containerName="revoke-keys" Nov 01 16:30:11.224684 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:11.224683 4363 cpu_manager.go:414] "containerMap: removing container" podUID="96c20c16-b48c-40b5-853c-edddd1225652" containerName="payload-processing" Nov 01 16:30:11.224873 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:11.224692 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96c20c16-b48c-40b5-853c-edddd1225652" containerName="payload-processing" Nov 01 16:30:11.224972 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:11.224954 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="57dbe32d-78fb-4168-a5f8-5fa0a38f2e2b" containerName="payload-pre-processing" Nov 01 16:30:11.225045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:11.224990 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="6097f042-a435-451a-8fba-818eeef14f43" containerName="revoke-keys" Nov 01 16:30:11.225045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:11.225007 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="96c20c16-b48c-40b5-853c-edddd1225652" containerName="payload-processing" Nov 01 16:30:11.230262 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:11.230243 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-6f8b7f8fdc-qfwn9" Nov 01 16:30:11.230725 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:11.230701 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-6f8b7f8fdc-qfwn9"] Nov 01 16:30:11.237155 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:11.237116 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6097f042-a435-451a-8fba-818eeef14f43-kube-api-access-5svl4\" (UniqueName: \"kubernetes.io/projected/6097f042-a435-451a-8fba-818eeef14f43-kube-api-access-5svl4\") pod \"6097f042-a435-451a-8fba-818eeef14f43\" (UID: \"6097f042-a435-451a-8fba-818eeef14f43\") " Nov 01 16:30:11.237415 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:11.237341 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6097f042-a435-451a-8fba-818eeef14f43-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6097f042-a435-451a-8fba-818eeef14f43-maas-api-openshift-ca\") pod \"6097f042-a435-451a-8fba-818eeef14f43\" (UID: \"6097f042-a435-451a-8fba-818eeef14f43\") " Nov 01 16:30:11.237676 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:11.237657 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6097f042-a435-451a-8fba-818eeef14f43-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6097f042-a435-451a-8fba-818eeef14f43-maas-api-xks-ca\") pod \"6097f042-a435-451a-8fba-818eeef14f43\" (UID: \"6097f042-a435-451a-8fba-818eeef14f43\") " Nov 01 16:30:11.239352 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:11.239325 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-6b6cfc\"" Nov 01 16:30:11.239491 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:11.239473 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-6b6cfc-dockercfg-gdkq6\"" Nov 01 16:30:11.239921 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:11.239900 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6097f042-a435-451a-8fba-818eeef14f43-maas-api-openshift-ca" pod "6097f042-a435-451a-8fba-818eeef14f43" (UID: "6097f042-a435-451a-8fba-818eeef14f43"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:30:11.241862 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:11.241834 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6097f042-a435-451a-8fba-818eeef14f43-maas-api-xks-ca" pod "6097f042-a435-451a-8fba-818eeef14f43" (UID: "6097f042-a435-451a-8fba-818eeef14f43"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:30:11.245943 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:11.245916 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6097f042-a435-451a-8fba-818eeef14f43-kube-api-access-5svl4" pod "6097f042-a435-451a-8fba-818eeef14f43" (UID: "6097f042-a435-451a-8fba-818eeef14f43"). InnerVolumeSpecName "kube-api-access-5svl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:11.367319 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:11.346208 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2cls\" (UniqueName: \"kubernetes.io/projected/2432fb41-4632-4522-a4c1-cdfcfcd9ea43-kube-api-access-j2cls\") pod \"payload-pre-processing-e2e-ipp-b-6b6cfc-6f8b7f8fdc-qfwn9\" (UID: \"2432fb41-4632-4522-a4c1-cdfcfcd9ea43\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-6f8b7f8fdc-qfwn9" Nov 01 16:30:11.367319 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:11.346314 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2432fb41-4632-4522-a4c1-cdfcfcd9ea43-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6b6cfc-6f8b7f8fdc-qfwn9\" (UID: \"2432fb41-4632-4522-a4c1-cdfcfcd9ea43\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-6f8b7f8fdc-qfwn9" Nov 01 16:30:11.367319 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:11.346631 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5svl4\" (UniqueName: \"kubernetes.io/projected/6097f042-a435-451a-8fba-818eeef14f43-kube-api-access-5svl4\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:11.367319 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:11.346651 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6097f042-a435-451a-8fba-818eeef14f43-maas-api-openshift-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:11.367319 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:11.346666 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6097f042-a435-451a-8fba-818eeef14f43-maas-api-xks-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:11.472841 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:11.455748 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2cls\" (UniqueName: \"kubernetes.io/projected/2432fb41-4632-4522-a4c1-cdfcfcd9ea43-kube-api-access-j2cls\") pod \"payload-pre-processing-e2e-ipp-b-6b6cfc-6f8b7f8fdc-qfwn9\" (UID: \"2432fb41-4632-4522-a4c1-cdfcfcd9ea43\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-6f8b7f8fdc-qfwn9" Nov 01 16:30:11.472841 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:11.455851 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2432fb41-4632-4522-a4c1-cdfcfcd9ea43-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6b6cfc-6f8b7f8fdc-qfwn9\" (UID: \"2432fb41-4632-4522-a4c1-cdfcfcd9ea43\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-6f8b7f8fdc-qfwn9" Nov 01 16:30:11.472841 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:11.468577 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2432fb41-4632-4522-a4c1-cdfcfcd9ea43-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6b6cfc-6f8b7f8fdc-qfwn9\" (UID: \"2432fb41-4632-4522-a4c1-cdfcfcd9ea43\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-6f8b7f8fdc-qfwn9" Nov 01 16:30:11.530041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:11.526916 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2cls\" (UniqueName: \"kubernetes.io/projected/2432fb41-4632-4522-a4c1-cdfcfcd9ea43-kube-api-access-j2cls\") pod \"payload-pre-processing-e2e-ipp-b-6b6cfc-6f8b7f8fdc-qfwn9\" (UID: \"2432fb41-4632-4522-a4c1-cdfcfcd9ea43\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-6f8b7f8fdc-qfwn9" Nov 01 16:30:11.764306 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:11.764202 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-6f8b7f8fdc-qfwn9" Nov 01 16:30:11.944315 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:11.940238 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d121-k6r77" event={"ID":"6097f042-a435-451a-8fba-818eeef14f43","Type":"ContainerDied","Data":"715ffd15f8b50383597573af7b8192677265d3059568be68fcea3404b05c3ddd"} Nov 01 16:30:11.944315 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:11.940296 4363 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="715ffd15f8b50383597573af7b8192677265d3059568be68fcea3404b05c3ddd" Nov 01 16:30:11.944315 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:11.940374 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d121-k6r77" Nov 01 16:30:11.955514 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:11.947191 4363 generic.go:356] "Generic (PLEG): container finished" podID="19456502-09eb-4dc7-8877-395f1b5e7218" containerID="78c2dd5df3b02f6d6dd5d479cbb2392b759b43038a569cc0550ee78de1e5ee13" exitCode=0 Nov 01 16:30:11.955514 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:11.947273 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" event={"ID":"19456502-09eb-4dc7-8877-395f1b5e7218","Type":"ContainerDied","Data":"78c2dd5df3b02f6d6dd5d479cbb2392b759b43038a569cc0550ee78de1e5ee13"} Nov 01 16:30:11.955514 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:11.947321 4363 scope.go:122] "RemoveContainer" containerID="a5bfdb3891ac394f0b406fe74719eba6447cb1299f7d34d424160ae4e9975d32" Nov 01 16:30:12.015043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:12.001227 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" Nov 01 16:30:12.041246 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:12.041194 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19456502-09eb-4dc7-8877-395f1b5e7218-istio-token\" (UniqueName: \"kubernetes.io/projected/19456502-09eb-4dc7-8877-395f1b5e7218-istio-token\") pod \"19456502-09eb-4dc7-8877-395f1b5e7218\" (UID: \"19456502-09eb-4dc7-8877-395f1b5e7218\") " Nov 01 16:30:12.041895 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:12.041871 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19456502-09eb-4dc7-8877-395f1b5e7218-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19456502-09eb-4dc7-8877-395f1b5e7218-istio-podinfo\") pod \"19456502-09eb-4dc7-8877-395f1b5e7218\" (UID: \"19456502-09eb-4dc7-8877-395f1b5e7218\") " Nov 01 16:30:12.042554 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:12.042532 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19456502-09eb-4dc7-8877-395f1b5e7218-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19456502-09eb-4dc7-8877-395f1b5e7218-credential-socket\") pod \"19456502-09eb-4dc7-8877-395f1b5e7218\" (UID: \"19456502-09eb-4dc7-8877-395f1b5e7218\") " Nov 01 16:30:12.042742 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:12.042727 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19456502-09eb-4dc7-8877-395f1b5e7218-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19456502-09eb-4dc7-8877-395f1b5e7218-workload-certs\") pod \"19456502-09eb-4dc7-8877-395f1b5e7218\" (UID: \"19456502-09eb-4dc7-8877-395f1b5e7218\") " Nov 01 16:30:12.043370 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:12.043108 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19456502-09eb-4dc7-8877-395f1b5e7218-workload-certs" pod "19456502-09eb-4dc7-8877-395f1b5e7218" (UID: "19456502-09eb-4dc7-8877-395f1b5e7218"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:30:12.044859 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:12.044413 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19456502-09eb-4dc7-8877-395f1b5e7218-istio-token" pod "19456502-09eb-4dc7-8877-395f1b5e7218" (UID: "19456502-09eb-4dc7-8877-395f1b5e7218"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:12.044859 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:12.044638 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19456502-09eb-4dc7-8877-395f1b5e7218-istio-podinfo" pod "19456502-09eb-4dc7-8877-395f1b5e7218" (UID: "19456502-09eb-4dc7-8877-395f1b5e7218"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:30:12.045578 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:12.045556 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19456502-09eb-4dc7-8877-395f1b5e7218-credential-socket" pod "19456502-09eb-4dc7-8877-395f1b5e7218" (UID: "19456502-09eb-4dc7-8877-395f1b5e7218"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:30:12.045746 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:12.045705 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19456502-09eb-4dc7-8877-395f1b5e7218-kube-api-access-5zc89\" (UniqueName: \"kubernetes.io/projected/19456502-09eb-4dc7-8877-395f1b5e7218-kube-api-access-5zc89\") pod \"19456502-09eb-4dc7-8877-395f1b5e7218\" (UID: \"19456502-09eb-4dc7-8877-395f1b5e7218\") " Nov 01 16:30:12.045943 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:12.045926 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19456502-09eb-4dc7-8877-395f1b5e7218-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19456502-09eb-4dc7-8877-395f1b5e7218-istio-envoy\") pod \"19456502-09eb-4dc7-8877-395f1b5e7218\" (UID: \"19456502-09eb-4dc7-8877-395f1b5e7218\") " Nov 01 16:30:12.046554 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:12.046537 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19456502-09eb-4dc7-8877-395f1b5e7218-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19456502-09eb-4dc7-8877-395f1b5e7218-istio-data\") pod \"19456502-09eb-4dc7-8877-395f1b5e7218\" (UID: \"19456502-09eb-4dc7-8877-395f1b5e7218\") " Nov 01 16:30:12.046954 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:12.046937 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19456502-09eb-4dc7-8877-395f1b5e7218-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19456502-09eb-4dc7-8877-395f1b5e7218-istiod-ca-cert\") pod \"19456502-09eb-4dc7-8877-395f1b5e7218\" (UID: \"19456502-09eb-4dc7-8877-395f1b5e7218\") " Nov 01 16:30:12.047511 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:12.047494 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19456502-09eb-4dc7-8877-395f1b5e7218-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19456502-09eb-4dc7-8877-395f1b5e7218-workload-socket\") pod \"19456502-09eb-4dc7-8877-395f1b5e7218\" (UID: \"19456502-09eb-4dc7-8877-395f1b5e7218\") " Nov 01 16:30:12.048502 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:12.046874 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19456502-09eb-4dc7-8877-395f1b5e7218-istio-data" pod "19456502-09eb-4dc7-8877-395f1b5e7218" (UID: "19456502-09eb-4dc7-8877-395f1b5e7218"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:30:12.048502 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:12.047375 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19456502-09eb-4dc7-8877-395f1b5e7218-istiod-ca-cert" pod "19456502-09eb-4dc7-8877-395f1b5e7218" (UID: "19456502-09eb-4dc7-8877-395f1b5e7218"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:30:12.048502 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:12.047860 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19456502-09eb-4dc7-8877-395f1b5e7218-workload-socket" pod "19456502-09eb-4dc7-8877-395f1b5e7218" (UID: "19456502-09eb-4dc7-8877-395f1b5e7218"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:30:12.048716 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:12.048554 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19456502-09eb-4dc7-8877-395f1b5e7218-istio-envoy" pod "19456502-09eb-4dc7-8877-395f1b5e7218" (UID: "19456502-09eb-4dc7-8877-395f1b5e7218"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:30:12.049149 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:12.049128 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19456502-09eb-4dc7-8877-395f1b5e7218-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:12.049289 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:12.049275 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19456502-09eb-4dc7-8877-395f1b5e7218-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:12.049429 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:12.049413 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19456502-09eb-4dc7-8877-395f1b5e7218-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:12.049584 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:12.049570 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19456502-09eb-4dc7-8877-395f1b5e7218-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:12.050714 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:12.049682 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19456502-09eb-4dc7-8877-395f1b5e7218-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:12.050917 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:12.050899 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19456502-09eb-4dc7-8877-395f1b5e7218-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:12.051140 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:12.051047 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19456502-09eb-4dc7-8877-395f1b5e7218-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:12.051960 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:12.051941 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19456502-09eb-4dc7-8877-395f1b5e7218-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:12.058531 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:12.058493 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19456502-09eb-4dc7-8877-395f1b5e7218-kube-api-access-5zc89" pod "19456502-09eb-4dc7-8877-395f1b5e7218" (UID: "19456502-09eb-4dc7-8877-395f1b5e7218"). InnerVolumeSpecName "kube-api-access-5zc89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:12.160052 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:12.159960 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zc89\" (UniqueName: \"kubernetes.io/projected/19456502-09eb-4dc7-8877-395f1b5e7218-kube-api-access-5zc89\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:12.311603 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:12.311568 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-6f8b7f8fdc-qfwn9"] Nov 01 16:30:12.956362 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:12.956308 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-6f8b7f8fdc-qfwn9" event={"ID":"2432fb41-4632-4522-a4c1-cdfcfcd9ea43","Type":"ContainerStarted","Data":"2763726f208d5318ded4fb777659b7530dd1480f44445a9d988c76ae598d41ef"} Nov 01 16:30:12.956805 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:12.956409 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-6f8b7f8fdc-qfwn9" Nov 01 16:30:12.956805 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:12.956441 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-6f8b7f8fdc-qfwn9" event={"ID":"2432fb41-4632-4522-a4c1-cdfcfcd9ea43","Type":"ContainerStarted","Data":"a1699ccb618f7144e4594c4142dabbddd4edd592423304b6b6154aca999224c9"} Nov 01 16:30:12.957985 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:12.957959 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" Nov 01 16:30:12.958156 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:12.957984 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz" event={"ID":"19456502-09eb-4dc7-8877-395f1b5e7218","Type":"ContainerDied","Data":"9e804e12cc576640c0e8b1dc53da53a1553ae04ae34311662d6329928754f346"} Nov 01 16:30:12.958156 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:12.958076 4363 scope.go:122] "RemoveContainer" containerID="78c2dd5df3b02f6d6dd5d479cbb2392b759b43038a569cc0550ee78de1e5ee13" Nov 01 16:30:13.037736 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:13.037678 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-6f8b7f8fdc-qfwn9" podStartSLOduration=2.0376634 podStartE2EDuration="2.0376634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:30:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:30:12.994283649 +0000 UTC m=+2546.326021301" watchObservedRunningTime="2026-11-01 16:30:13.0376634 +0000 UTC m=+2546.369401050" Nov 01 16:30:13.056851 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:13.056822 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz"] Nov 01 16:30:13.063496 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:13.063470 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-tt9zz"] Nov 01 16:30:14.802222 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:14.802180 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19456502-09eb-4dc7-8877-395f1b5e7218" path="/var/lib/kubelet/pods/19456502-09eb-4dc7-8877-395f1b5e7218/volumes" Nov 01 16:30:15.791275 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:15.791246 4363 scope.go:122] "RemoveContainer" containerID="bf093ad2ce9878574e5cb34b04b3a631b39f51041f8501514b7501096882428d" Nov 01 16:30:15.791496 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:15.791461 4363 scope.go:122] "RemoveContainer" containerID="6eb76660363a99a1621ceeb8af832c612399469e04b8d10c88c61ad16b9885f0" Nov 01 16:30:15.792658 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:15.791986 4363 scope.go:122] "RemoveContainer" containerID="62634e1cdc0137521d9aae0e262e834c1b7895b0e3535cce0de8d8dcc23774c2" Nov 01 16:30:15.792658 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:15.792120 4363 scope.go:122] "RemoveContainer" containerID="a26c9dc5112f36ec783326824458696b1fe7f1cc659a5cafd81e5d3fbece4164" Nov 01 16:30:16.802063 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:16.801404 4363 scope.go:122] "RemoveContainer" containerID="0b13f038fd7dc35b1ffd3dacdbb92d7033f23cb56dc4197b776e632fad47c9fd" Nov 01 16:30:16.816308 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:16.815587 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d121-5595856559-g7tk9"] Nov 01 16:30:16.884318 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:16.883690 4363 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-shared-b-w0-c8d121: configmap "payload-processing-plugins-e2e-shared-b-w0-c8d121" not found Nov 01 16:30:16.906482 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:16.903096 4363 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/55c3d81b-79de-44cb-9838-8b07da7c911b-plugins-config-volume podName:55c3d81b-79de-44cb-9838-8b07da7c911b nodeName:}" failed. No retries permitted until 2026-11-01 16:30:17.403068178 +0000 UTC m=+2550.734805810 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/55c3d81b-79de-44cb-9838-8b07da7c911b-plugins-config-volume") pod "payload-pre-processing-e2e-shared-b-w0-c8d121-5595856559-g7tk9" (UID: "55c3d81b-79de-44cb-9838-8b07da7c911b") : configmap "payload-processing-plugins-e2e-shared-b-w0-c8d121" not found Nov 01 16:30:17.000950 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:17.000916 4363 generic.go:356] "Generic (PLEG): container finished" podID="cfb264a2-0df1-4b83-bd6d-a355f75a2aca" containerID="0ff6bcae89d336d435baa7f3e1ecf6c027ebec3d68f3582d0d32b235aeb7c1f8" exitCode=60 Nov 01 16:30:17.001154 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:17.000989 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892510-cg9fh" event={"ID":"cfb264a2-0df1-4b83-bd6d-a355f75a2aca","Type":"ContainerDied","Data":"0ff6bcae89d336d435baa7f3e1ecf6c027ebec3d68f3582d0d32b235aeb7c1f8"} Nov 01 16:30:17.001154 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:17.001070 4363 scope.go:122] "RemoveContainer" containerID="a26c9dc5112f36ec783326824458696b1fe7f1cc659a5cafd81e5d3fbece4164" Nov 01 16:30:17.001481 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:17.001465 4363 scope.go:122] "RemoveContainer" containerID="0ff6bcae89d336d435baa7f3e1ecf6c027ebec3d68f3582d0d32b235aeb7c1f8" Nov 01 16:30:17.001860 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:17.001838 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892510-cg9fh_odh-ai-gateway-infra(cfb264a2-0df1-4b83-bd6d-a355f75a2aca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892510-cg9fh" podUID="cfb264a2-0df1-4b83-bd6d-a355f75a2aca" Nov 01 16:30:17.003572 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:17.003547 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fd73f-29892510-768f5" event={"ID":"1817f9a5-677f-43b6-a8d0-b80a92870530","Type":"ContainerStarted","Data":"53680b713ee0201ca9ce021d0c2459de79b3bde47ad30de378d162f92e828c5d"} Nov 01 16:30:17.006008 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:17.005982 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892510-947xq" event={"ID":"07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d","Type":"ContainerStarted","Data":"5c423627148c8bab3182239673836875c167b5df1e3745d526c1de09af9f6fdb"} Nov 01 16:30:17.008511 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:17.008485 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5ccd7b-29892510-9v7dk" event={"ID":"89cbf816-3afe-455a-8e90-5606573018b0","Type":"ContainerStarted","Data":"e6b8efe1b5af14a2fef76c4941fb7b2e6d26a57e8b18c142a7b08eeb408a6f7c"} Nov 01 16:30:17.501963 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:17.501922 4363 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-shared-b-w0-c8d121: configmap "payload-processing-plugins-e2e-shared-b-w0-c8d121" not found Nov 01 16:30:17.502157 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:17.502010 4363 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/55c3d81b-79de-44cb-9838-8b07da7c911b-plugins-config-volume podName:55c3d81b-79de-44cb-9838-8b07da7c911b nodeName:}" failed. No retries permitted until 2026-11-01 16:30:18.501989807 +0000 UTC m=+2551.833727448 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/55c3d81b-79de-44cb-9838-8b07da7c911b-plugins-config-volume") pod "payload-pre-processing-e2e-shared-b-w0-c8d121-5595856559-g7tk9" (UID: "55c3d81b-79de-44cb-9838-8b07da7c911b") : configmap "payload-processing-plugins-e2e-shared-b-w0-c8d121" not found Nov 01 16:30:17.790527 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:17.790498 4363 scope.go:122] "RemoveContainer" containerID="4af16d31c7a66cefc30fa843d1a2c8e0ed0aa0503cfbd9989794f4ca018c7db0" Nov 01 16:30:17.790743 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:17.790720 4363 scope.go:122] "RemoveContainer" containerID="4534b751b43106ea725cec2c13527b2e5d6975b9d981b557fba81ad950db324a" Nov 01 16:30:17.790821 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:17.790781 4363 scope.go:122] "RemoveContainer" containerID="ba46e08edd70040f475720e9fa239126d5f5b81fdfc280dedd2609e003acfc71" Nov 01 16:30:18.020721 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:18.020688 4363 generic.go:356] "Generic (PLEG): container finished" podID="1817f9a5-677f-43b6-a8d0-b80a92870530" containerID="53680b713ee0201ca9ce021d0c2459de79b3bde47ad30de378d162f92e828c5d" exitCode=60 Nov 01 16:30:18.021177 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:18.020769 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fd73f-29892510-768f5" event={"ID":"1817f9a5-677f-43b6-a8d0-b80a92870530","Type":"ContainerDied","Data":"53680b713ee0201ca9ce021d0c2459de79b3bde47ad30de378d162f92e828c5d"} Nov 01 16:30:18.021177 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:18.020849 4363 scope.go:122] "RemoveContainer" containerID="bf093ad2ce9878574e5cb34b04b3a631b39f51041f8501514b7501096882428d" Nov 01 16:30:18.021326 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:18.021269 4363 scope.go:122] "RemoveContainer" containerID="53680b713ee0201ca9ce021d0c2459de79b3bde47ad30de378d162f92e828c5d" Nov 01 16:30:18.021698 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:18.021671 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-3fd73f-29892510-768f5_odh-ai-gateway-infra(1817f9a5-677f-43b6-a8d0-b80a92870530)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fd73f-29892510-768f5" podUID="1817f9a5-677f-43b6-a8d0-b80a92870530" Nov 01 16:30:18.023462 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:18.023442 4363 generic.go:356] "Generic (PLEG): container finished" podID="07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d" containerID="5c423627148c8bab3182239673836875c167b5df1e3745d526c1de09af9f6fdb" exitCode=60 Nov 01 16:30:18.023560 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:18.023533 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892510-947xq" event={"ID":"07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d","Type":"ContainerDied","Data":"5c423627148c8bab3182239673836875c167b5df1e3745d526c1de09af9f6fdb"} Nov 01 16:30:18.024595 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:18.023942 4363 scope.go:122] "RemoveContainer" containerID="5c423627148c8bab3182239673836875c167b5df1e3745d526c1de09af9f6fdb" Nov 01 16:30:18.024595 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:18.024303 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892510-947xq_odh-ai-gateway-infra(07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892510-947xq" podUID="07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d" Nov 01 16:30:18.025604 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:18.025585 4363 generic.go:356] "Generic (PLEG): container finished" podID="89cbf816-3afe-455a-8e90-5606573018b0" containerID="e6b8efe1b5af14a2fef76c4941fb7b2e6d26a57e8b18c142a7b08eeb408a6f7c" exitCode=60 Nov 01 16:30:18.025701 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:18.025617 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5ccd7b-29892510-9v7dk" event={"ID":"89cbf816-3afe-455a-8e90-5606573018b0","Type":"ContainerDied","Data":"e6b8efe1b5af14a2fef76c4941fb7b2e6d26a57e8b18c142a7b08eeb408a6f7c"} Nov 01 16:30:18.026011 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:18.025989 4363 scope.go:122] "RemoveContainer" containerID="e6b8efe1b5af14a2fef76c4941fb7b2e6d26a57e8b18c142a7b08eeb408a6f7c" Nov 01 16:30:18.026557 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:18.026532 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-5ccd7b-29892510-9v7dk_odh-ai-gateway-infra(89cbf816-3afe-455a-8e90-5606573018b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5ccd7b-29892510-9v7dk" podUID="89cbf816-3afe-455a-8e90-5606573018b0" Nov 01 16:30:18.030742 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:18.030714 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892510-zjcr9" event={"ID":"954da573-4dff-4a25-a180-17fe21a92841","Type":"ContainerStarted","Data":"7d9e943425d0c8df3b03392276390827409b6c76e2b4e7c70a34047132cc537d"} Nov 01 16:30:18.032900 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:18.032878 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892510-cg9fh"] Nov 01 16:30:18.041375 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:18.041350 4363 scope.go:122] "RemoveContainer" containerID="6eb76660363a99a1621ceeb8af832c612399469e04b8d10c88c61ad16b9885f0" Nov 01 16:30:18.065859 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:18.065834 4363 scope.go:122] "RemoveContainer" containerID="62634e1cdc0137521d9aae0e262e834c1b7895b0e3535cce0de8d8dcc23774c2" Nov 01 16:30:18.204924 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:18.147604 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5ccd7b-29892510-9v7dk"] Nov 01 16:30:18.204924 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:18.178224 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fd73f-29892510-768f5"] Nov 01 16:30:18.208335 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:18.208304 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892510-947xq"] Nov 01 16:30:18.319433 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:18.319390 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:30:18.400835 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:18.400766 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892510-cg9fh" Nov 01 16:30:18.521391 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:18.521354 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfb264a2-0df1-4b83-bd6d-a355f75a2aca-kube-api-access-w85dl\" (UniqueName: \"kubernetes.io/projected/cfb264a2-0df1-4b83-bd6d-a355f75a2aca-kube-api-access-w85dl\") pod \"cfb264a2-0df1-4b83-bd6d-a355f75a2aca\" (UID: \"cfb264a2-0df1-4b83-bd6d-a355f75a2aca\") " Nov 01 16:30:18.521591 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:18.521493 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfb264a2-0df1-4b83-bd6d-a355f75a2aca-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb264a2-0df1-4b83-bd6d-a355f75a2aca-maas-api-service-ca\") pod \"cfb264a2-0df1-4b83-bd6d-a355f75a2aca\" (UID: \"cfb264a2-0df1-4b83-bd6d-a355f75a2aca\") " Nov 01 16:30:18.522665 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:18.522190 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfb264a2-0df1-4b83-bd6d-a355f75a2aca-maas-api-service-ca" pod "cfb264a2-0df1-4b83-bd6d-a355f75a2aca" (UID: "cfb264a2-0df1-4b83-bd6d-a355f75a2aca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:30:18.522665 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:18.522511 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb264a2-0df1-4b83-bd6d-a355f75a2aca-maas-api-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:18.522665 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:18.522587 4363 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-shared-b-w0-c8d121: configmap "payload-processing-plugins-e2e-shared-b-w0-c8d121" not found Nov 01 16:30:18.522665 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:18.522660 4363 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/55c3d81b-79de-44cb-9838-8b07da7c911b-plugins-config-volume podName:55c3d81b-79de-44cb-9838-8b07da7c911b nodeName:}" failed. No retries permitted until 2026-11-01 16:30:20.52263911 +0000 UTC m=+2553.854376751 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/55c3d81b-79de-44cb-9838-8b07da7c911b-plugins-config-volume") pod "payload-pre-processing-e2e-shared-b-w0-c8d121-5595856559-g7tk9" (UID: "55c3d81b-79de-44cb-9838-8b07da7c911b") : configmap "payload-processing-plugins-e2e-shared-b-w0-c8d121" not found Nov 01 16:30:18.524334 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:18.524305 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfb264a2-0df1-4b83-bd6d-a355f75a2aca-kube-api-access-w85dl" pod "cfb264a2-0df1-4b83-bd6d-a355f75a2aca" (UID: "cfb264a2-0df1-4b83-bd6d-a355f75a2aca"). InnerVolumeSpecName "kube-api-access-w85dl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:18.623821 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:18.623783 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w85dl\" (UniqueName: \"kubernetes.io/projected/cfb264a2-0df1-4b83-bd6d-a355f75a2aca-kube-api-access-w85dl\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:18.790828 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:18.790805 4363 scope.go:122] "RemoveContainer" containerID="f8aa6f0f37be9228c6a938c8d18ed9a8609fb8eb38fb58fbed58ad8cb1de36a6" Nov 01 16:30:19.046599 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.046574 4363 generic.go:356] "Generic (PLEG): container finished" podID="975c2d1d-45a8-4439-923e-59f70d69ea04" containerID="f7b1dfa26686314214737b339881b2b0bdb34a0fda9652d6fd07fdda35d53132" exitCode=60 Nov 01 16:30:19.047104 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.046643 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2b94b5-29892510-9n2h2" event={"ID":"975c2d1d-45a8-4439-923e-59f70d69ea04","Type":"ContainerDied","Data":"f7b1dfa26686314214737b339881b2b0bdb34a0fda9652d6fd07fdda35d53132"} Nov 01 16:30:19.047104 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.046716 4363 scope.go:122] "RemoveContainer" containerID="4af16d31c7a66cefc30fa843d1a2c8e0ed0aa0503cfbd9989794f4ca018c7db0" Nov 01 16:30:19.047655 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.047638 4363 scope.go:122] "RemoveContainer" containerID="f7b1dfa26686314214737b339881b2b0bdb34a0fda9652d6fd07fdda35d53132" Nov 01 16:30:19.048072 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:19.048046 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-2b94b5-29892510-9n2h2_odh-ai-gateway-infra(975c2d1d-45a8-4439-923e-59f70d69ea04)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2b94b5-29892510-9n2h2" podUID="975c2d1d-45a8-4439-923e-59f70d69ea04" Nov 01 16:30:19.050195 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.050173 4363 generic.go:356] "Generic (PLEG): container finished" podID="aa25d700-0231-4655-a067-4d637ea4c5c0" containerID="8b19f353685302190e29d4a1e2f4f9c24be5ed8e9a4f3e21ff1436f53062bd1b" exitCode=60 Nov 01 16:30:19.050303 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.050254 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892510-t259f" event={"ID":"aa25d700-0231-4655-a067-4d637ea4c5c0","Type":"ContainerDied","Data":"8b19f353685302190e29d4a1e2f4f9c24be5ed8e9a4f3e21ff1436f53062bd1b"} Nov 01 16:30:19.050698 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.050680 4363 scope.go:122] "RemoveContainer" containerID="8b19f353685302190e29d4a1e2f4f9c24be5ed8e9a4f3e21ff1436f53062bd1b" Nov 01 16:30:19.051133 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:19.051106 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892510-t259f_odh-ai-gateway-infra(aa25d700-0231-4655-a067-4d637ea4c5c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892510-t259f" podUID="aa25d700-0231-4655-a067-4d637ea4c5c0" Nov 01 16:30:19.052448 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.052424 4363 generic.go:356] "Generic (PLEG): container finished" podID="bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68" containerID="a212b252ec3357713efe563e820807c61191df5d0e1908360fda2514d784d529" exitCode=60 Nov 01 16:30:19.052569 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.052480 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c8537c-29892510-6c55f" event={"ID":"bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68","Type":"ContainerDied","Data":"a212b252ec3357713efe563e820807c61191df5d0e1908360fda2514d784d529"} Nov 01 16:30:19.052935 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.052915 4363 scope.go:122] "RemoveContainer" containerID="a212b252ec3357713efe563e820807c61191df5d0e1908360fda2514d784d529" Nov 01 16:30:19.053376 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:19.053349 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-c8537c-29892510-6c55f_odh-ai-gateway-infra(bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c8537c-29892510-6c55f" podUID="bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68" Nov 01 16:30:19.054854 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.054835 4363 generic.go:356] "Generic (PLEG): container finished" podID="954da573-4dff-4a25-a180-17fe21a92841" containerID="7d9e943425d0c8df3b03392276390827409b6c76e2b4e7c70a34047132cc537d" exitCode=60 Nov 01 16:30:19.054965 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.054921 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892510-zjcr9" event={"ID":"954da573-4dff-4a25-a180-17fe21a92841","Type":"ContainerDied","Data":"7d9e943425d0c8df3b03392276390827409b6c76e2b4e7c70a34047132cc537d"} Nov 01 16:30:19.055392 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.055372 4363 scope.go:122] "RemoveContainer" containerID="7d9e943425d0c8df3b03392276390827409b6c76e2b4e7c70a34047132cc537d" Nov 01 16:30:19.055734 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:19.055708 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892510-zjcr9_odh-ai-gateway-infra(954da573-4dff-4a25-a180-17fe21a92841)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892510-zjcr9" podUID="954da573-4dff-4a25-a180-17fe21a92841" Nov 01 16:30:19.057303 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.057286 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892510-cg9fh" Nov 01 16:30:19.057397 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.057337 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892510-cg9fh" event={"ID":"cfb264a2-0df1-4b83-bd6d-a355f75a2aca","Type":"ContainerDied","Data":"c093a8ccd7bf648104a27f227503586e8b2ffdc63ccc762e3abae6a8ae1aeaad"} Nov 01 16:30:19.059863 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.059838 4363 generic.go:356] "Generic (PLEG): container finished" podID="c8ea572a-0005-426a-9017-de3331cbcbd7" containerID="4f2463064534c9e5b955d6af551a388d04d3253128b18b6059ce214a47d6f1e6" exitCode=0 Nov 01 16:30:19.059957 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.059864 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" event={"ID":"c8ea572a-0005-426a-9017-de3331cbcbd7","Type":"ContainerDied","Data":"4f2463064534c9e5b955d6af551a388d04d3253128b18b6059ce214a47d6f1e6"} Nov 01 16:30:19.060706 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.060323 4363 scope.go:122] "RemoveContainer" containerID="4f2463064534c9e5b955d6af551a388d04d3253128b18b6059ce214a47d6f1e6" Nov 01 16:30:19.062399 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.062371 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:30:19.062501 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.062456 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:30:19.062568 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.062506 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:30:19.077948 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.077739 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892510-zjcr9"] Nov 01 16:30:19.110540 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.110509 4363 scope.go:122] "RemoveContainer" containerID="ba46e08edd70040f475720e9fa239126d5f5b81fdfc280dedd2609e003acfc71" Nov 01 16:30:19.178403 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.162110 4363 scope.go:122] "RemoveContainer" containerID="4534b751b43106ea725cec2c13527b2e5d6975b9d981b557fba81ad950db324a" Nov 01 16:30:19.210322 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.210257 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892510-cg9fh"] Nov 01 16:30:19.211302 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.211281 4363 scope.go:122] "RemoveContainer" containerID="0b13f038fd7dc35b1ffd3dacdbb92d7033f23cb56dc4197b776e632fad47c9fd" Nov 01 16:30:19.243136 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.229162 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892510-cg9fh"] Nov 01 16:30:19.326870 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.326847 4363 scope.go:122] "RemoveContainer" containerID="0ff6bcae89d336d435baa7f3e1ecf6c027ebec3d68f3582d0d32b235aeb7c1f8" Nov 01 16:30:19.376386 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.376356 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892510-947xq" Nov 01 16:30:19.416989 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.416944 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5ccd7b-29892510-9v7dk" Nov 01 16:30:19.432546 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.432509 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d-maas-api-service-ca\") pod \"07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d\" (UID: \"07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d\") " Nov 01 16:30:19.432733 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.432611 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89cbf816-3afe-455a-8e90-5606573018b0-kube-api-access-4l282\" (UniqueName: \"kubernetes.io/projected/89cbf816-3afe-455a-8e90-5606573018b0-kube-api-access-4l282\") pod \"89cbf816-3afe-455a-8e90-5606573018b0\" (UID: \"89cbf816-3afe-455a-8e90-5606573018b0\") " Nov 01 16:30:19.432836 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.432815 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89cbf816-3afe-455a-8e90-5606573018b0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89cbf816-3afe-455a-8e90-5606573018b0-maas-api-service-ca\") pod \"89cbf816-3afe-455a-8e90-5606573018b0\" (UID: \"89cbf816-3afe-455a-8e90-5606573018b0\") " Nov 01 16:30:19.433948 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.433032 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d-kube-api-access-8t8pv\" (UniqueName: \"kubernetes.io/projected/07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d-kube-api-access-8t8pv\") pod \"07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d\" (UID: \"07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d\") " Nov 01 16:30:19.433948 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.433180 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d-maas-api-service-ca" pod "07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d" (UID: "07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:30:19.433948 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.433595 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89cbf816-3afe-455a-8e90-5606573018b0-maas-api-service-ca" pod "89cbf816-3afe-455a-8e90-5606573018b0" (UID: "89cbf816-3afe-455a-8e90-5606573018b0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:30:19.433948 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.433895 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d-maas-api-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:19.433948 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.433918 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89cbf816-3afe-455a-8e90-5606573018b0-maas-api-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:19.436852 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.436830 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d-kube-api-access-8t8pv" pod "07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d" (UID: "07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d"). InnerVolumeSpecName "kube-api-access-8t8pv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:19.437501 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.437476 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89cbf816-3afe-455a-8e90-5606573018b0-kube-api-access-4l282" pod "89cbf816-3afe-455a-8e90-5606573018b0" (UID: "89cbf816-3afe-455a-8e90-5606573018b0"). InnerVolumeSpecName "kube-api-access-4l282". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:19.538692 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.538655 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l282\" (UniqueName: \"kubernetes.io/projected/89cbf816-3afe-455a-8e90-5606573018b0-kube-api-access-4l282\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:19.538846 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.538696 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t8pv\" (UniqueName: \"kubernetes.io/projected/07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d-kube-api-access-8t8pv\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:19.539462 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.539442 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fd73f-29892510-768f5" Nov 01 16:30:19.640282 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.640246 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1817f9a5-677f-43b6-a8d0-b80a92870530-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1817f9a5-677f-43b6-a8d0-b80a92870530-maas-api-service-ca\") pod \"1817f9a5-677f-43b6-a8d0-b80a92870530\" (UID: \"1817f9a5-677f-43b6-a8d0-b80a92870530\") " Nov 01 16:30:19.640282 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.640287 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1817f9a5-677f-43b6-a8d0-b80a92870530-kube-api-access-7p8bl\" (UniqueName: \"kubernetes.io/projected/1817f9a5-677f-43b6-a8d0-b80a92870530-kube-api-access-7p8bl\") pod \"1817f9a5-677f-43b6-a8d0-b80a92870530\" (UID: \"1817f9a5-677f-43b6-a8d0-b80a92870530\") " Nov 01 16:30:19.640702 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.640676 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1817f9a5-677f-43b6-a8d0-b80a92870530-maas-api-service-ca" pod "1817f9a5-677f-43b6-a8d0-b80a92870530" (UID: "1817f9a5-677f-43b6-a8d0-b80a92870530"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:30:19.640840 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.640824 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1817f9a5-677f-43b6-a8d0-b80a92870530-maas-api-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:19.642631 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.642606 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1817f9a5-677f-43b6-a8d0-b80a92870530-kube-api-access-7p8bl" pod "1817f9a5-677f-43b6-a8d0-b80a92870530" (UID: "1817f9a5-677f-43b6-a8d0-b80a92870530"). InnerVolumeSpecName "kube-api-access-7p8bl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:19.742642 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:19.742601 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p8bl\" (UniqueName: \"kubernetes.io/projected/1817f9a5-677f-43b6-a8d0-b80a92870530-kube-api-access-7p8bl\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:20.136582 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:20.136356 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892510-t259f"] Nov 01 16:30:20.143472 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:20.141845 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" event={"ID":"c8ea572a-0005-426a-9017-de3331cbcbd7","Type":"ContainerStarted","Data":"cd303c7ee68e6eb367148892c128c853bc6edeca3af7cdc2bde9fc3e9ac6fe81"} Nov 01 16:30:20.160161 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:20.147219 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fd73f-29892510-768f5" Nov 01 16:30:20.160161 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:20.147245 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fd73f-29892510-768f5" event={"ID":"1817f9a5-677f-43b6-a8d0-b80a92870530","Type":"ContainerDied","Data":"c94d1eefc15d5d633e5c2749b8f96174ddc920fae4fc6ad2e93053c8fbaa6ba6"} Nov 01 16:30:20.160161 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:20.147300 4363 scope.go:122] "RemoveContainer" containerID="53680b713ee0201ca9ce021d0c2459de79b3bde47ad30de378d162f92e828c5d" Nov 01 16:30:20.160161 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:20.151107 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c8537c-29892510-6c55f"] Nov 01 16:30:20.163702 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:20.163672 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2b94b5-29892510-9n2h2"] Nov 01 16:30:20.164888 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:20.164833 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892510-947xq" event={"ID":"07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d","Type":"ContainerDied","Data":"846341ba43997e46302d5884a09d0ef8f4d8c59231847c92f7a8f6a69943acfc"} Nov 01 16:30:20.165008 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:20.164988 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892510-947xq" Nov 01 16:30:20.249318 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:20.249212 4363 scope.go:122] "RemoveContainer" containerID="5c423627148c8bab3182239673836875c167b5df1e3745d526c1de09af9f6fdb" Nov 01 16:30:20.265168 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:20.259460 4363 generic.go:356] "Generic (PLEG): container finished" podID="4e98f9e8-2337-406a-807c-d20dc90a35a2" containerID="0feefb2c27e1c5d70bdd5b069f2a662f86d74cbf133a9f0d860f0df4ef08187e" exitCode=60 Nov 01 16:30:20.265168 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:20.259572 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892510-n7htt" event={"ID":"4e98f9e8-2337-406a-807c-d20dc90a35a2","Type":"ContainerDied","Data":"0feefb2c27e1c5d70bdd5b069f2a662f86d74cbf133a9f0d860f0df4ef08187e"} Nov 01 16:30:20.265168 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:20.260202 4363 scope.go:122] "RemoveContainer" containerID="0feefb2c27e1c5d70bdd5b069f2a662f86d74cbf133a9f0d860f0df4ef08187e" Nov 01 16:30:20.265168 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:20.260638 4363 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-43fac8-29892510-n7htt_odh-ai-gateway-infra(4e98f9e8-2337-406a-807c-d20dc90a35a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892510-n7htt" podUID="4e98f9e8-2337-406a-807c-d20dc90a35a2" Nov 01 16:30:20.272944 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:20.270814 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5ccd7b-29892510-9v7dk" event={"ID":"89cbf816-3afe-455a-8e90-5606573018b0","Type":"ContainerDied","Data":"27d79b5cf0af22eb737704085c51e5204d909528517d17f1cd3966802eb380a8"} Nov 01 16:30:20.272944 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:20.270945 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5ccd7b-29892510-9v7dk" Nov 01 16:30:20.333700 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:20.332311 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fd73f-29892510-768f5"] Nov 01 16:30:20.338537 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:20.338476 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3fd73f-29892510-768f5"] Nov 01 16:30:20.338943 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:20.338905 4363 scope.go:122] "RemoveContainer" containerID="f8aa6f0f37be9228c6a938c8d18ed9a8609fb8eb38fb58fbed58ad8cb1de36a6" Nov 01 16:30:20.356910 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:20.356888 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" Nov 01 16:30:20.358215 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:20.357953 4363 patch_prober.go:29] interesting pod/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body= Nov 01 16:30:20.358215 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:20.358047 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" podUID="c8ea572a-0005-426a-9017-de3331cbcbd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" Nov 01 16:30:20.433699 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:20.433615 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5ccd7b-29892510-9v7dk"] Nov 01 16:30:20.441797 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:20.441766 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5ccd7b-29892510-9v7dk"] Nov 01 16:30:20.441975 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:20.441958 4363 scope.go:122] "RemoveContainer" containerID="e6b8efe1b5af14a2fef76c4941fb7b2e6d26a57e8b18c142a7b08eeb408a6f7c" Nov 01 16:30:20.466410 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:20.466379 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892510-947xq"] Nov 01 16:30:20.473772 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:20.473705 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892510-947xq"] Nov 01 16:30:20.546701 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:20.546671 4363 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-shared-b-w0-c8d121: configmap "payload-processing-plugins-e2e-shared-b-w0-c8d121" not found Nov 01 16:30:20.546850 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:20.546767 4363 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/55c3d81b-79de-44cb-9838-8b07da7c911b-plugins-config-volume podName:55c3d81b-79de-44cb-9838-8b07da7c911b nodeName:}" failed. No retries permitted until 2026-11-01 16:30:24.546745154 +0000 UTC m=+2557.878482782 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/55c3d81b-79de-44cb-9838-8b07da7c911b-plugins-config-volume") pod "payload-pre-processing-e2e-shared-b-w0-c8d121-5595856559-g7tk9" (UID: "55c3d81b-79de-44cb-9838-8b07da7c911b") : configmap "payload-processing-plugins-e2e-shared-b-w0-c8d121" not found Nov 01 16:30:20.595106 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:20.591257 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892510-t259f" Nov 01 16:30:20.647777 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:20.647545 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892510-zjcr9" Nov 01 16:30:20.749772 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:20.749591 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa25d700-0231-4655-a067-4d637ea4c5c0-kube-api-access-w72j4\" (UniqueName: \"kubernetes.io/projected/aa25d700-0231-4655-a067-4d637ea4c5c0-kube-api-access-w72j4\") pod \"aa25d700-0231-4655-a067-4d637ea4c5c0\" (UID: \"aa25d700-0231-4655-a067-4d637ea4c5c0\") " Nov 01 16:30:20.749772 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:20.749672 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/954da573-4dff-4a25-a180-17fe21a92841-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/954da573-4dff-4a25-a180-17fe21a92841-maas-api-service-ca\") pod \"954da573-4dff-4a25-a180-17fe21a92841\" (UID: \"954da573-4dff-4a25-a180-17fe21a92841\") " Nov 01 16:30:20.750125 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:20.749790 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/954da573-4dff-4a25-a180-17fe21a92841-kube-api-access-wsbp2\" (UniqueName: \"kubernetes.io/projected/954da573-4dff-4a25-a180-17fe21a92841-kube-api-access-wsbp2\") pod \"954da573-4dff-4a25-a180-17fe21a92841\" (UID: \"954da573-4dff-4a25-a180-17fe21a92841\") " Nov 01 16:30:20.750125 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:20.749877 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa25d700-0231-4655-a067-4d637ea4c5c0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa25d700-0231-4655-a067-4d637ea4c5c0-maas-api-service-ca\") pod \"aa25d700-0231-4655-a067-4d637ea4c5c0\" (UID: \"aa25d700-0231-4655-a067-4d637ea4c5c0\") " Nov 01 16:30:20.750248 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:20.750188 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/954da573-4dff-4a25-a180-17fe21a92841-maas-api-service-ca" pod "954da573-4dff-4a25-a180-17fe21a92841" (UID: "954da573-4dff-4a25-a180-17fe21a92841"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:30:20.751273 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:20.750563 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/954da573-4dff-4a25-a180-17fe21a92841-maas-api-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:20.751273 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:20.750881 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa25d700-0231-4655-a067-4d637ea4c5c0-maas-api-service-ca" pod "aa25d700-0231-4655-a067-4d637ea4c5c0" (UID: "aa25d700-0231-4655-a067-4d637ea4c5c0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:30:20.753812 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:20.753785 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/954da573-4dff-4a25-a180-17fe21a92841-kube-api-access-wsbp2" pod "954da573-4dff-4a25-a180-17fe21a92841" (UID: "954da573-4dff-4a25-a180-17fe21a92841"). InnerVolumeSpecName "kube-api-access-wsbp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:20.756493 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:20.756133 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c8537c-29892510-6c55f" Nov 01 16:30:20.762426 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:20.762404 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa25d700-0231-4655-a067-4d637ea4c5c0-kube-api-access-w72j4" pod "aa25d700-0231-4655-a067-4d637ea4c5c0" (UID: "aa25d700-0231-4655-a067-4d637ea4c5c0"). InnerVolumeSpecName "kube-api-access-w72j4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:20.972455 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:20.950718 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w72j4\" (UniqueName: \"kubernetes.io/projected/aa25d700-0231-4655-a067-4d637ea4c5c0-kube-api-access-w72j4\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:20.972455 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:20.950753 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsbp2\" (UniqueName: \"kubernetes.io/projected/954da573-4dff-4a25-a180-17fe21a92841-kube-api-access-wsbp2\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:20.972455 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:20.950771 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa25d700-0231-4655-a067-4d637ea4c5c0-maas-api-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:21.009541 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:21.009451 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d" path="/var/lib/kubelet/pods/07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d/volumes" Nov 01 16:30:21.010621 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:21.010602 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1817f9a5-677f-43b6-a8d0-b80a92870530" path="/var/lib/kubelet/pods/1817f9a5-677f-43b6-a8d0-b80a92870530/volumes" Nov 01 16:30:21.011451 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:21.011434 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89cbf816-3afe-455a-8e90-5606573018b0" path="/var/lib/kubelet/pods/89cbf816-3afe-455a-8e90-5606573018b0/volumes" Nov 01 16:30:21.012314 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:21.012298 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfb264a2-0df1-4b83-bd6d-a355f75a2aca" path="/var/lib/kubelet/pods/cfb264a2-0df1-4b83-bd6d-a355f75a2aca/volumes" Nov 01 16:30:21.014588 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:21.014563 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2b94b5-29892510-9n2h2" Nov 01 16:30:21.052245 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:21.052211 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68-maas-api-service-ca\") pod \"bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68\" (UID: \"bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68\") " Nov 01 16:30:21.052579 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:21.052561 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68-kube-api-access-jpjcm\" (UniqueName: \"kubernetes.io/projected/bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68-kube-api-access-jpjcm\") pod \"bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68\" (UID: \"bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68\") " Nov 01 16:30:21.052830 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:21.052800 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68-maas-api-service-ca" pod "bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68" (UID: "bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:30:21.053954 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:21.053932 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68-maas-api-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:21.056311 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:21.056284 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68-kube-api-access-jpjcm" pod "bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68" (UID: "bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68"). InnerVolumeSpecName "kube-api-access-jpjcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:21.154762 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:21.154665 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/975c2d1d-45a8-4439-923e-59f70d69ea04-kube-api-access-fbzq9\" (UniqueName: \"kubernetes.io/projected/975c2d1d-45a8-4439-923e-59f70d69ea04-kube-api-access-fbzq9\") pod \"975c2d1d-45a8-4439-923e-59f70d69ea04\" (UID: \"975c2d1d-45a8-4439-923e-59f70d69ea04\") " Nov 01 16:30:21.154762 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:21.154723 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/975c2d1d-45a8-4439-923e-59f70d69ea04-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/975c2d1d-45a8-4439-923e-59f70d69ea04-maas-api-service-ca\") pod \"975c2d1d-45a8-4439-923e-59f70d69ea04\" (UID: \"975c2d1d-45a8-4439-923e-59f70d69ea04\") " Nov 01 16:30:21.155374 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:21.155161 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/975c2d1d-45a8-4439-923e-59f70d69ea04-maas-api-service-ca" pod "975c2d1d-45a8-4439-923e-59f70d69ea04" (UID: "975c2d1d-45a8-4439-923e-59f70d69ea04"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:30:21.155705 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:21.155679 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpjcm\" (UniqueName: \"kubernetes.io/projected/bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68-kube-api-access-jpjcm\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:21.155800 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:21.155705 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/975c2d1d-45a8-4439-923e-59f70d69ea04-maas-api-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:21.156887 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:21.156862 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/975c2d1d-45a8-4439-923e-59f70d69ea04-kube-api-access-fbzq9" pod "975c2d1d-45a8-4439-923e-59f70d69ea04" (UID: "975c2d1d-45a8-4439-923e-59f70d69ea04"). InnerVolumeSpecName "kube-api-access-fbzq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:21.257581 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:21.257547 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbzq9\" (UniqueName: \"kubernetes.io/projected/975c2d1d-45a8-4439-923e-59f70d69ea04-kube-api-access-fbzq9\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:21.353946 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:21.353783 4363 patch_prober.go:29] interesting pod/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body= Nov 01 16:30:21.353946 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:21.353839 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" podUID="c8ea572a-0005-426a-9017-de3331cbcbd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" Nov 01 16:30:21.363363 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:21.363339 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c8537c-29892510-6c55f" Nov 01 16:30:21.363363 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:21.363354 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c8537c-29892510-6c55f" event={"ID":"bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68","Type":"ContainerDied","Data":"70b05a37cdaa3d596cf267dc378711be0fe08997a5c4c6fed2fd0c11bfa37d16"} Nov 01 16:30:21.363601 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:21.363412 4363 scope.go:122] "RemoveContainer" containerID="a212b252ec3357713efe563e820807c61191df5d0e1908360fda2514d784d529" Nov 01 16:30:21.365011 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:21.364987 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892510-zjcr9" event={"ID":"954da573-4dff-4a25-a180-17fe21a92841","Type":"ContainerDied","Data":"a9e54d3e09fd8e79ed9fcbd5679f4ac69858299185741216f7d5562a3c095871"} Nov 01 16:30:21.365150 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:21.365082 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892510-zjcr9" Nov 01 16:30:21.373365 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:21.373170 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2b94b5-29892510-9n2h2" Nov 01 16:30:21.373365 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:21.373237 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2b94b5-29892510-9n2h2" event={"ID":"975c2d1d-45a8-4439-923e-59f70d69ea04","Type":"ContainerDied","Data":"a18ba1bcb3a65bb076ca84b0c6ab6d2ab6cd8cbc786de9a9de7438c07a262cf7"} Nov 01 16:30:21.374674 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:21.374647 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892510-t259f" event={"ID":"aa25d700-0231-4655-a067-4d637ea4c5c0","Type":"ContainerDied","Data":"3aa4c52fa7004be9dbe1ed60dc671eb85f07ec9b7bbd486f3ca55db8b721ee4a"} Nov 01 16:30:21.374807 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:21.374676 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892510-t259f" Nov 01 16:30:21.394701 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:21.394670 4363 scope.go:122] "RemoveContainer" containerID="7d9e943425d0c8df3b03392276390827409b6c76e2b4e7c70a34047132cc537d" Nov 01 16:30:21.511946 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:21.510080 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892510-zjcr9"] Nov 01 16:30:21.594435 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:21.589471 4363 scope.go:122] "RemoveContainer" containerID="f7b1dfa26686314214737b339881b2b0bdb34a0fda9652d6fd07fdda35d53132" Nov 01 16:30:21.674578 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:21.671292 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892510-n7htt"] Nov 01 16:30:21.745798 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:21.745693 4363 scope.go:122] "RemoveContainer" containerID="8b19f353685302190e29d4a1e2f4f9c24be5ed8e9a4f3e21ff1436f53062bd1b" Nov 01 16:30:21.751490 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:21.751429 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892510-zjcr9"] Nov 01 16:30:21.854159 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:21.853339 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c8537c-29892510-6c55f"] Nov 01 16:30:21.867061 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:21.860746 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c8537c-29892510-6c55f"] Nov 01 16:30:21.874044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:21.870071 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892510-t259f"] Nov 01 16:30:21.879700 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:21.879458 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892510-t259f"] Nov 01 16:30:21.885091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:21.884992 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2b94b5-29892510-9n2h2"] Nov 01 16:30:21.940599 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:21.940552 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2b94b5-29892510-9n2h2"] Nov 01 16:30:22.013461 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:22.013254 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d121-5595856559-g7tk9" podUID="55c3d81b-79de-44cb-9838-8b07da7c911b" containerName="payload-pre-processing" containerID="cri-o://176a0024fc3496d320e703914d7fca1fb729234e11b273152fa97d284fe177f9" gracePeriod=25 Nov 01 16:30:22.137883 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:22.137857 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892510-n7htt" Nov 01 16:30:22.155243 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:22.155212 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e98f9e8-2337-406a-807c-d20dc90a35a2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e98f9e8-2337-406a-807c-d20dc90a35a2-maas-api-service-ca\") pod \"4e98f9e8-2337-406a-807c-d20dc90a35a2\" (UID: \"4e98f9e8-2337-406a-807c-d20dc90a35a2\") " Nov 01 16:30:22.155716 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:22.155353 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e98f9e8-2337-406a-807c-d20dc90a35a2-kube-api-access-xvvkl\" (UniqueName: \"kubernetes.io/projected/4e98f9e8-2337-406a-807c-d20dc90a35a2-kube-api-access-xvvkl\") pod \"4e98f9e8-2337-406a-807c-d20dc90a35a2\" (UID: \"4e98f9e8-2337-406a-807c-d20dc90a35a2\") " Nov 01 16:30:22.155716 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:22.155643 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e98f9e8-2337-406a-807c-d20dc90a35a2-maas-api-service-ca" pod "4e98f9e8-2337-406a-807c-d20dc90a35a2" (UID: "4e98f9e8-2337-406a-807c-d20dc90a35a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:30:22.156092 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:22.156071 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e98f9e8-2337-406a-807c-d20dc90a35a2-maas-api-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:22.159232 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:22.159205 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e98f9e8-2337-406a-807c-d20dc90a35a2-kube-api-access-xvvkl" pod "4e98f9e8-2337-406a-807c-d20dc90a35a2" (UID: "4e98f9e8-2337-406a-807c-d20dc90a35a2"). InnerVolumeSpecName "kube-api-access-xvvkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:22.258667 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:22.258587 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvvkl\" (UniqueName: \"kubernetes.io/projected/4e98f9e8-2337-406a-807c-d20dc90a35a2-kube-api-access-xvvkl\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:22.353612 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:22.353584 4363 patch_prober.go:29] interesting pod/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body= Nov 01 16:30:22.353759 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:22.353631 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" podUID="c8ea572a-0005-426a-9017-de3331cbcbd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" Nov 01 16:30:22.430101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:22.430062 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892510-n7htt" event={"ID":"4e98f9e8-2337-406a-807c-d20dc90a35a2","Type":"ContainerDied","Data":"f29d1886523ea21b1a407d53caec7acc4206e022c7bab32453cde5db7a425ea2"} Nov 01 16:30:22.430430 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:22.430410 4363 scope.go:122] "RemoveContainer" containerID="0feefb2c27e1c5d70bdd5b069f2a662f86d74cbf133a9f0d860f0df4ef08187e" Nov 01 16:30:22.430631 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:22.430095 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892510-n7htt" Nov 01 16:30:22.456066 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:22.456041 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892510-n7htt"] Nov 01 16:30:22.464043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:22.463995 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892510-n7htt"] Nov 01 16:30:22.818440 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:22.818400 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e98f9e8-2337-406a-807c-d20dc90a35a2" path="/var/lib/kubelet/pods/4e98f9e8-2337-406a-807c-d20dc90a35a2/volumes" Nov 01 16:30:22.818930 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:22.818915 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="954da573-4dff-4a25-a180-17fe21a92841" path="/var/lib/kubelet/pods/954da573-4dff-4a25-a180-17fe21a92841/volumes" Nov 01 16:30:22.819385 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:22.819370 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="975c2d1d-45a8-4439-923e-59f70d69ea04" path="/var/lib/kubelet/pods/975c2d1d-45a8-4439-923e-59f70d69ea04/volumes" Nov 01 16:30:22.820113 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:22.820097 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa25d700-0231-4655-a067-4d637ea4c5c0" path="/var/lib/kubelet/pods/aa25d700-0231-4655-a067-4d637ea4c5c0/volumes" Nov 01 16:30:22.820525 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:22.820511 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68" path="/var/lib/kubelet/pods/bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68/volumes" Nov 01 16:30:23.354032 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:23.353982 4363 patch_prober.go:29] interesting pod/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body= Nov 01 16:30:23.354698 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:23.354642 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" podUID="c8ea572a-0005-426a-9017-de3331cbcbd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" Nov 01 16:30:23.971565 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:23.971535 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-6f8b7f8fdc-qfwn9" Nov 01 16:30:24.353762 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.353728 4363 patch_prober.go:29] interesting pod/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body= Nov 01 16:30:24.353946 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.353788 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" podUID="c8ea572a-0005-426a-9017-de3331cbcbd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" Nov 01 16:30:24.566150 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:24.566114 4363 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-shared-b-w0-c8d121: configmap "payload-processing-plugins-e2e-shared-b-w0-c8d121" not found Nov 01 16:30:24.566617 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:24.566199 4363 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/55c3d81b-79de-44cb-9838-8b07da7c911b-plugins-config-volume podName:55c3d81b-79de-44cb-9838-8b07da7c911b nodeName:}" failed. No retries permitted until 2026-11-01 16:30:32.566178495 +0000 UTC m=+2565.897916137 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/55c3d81b-79de-44cb-9838-8b07da7c911b-plugins-config-volume") pod "payload-pre-processing-e2e-shared-b-w0-c8d121-5595856559-g7tk9" (UID: "55c3d81b-79de-44cb-9838-8b07da7c911b") : configmap "payload-processing-plugins-e2e-shared-b-w0-c8d121" not found Nov 01 16:30:24.831683 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.831593 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6b6cfc-5c65865b75-87k56"] Nov 01 16:30:24.832747 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.832725 4363 cpu_manager.go:414] "containerMap: removing container" podUID="07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d" containerName="cleanup" Nov 01 16:30:24.832861 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.832749 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d" containerName="cleanup" Nov 01 16:30:24.832861 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.832764 4363 cpu_manager.go:414] "containerMap: removing container" podUID="19456502-09eb-4dc7-8877-395f1b5e7218" containerName="istio-proxy" Nov 01 16:30:24.832861 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.832772 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19456502-09eb-4dc7-8877-395f1b5e7218" containerName="istio-proxy" Nov 01 16:30:24.832861 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.832787 4363 cpu_manager.go:414] "containerMap: removing container" podUID="89cbf816-3afe-455a-8e90-5606573018b0" containerName="cleanup" Nov 01 16:30:24.832861 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.832794 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89cbf816-3afe-455a-8e90-5606573018b0" containerName="cleanup" Nov 01 16:30:24.832861 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.832820 4363 cpu_manager.go:414] "containerMap: removing container" podUID="1817f9a5-677f-43b6-a8d0-b80a92870530" containerName="cleanup" Nov 01 16:30:24.832861 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.832827 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1817f9a5-677f-43b6-a8d0-b80a92870530" containerName="cleanup" Nov 01 16:30:24.832861 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.832837 4363 cpu_manager.go:414] "containerMap: removing container" podUID="4e98f9e8-2337-406a-807c-d20dc90a35a2" containerName="cleanup" Nov 01 16:30:24.832861 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.832844 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e98f9e8-2337-406a-807c-d20dc90a35a2" containerName="cleanup" Nov 01 16:30:24.832861 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.832855 4363 cpu_manager.go:414] "containerMap: removing container" podUID="954da573-4dff-4a25-a180-17fe21a92841" containerName="cleanup" Nov 01 16:30:24.832861 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.832862 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="954da573-4dff-4a25-a180-17fe21a92841" containerName="cleanup" Nov 01 16:30:24.833317 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.832878 4363 cpu_manager.go:414] "containerMap: removing container" podUID="aa25d700-0231-4655-a067-4d637ea4c5c0" containerName="cleanup" Nov 01 16:30:24.833317 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.832885 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa25d700-0231-4655-a067-4d637ea4c5c0" containerName="cleanup" Nov 01 16:30:24.833317 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.832899 4363 cpu_manager.go:414] "containerMap: removing container" podUID="bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68" containerName="cleanup" Nov 01 16:30:24.833317 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.832905 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68" containerName="cleanup" Nov 01 16:30:24.833317 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.832916 4363 cpu_manager.go:414] "containerMap: removing container" podUID="89cbf816-3afe-455a-8e90-5606573018b0" containerName="cleanup" Nov 01 16:30:24.833317 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.832923 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89cbf816-3afe-455a-8e90-5606573018b0" containerName="cleanup" Nov 01 16:30:24.833317 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.832933 4363 cpu_manager.go:414] "containerMap: removing container" podUID="954da573-4dff-4a25-a180-17fe21a92841" containerName="cleanup" Nov 01 16:30:24.833317 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.832939 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="954da573-4dff-4a25-a180-17fe21a92841" containerName="cleanup" Nov 01 16:30:24.833317 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.832971 4363 cpu_manager.go:414] "containerMap: removing container" podUID="975c2d1d-45a8-4439-923e-59f70d69ea04" containerName="cleanup" Nov 01 16:30:24.833317 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.832978 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="975c2d1d-45a8-4439-923e-59f70d69ea04" containerName="cleanup" Nov 01 16:30:24.833317 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.832987 4363 cpu_manager.go:414] "containerMap: removing container" podUID="aa25d700-0231-4655-a067-4d637ea4c5c0" containerName="cleanup" Nov 01 16:30:24.833317 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.832993 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa25d700-0231-4655-a067-4d637ea4c5c0" containerName="cleanup" Nov 01 16:30:24.833317 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.833005 4363 cpu_manager.go:414] "containerMap: removing container" podUID="1817f9a5-677f-43b6-a8d0-b80a92870530" containerName="cleanup" Nov 01 16:30:24.833317 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.833027 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1817f9a5-677f-43b6-a8d0-b80a92870530" containerName="cleanup" Nov 01 16:30:24.833317 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.833040 4363 cpu_manager.go:414] "containerMap: removing container" podUID="4e98f9e8-2337-406a-807c-d20dc90a35a2" containerName="cleanup" Nov 01 16:30:24.833317 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.833047 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e98f9e8-2337-406a-807c-d20dc90a35a2" containerName="cleanup" Nov 01 16:30:24.833317 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.833061 4363 cpu_manager.go:414] "containerMap: removing container" podUID="89cbf816-3afe-455a-8e90-5606573018b0" containerName="cleanup" Nov 01 16:30:24.833317 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.833068 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89cbf816-3afe-455a-8e90-5606573018b0" containerName="cleanup" Nov 01 16:30:24.833317 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.833084 4363 cpu_manager.go:414] "containerMap: removing container" podUID="975c2d1d-45a8-4439-923e-59f70d69ea04" containerName="cleanup" Nov 01 16:30:24.833317 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.833092 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="975c2d1d-45a8-4439-923e-59f70d69ea04" containerName="cleanup" Nov 01 16:30:24.833317 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.833109 4363 cpu_manager.go:414] "containerMap: removing container" podUID="07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d" containerName="cleanup" Nov 01 16:30:24.833317 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.833115 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d" containerName="cleanup" Nov 01 16:30:24.833317 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.833131 4363 cpu_manager.go:414] "containerMap: removing container" podUID="975c2d1d-45a8-4439-923e-59f70d69ea04" containerName="cleanup" Nov 01 16:30:24.833317 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.833138 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="975c2d1d-45a8-4439-923e-59f70d69ea04" containerName="cleanup" Nov 01 16:30:24.833317 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.833150 4363 cpu_manager.go:414] "containerMap: removing container" podUID="cfb264a2-0df1-4b83-bd6d-a355f75a2aca" containerName="cleanup" Nov 01 16:30:24.833317 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.833156 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfb264a2-0df1-4b83-bd6d-a355f75a2aca" containerName="cleanup" Nov 01 16:30:24.833317 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.833167 4363 cpu_manager.go:414] "containerMap: removing container" podUID="cfb264a2-0df1-4b83-bd6d-a355f75a2aca" containerName="cleanup" Nov 01 16:30:24.833317 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.833173 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfb264a2-0df1-4b83-bd6d-a355f75a2aca" containerName="cleanup" Nov 01 16:30:24.833317 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.833220 4363 cpu_manager.go:414] "containerMap: removing container" podUID="1817f9a5-677f-43b6-a8d0-b80a92870530" containerName="cleanup" Nov 01 16:30:24.833317 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.833227 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1817f9a5-677f-43b6-a8d0-b80a92870530" containerName="cleanup" Nov 01 16:30:24.833317 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.833243 4363 cpu_manager.go:414] "containerMap: removing container" podUID="bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68" containerName="cleanup" Nov 01 16:30:24.833317 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.833250 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68" containerName="cleanup" Nov 01 16:30:24.835431 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.833440 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="975c2d1d-45a8-4439-923e-59f70d69ea04" containerName="cleanup" Nov 01 16:30:24.835431 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.833457 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfb264a2-0df1-4b83-bd6d-a355f75a2aca" containerName="cleanup" Nov 01 16:30:24.835431 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.833467 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfb264a2-0df1-4b83-bd6d-a355f75a2aca" containerName="cleanup" Nov 01 16:30:24.835431 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.833482 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="1817f9a5-677f-43b6-a8d0-b80a92870530" containerName="cleanup" Nov 01 16:30:24.835431 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.833492 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa25d700-0231-4655-a067-4d637ea4c5c0" containerName="cleanup" Nov 01 16:30:24.835431 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.833500 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68" containerName="cleanup" Nov 01 16:30:24.835431 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.833507 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="19456502-09eb-4dc7-8877-395f1b5e7218" containerName="istio-proxy" Nov 01 16:30:24.835431 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.833514 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68" containerName="cleanup" Nov 01 16:30:24.835431 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.833523 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e98f9e8-2337-406a-807c-d20dc90a35a2" containerName="cleanup" Nov 01 16:30:24.835431 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.833533 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="954da573-4dff-4a25-a180-17fe21a92841" containerName="cleanup" Nov 01 16:30:24.835431 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.833542 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d" containerName="cleanup" Nov 01 16:30:24.835431 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.833550 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="89cbf816-3afe-455a-8e90-5606573018b0" containerName="cleanup" Nov 01 16:30:24.835431 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.833560 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="1817f9a5-677f-43b6-a8d0-b80a92870530" containerName="cleanup" Nov 01 16:30:24.835431 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.833569 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e98f9e8-2337-406a-807c-d20dc90a35a2" containerName="cleanup" Nov 01 16:30:24.835431 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.833576 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="954da573-4dff-4a25-a180-17fe21a92841" containerName="cleanup" Nov 01 16:30:24.835431 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.833584 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa25d700-0231-4655-a067-4d637ea4c5c0" containerName="cleanup" Nov 01 16:30:24.835431 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.833593 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="954da573-4dff-4a25-a180-17fe21a92841" containerName="cleanup" Nov 01 16:30:24.835431 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.833601 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d" containerName="cleanup" Nov 01 16:30:24.835431 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.833609 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa25d700-0231-4655-a067-4d637ea4c5c0" containerName="cleanup" Nov 01 16:30:24.835431 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.833619 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="1817f9a5-677f-43b6-a8d0-b80a92870530" containerName="cleanup" Nov 01 16:30:24.835431 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.833667 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e98f9e8-2337-406a-807c-d20dc90a35a2" containerName="cleanup" Nov 01 16:30:24.835431 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.833679 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="89cbf816-3afe-455a-8e90-5606573018b0" containerName="cleanup" Nov 01 16:30:24.835431 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.833690 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="975c2d1d-45a8-4439-923e-59f70d69ea04" containerName="cleanup" Nov 01 16:30:24.840266 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.840249 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6b6cfc-5c65865b75-87k56" Nov 01 16:30:24.840803 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.840281 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6b6cfc-5c65865b75-87k56"] Nov 01 16:30:24.844178 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.844157 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-6b6cfc-serving-cert\"" Nov 01 16:30:24.980037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:24.972387 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bae9e9c-d6f5-4b49-be31-c793e0b293ef-metrics-certs\") pod \"maas-api-e2e-ipp-b-6b6cfc-5c65865b75-87k56\" (UID: \"3bae9e9c-d6f5-4b49-be31-c793e0b293ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6b6cfc-5c65865b75-87k56" Nov 01 16:30:25.025562 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:25.025522 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bae9e9c-d6f5-4b49-be31-c793e0b293ef-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6b6cfc-5c65865b75-87k56\" (UID: \"3bae9e9c-d6f5-4b49-be31-c793e0b293ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6b6cfc-5c65865b75-87k56" Nov 01 16:30:25.025998 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:25.025971 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bae9e9c-d6f5-4b49-be31-c793e0b293ef-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6b6cfc-5c65865b75-87k56\" (UID: \"3bae9e9c-d6f5-4b49-be31-c793e0b293ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6b6cfc-5c65865b75-87k56" Nov 01 16:30:25.026622 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:25.026593 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vtf9\" (UniqueName: \"kubernetes.io/projected/3bae9e9c-d6f5-4b49-be31-c793e0b293ef-kube-api-access-5vtf9\") pod \"maas-api-e2e-ipp-b-6b6cfc-5c65865b75-87k56\" (UID: \"3bae9e9c-d6f5-4b49-be31-c793e0b293ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6b6cfc-5c65865b75-87k56" Nov 01 16:30:25.129496 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:25.129463 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vtf9\" (UniqueName: \"kubernetes.io/projected/3bae9e9c-d6f5-4b49-be31-c793e0b293ef-kube-api-access-5vtf9\") pod \"maas-api-e2e-ipp-b-6b6cfc-5c65865b75-87k56\" (UID: \"3bae9e9c-d6f5-4b49-be31-c793e0b293ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6b6cfc-5c65865b75-87k56" Nov 01 16:30:25.129884 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:25.129867 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bae9e9c-d6f5-4b49-be31-c793e0b293ef-metrics-certs\") pod \"maas-api-e2e-ipp-b-6b6cfc-5c65865b75-87k56\" (UID: \"3bae9e9c-d6f5-4b49-be31-c793e0b293ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6b6cfc-5c65865b75-87k56" Nov 01 16:30:25.130124 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:25.130093 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bae9e9c-d6f5-4b49-be31-c793e0b293ef-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6b6cfc-5c65865b75-87k56\" (UID: \"3bae9e9c-d6f5-4b49-be31-c793e0b293ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6b6cfc-5c65865b75-87k56" Nov 01 16:30:25.130313 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:25.130280 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bae9e9c-d6f5-4b49-be31-c793e0b293ef-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6b6cfc-5c65865b75-87k56\" (UID: \"3bae9e9c-d6f5-4b49-be31-c793e0b293ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6b6cfc-5c65865b75-87k56" Nov 01 16:30:25.131006 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:25.130976 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bae9e9c-d6f5-4b49-be31-c793e0b293ef-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6b6cfc-5c65865b75-87k56\" (UID: \"3bae9e9c-d6f5-4b49-be31-c793e0b293ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6b6cfc-5c65865b75-87k56" Nov 01 16:30:25.134078 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:25.132658 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bae9e9c-d6f5-4b49-be31-c793e0b293ef-metrics-certs\") pod \"maas-api-e2e-ipp-b-6b6cfc-5c65865b75-87k56\" (UID: \"3bae9e9c-d6f5-4b49-be31-c793e0b293ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6b6cfc-5c65865b75-87k56" Nov 01 16:30:25.134078 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:25.132908 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bae9e9c-d6f5-4b49-be31-c793e0b293ef-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6b6cfc-5c65865b75-87k56\" (UID: \"3bae9e9c-d6f5-4b49-be31-c793e0b293ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6b6cfc-5c65865b75-87k56" Nov 01 16:30:25.136837 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:25.136812 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vtf9\" (UniqueName: \"kubernetes.io/projected/3bae9e9c-d6f5-4b49-be31-c793e0b293ef-kube-api-access-5vtf9\") pod \"maas-api-e2e-ipp-b-6b6cfc-5c65865b75-87k56\" (UID: \"3bae9e9c-d6f5-4b49-be31-c793e0b293ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6b6cfc-5c65865b75-87k56" Nov 01 16:30:25.157734 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:25.157714 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6b6cfc-5c65865b75-87k56" Nov 01 16:30:25.354537 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:25.354098 4363 patch_prober.go:29] interesting pod/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body= Nov 01 16:30:25.354537 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:25.354157 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" podUID="c8ea572a-0005-426a-9017-de3331cbcbd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" Nov 01 16:30:25.743115 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:25.743075 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6b6cfc-5c65865b75-87k56"] Nov 01 16:30:25.743834 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:30:25.743793 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bae9e9c_d6f5_4b49_be31_c793e0b293ef.slice/crio-844ca0723e25930379ab6d1d1e03d6a52e58c0efaa86f8e4b2d693abf1c81748 WatchSource:0}: Error finding container 844ca0723e25930379ab6d1d1e03d6a52e58c0efaa86f8e4b2d693abf1c81748: Status 404 returned error can't find the container with id 844ca0723e25930379ab6d1d1e03d6a52e58c0efaa86f8e4b2d693abf1c81748 Nov 01 16:30:26.340030 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:26.339990 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" Nov 01 16:30:26.358464 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:26.358426 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" Nov 01 16:30:26.562394 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:26.562198 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6b6cfc-5c65865b75-87k56" event={"ID":"3bae9e9c-d6f5-4b49-be31-c793e0b293ef","Type":"ContainerStarted","Data":"55adff084fad0057b633b6bafbf4af965c421fe749797d5d5e514c2e56f26596"} Nov 01 16:30:26.563419 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:26.562742 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6b6cfc-5c65865b75-87k56" event={"ID":"3bae9e9c-d6f5-4b49-be31-c793e0b293ef","Type":"ContainerStarted","Data":"844ca0723e25930379ab6d1d1e03d6a52e58c0efaa86f8e4b2d693abf1c81748"} Nov 01 16:30:26.563419 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:26.562784 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6b6cfc-5c65865b75-87k56" Nov 01 16:30:26.563419 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:26.563372 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" Nov 01 16:30:26.580772 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:26.580727 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6b6cfc-5c65865b75-87k56" podStartSLOduration=2.1403577719999998 podStartE2EDuration="2.580713334s" totalImagesPullingTime="440.355562ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:30:26.579822839 +0000 UTC m=+2559.911560494" watchObservedRunningTime="2026-11-01 16:30:26.580713334 +0000 UTC m=+2559.912450983" Nov 01 16:30:28.588258 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:28.588219 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d121-5595856559-g7tk9" podUID="55c3d81b-79de-44cb-9838-8b07da7c911b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:30:29.723122 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:29.723080 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d121-k6r77"] Nov 01 16:30:29.727427 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:29.727395 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d121-k6r77"] Nov 01 16:30:30.800041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:30.799816 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6097f042-a435-451a-8fba-818eeef14f43" path="/var/lib/kubelet/pods/6097f042-a435-451a-8fba-818eeef14f43/volumes" Nov 01 16:30:32.581426 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:32.581393 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6b6cfc-5c65865b75-87k56" Nov 01 16:30:32.652786 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:32.652753 4363 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-shared-b-w0-c8d121: configmap "payload-processing-plugins-e2e-shared-b-w0-c8d121" not found Nov 01 16:30:32.653084 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:32.653061 4363 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/55c3d81b-79de-44cb-9838-8b07da7c911b-plugins-config-volume podName:55c3d81b-79de-44cb-9838-8b07da7c911b nodeName:}" failed. No retries permitted until 2026-11-01 16:30:48.653036559 +0000 UTC m=+2581.984774207 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/55c3d81b-79de-44cb-9838-8b07da7c911b-plugins-config-volume") pod "payload-pre-processing-e2e-shared-b-w0-c8d121-5595856559-g7tk9" (UID: "55c3d81b-79de-44cb-9838-8b07da7c911b") : configmap "payload-processing-plugins-e2e-shared-b-w0-c8d121" not found Nov 01 16:30:37.072234 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:37.072198 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-028c46-9f59dd876-9bxwr"] Nov 01 16:30:37.073904 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:37.073875 4363 cpu_manager.go:414] "containerMap: removing container" podUID="19456502-09eb-4dc7-8877-395f1b5e7218" containerName="istio-proxy" Nov 01 16:30:37.074154 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:37.074136 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19456502-09eb-4dc7-8877-395f1b5e7218" containerName="istio-proxy" Nov 01 16:30:37.074342 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:37.074328 4363 cpu_manager.go:414] "containerMap: removing container" podUID="954da573-4dff-4a25-a180-17fe21a92841" containerName="cleanup" Nov 01 16:30:37.074437 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:37.074425 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="954da573-4dff-4a25-a180-17fe21a92841" containerName="cleanup" Nov 01 16:30:37.074527 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:37.074516 4363 cpu_manager.go:414] "containerMap: removing container" podUID="aa25d700-0231-4655-a067-4d637ea4c5c0" containerName="cleanup" Nov 01 16:30:37.074630 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:37.074617 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa25d700-0231-4655-a067-4d637ea4c5c0" containerName="cleanup" Nov 01 16:30:37.074717 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:37.074706 4363 cpu_manager.go:414] "containerMap: removing container" podUID="4e98f9e8-2337-406a-807c-d20dc90a35a2" containerName="cleanup" Nov 01 16:30:37.074800 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:37.074789 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e98f9e8-2337-406a-807c-d20dc90a35a2" containerName="cleanup" Nov 01 16:30:37.074954 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:37.074941 4363 cpu_manager.go:414] "containerMap: removing container" podUID="07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d" containerName="cleanup" Nov 01 16:30:37.075078 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:37.075062 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d" containerName="cleanup" Nov 01 16:30:37.075554 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:37.075537 4363 cpu_manager.go:414] "containerMap: removing container" podUID="cfb264a2-0df1-4b83-bd6d-a355f75a2aca" containerName="cleanup" Nov 01 16:30:37.075678 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:37.075665 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfb264a2-0df1-4b83-bd6d-a355f75a2aca" containerName="cleanup" Nov 01 16:30:37.075812 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:37.075801 4363 cpu_manager.go:414] "containerMap: removing container" podUID="bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68" containerName="cleanup" Nov 01 16:30:37.075896 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:37.075885 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68" containerName="cleanup" Nov 01 16:30:37.076246 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:37.076228 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="89cbf816-3afe-455a-8e90-5606573018b0" containerName="cleanup" Nov 01 16:30:37.076383 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:37.076370 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="975c2d1d-45a8-4439-923e-59f70d69ea04" containerName="cleanup" Nov 01 16:30:37.076481 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:37.076470 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfb264a2-0df1-4b83-bd6d-a355f75a2aca" containerName="cleanup" Nov 01 16:30:37.076587 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:37.076576 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="19456502-09eb-4dc7-8877-395f1b5e7218" containerName="istio-proxy" Nov 01 16:30:37.076682 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:37.076671 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbcaa075-9b5d-44ce-8dc5-704ee6fd5d68" containerName="cleanup" Nov 01 16:30:37.076776 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:37.076766 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="07c858bd-2ffa-4bbb-8123-5fdb4dc5bd0d" containerName="cleanup" Nov 01 16:30:37.083240 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:37.082931 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-028c46-9f59dd876-9bxwr"] Nov 01 16:30:37.083240 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:37.083136 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-028c46-9f59dd876-9bxwr" Nov 01 16:30:37.086856 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:37.086829 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-028c46-dockercfg-mfxmz\"" Nov 01 16:30:37.154253 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:37.154224 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-028c46\"" Nov 01 16:30:37.237174 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:37.235700 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c033e4f6-d864-4579-9793-037ec810172d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-028c46-9f59dd876-9bxwr\" (UID: \"c033e4f6-d864-4579-9793-037ec810172d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-028c46-9f59dd876-9bxwr" Nov 01 16:30:37.237174 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:37.236147 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px5pr\" (UniqueName: \"kubernetes.io/projected/c033e4f6-d864-4579-9793-037ec810172d-kube-api-access-px5pr\") pod \"payload-processing-e2e-ipp-a-028c46-9f59dd876-9bxwr\" (UID: \"c033e4f6-d864-4579-9793-037ec810172d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-028c46-9f59dd876-9bxwr" Nov 01 16:30:37.356052 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:37.352228 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px5pr\" (UniqueName: \"kubernetes.io/projected/c033e4f6-d864-4579-9793-037ec810172d-kube-api-access-px5pr\") pod \"payload-processing-e2e-ipp-a-028c46-9f59dd876-9bxwr\" (UID: \"c033e4f6-d864-4579-9793-037ec810172d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-028c46-9f59dd876-9bxwr" Nov 01 16:30:37.356052 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:37.352346 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c033e4f6-d864-4579-9793-037ec810172d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-028c46-9f59dd876-9bxwr\" (UID: \"c033e4f6-d864-4579-9793-037ec810172d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-028c46-9f59dd876-9bxwr" Nov 01 16:30:37.356052 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:37.353887 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c033e4f6-d864-4579-9793-037ec810172d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-028c46-9f59dd876-9bxwr\" (UID: \"c033e4f6-d864-4579-9793-037ec810172d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-028c46-9f59dd876-9bxwr" Nov 01 16:30:37.385341 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:37.381738 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px5pr\" (UniqueName: \"kubernetes.io/projected/c033e4f6-d864-4579-9793-037ec810172d-kube-api-access-px5pr\") pod \"payload-processing-e2e-ipp-a-028c46-9f59dd876-9bxwr\" (UID: \"c033e4f6-d864-4579-9793-037ec810172d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-028c46-9f59dd876-9bxwr" Nov 01 16:30:37.409072 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:37.407625 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-028c46-9f59dd876-9bxwr" Nov 01 16:30:37.582040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:37.570173 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl"] Nov 01 16:30:37.582040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:37.570431 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" podUID="6120a22c-a6cc-4cf9-a478-7566b8bc4694" containerName="istio-proxy" containerID="cri-o://e04e57a9a871051c678b60133c6707b3c87ab05bda4280e70f540283e547b80f" gracePeriod=30 Nov 01 16:30:37.980301 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:37.980274 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-028c46-9f59dd876-9bxwr"] Nov 01 16:30:38.607249 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:38.607204 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d121-5595856559-g7tk9" podUID="55c3d81b-79de-44cb-9838-8b07da7c911b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:30:38.964693 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:38.964600 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-028c46-9f59dd876-9bxwr" event={"ID":"c033e4f6-d864-4579-9793-037ec810172d","Type":"ContainerStarted","Data":"10fa56441b78b5bb8865f2e69663e7c72b1a27d3fb2b4ac37af8a6d59c3125e9"} Nov 01 16:30:38.964693 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:38.964670 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-028c46-9f59dd876-9bxwr" event={"ID":"c033e4f6-d864-4579-9793-037ec810172d","Type":"ContainerStarted","Data":"de4ff1353bb7ee3e89567f6e75f7e66b54f3df6d5eaf22d92a79605470bbbf1d"} Nov 01 16:30:38.964956 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:38.964698 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-028c46-9f59dd876-9bxwr" Nov 01 16:30:39.016134 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:39.016059 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-028c46-9f59dd876-9bxwr" podStartSLOduration=2.016003403 podStartE2EDuration="2.016003403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:30:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:30:38.993432518 +0000 UTC m=+2572.325170177" watchObservedRunningTime="2026-11-01 16:30:39.016003403 +0000 UTC m=+2572.347741056" Nov 01 16:30:40.626587 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:40.626554 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-e1e630/ipp-route-e1e630-kserve-85dbb46dd5-qqv5q"] Nov 01 16:30:42.452274 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:42.452221 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d121-5595856559-g7tk9" Nov 01 16:30:42.505956 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:42.502678 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55c3d81b-79de-44cb-9838-8b07da7c911b-kube-api-access-k46kd\" (UniqueName: \"kubernetes.io/projected/55c3d81b-79de-44cb-9838-8b07da7c911b-kube-api-access-k46kd\") pod \"55c3d81b-79de-44cb-9838-8b07da7c911b\" (UID: \"55c3d81b-79de-44cb-9838-8b07da7c911b\") " Nov 01 16:30:42.505956 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:42.502787 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55c3d81b-79de-44cb-9838-8b07da7c911b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55c3d81b-79de-44cb-9838-8b07da7c911b-plugins-config-volume\") pod \"55c3d81b-79de-44cb-9838-8b07da7c911b\" (UID: \"55c3d81b-79de-44cb-9838-8b07da7c911b\") " Nov 01 16:30:42.513332 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:42.513292 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55c3d81b-79de-44cb-9838-8b07da7c911b-plugins-config-volume" pod "55c3d81b-79de-44cb-9838-8b07da7c911b" (UID: "55c3d81b-79de-44cb-9838-8b07da7c911b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:30:42.513543 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:42.513320 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55c3d81b-79de-44cb-9838-8b07da7c911b-kube-api-access-k46kd" pod "55c3d81b-79de-44cb-9838-8b07da7c911b" (UID: "55c3d81b-79de-44cb-9838-8b07da7c911b"). InnerVolumeSpecName "kube-api-access-k46kd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:42.606027 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:42.605993 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k46kd\" (UniqueName: \"kubernetes.io/projected/55c3d81b-79de-44cb-9838-8b07da7c911b-kube-api-access-k46kd\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:42.606240 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:42.606045 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55c3d81b-79de-44cb-9838-8b07da7c911b-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:42.957293 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:42.957260 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" Nov 01 16:30:42.973804 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:42.973774 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t"] Nov 01 16:30:43.020629 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.018644 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6120a22c-a6cc-4cf9-a478-7566b8bc4694-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6120a22c-a6cc-4cf9-a478-7566b8bc4694-istio-podinfo\") pod \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\" (UID: \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\") " Nov 01 16:30:43.020629 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.018729 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6120a22c-a6cc-4cf9-a478-7566b8bc4694-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6120a22c-a6cc-4cf9-a478-7566b8bc4694-credential-socket\") pod \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\" (UID: \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\") " Nov 01 16:30:43.020629 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.018784 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6120a22c-a6cc-4cf9-a478-7566b8bc4694-istio-token\" (UniqueName: \"kubernetes.io/projected/6120a22c-a6cc-4cf9-a478-7566b8bc4694-istio-token\") pod \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\" (UID: \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\") " Nov 01 16:30:43.020629 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.018853 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6120a22c-a6cc-4cf9-a478-7566b8bc4694-kube-api-access-lq94m\" (UniqueName: \"kubernetes.io/projected/6120a22c-a6cc-4cf9-a478-7566b8bc4694-kube-api-access-lq94m\") pod \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\" (UID: \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\") " Nov 01 16:30:43.020629 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.018900 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6120a22c-a6cc-4cf9-a478-7566b8bc4694-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6120a22c-a6cc-4cf9-a478-7566b8bc4694-istio-data\") pod \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\" (UID: \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\") " Nov 01 16:30:43.020629 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.018940 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6120a22c-a6cc-4cf9-a478-7566b8bc4694-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6120a22c-a6cc-4cf9-a478-7566b8bc4694-istio-envoy\") pod \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\" (UID: \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\") " Nov 01 16:30:43.020629 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.018995 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6120a22c-a6cc-4cf9-a478-7566b8bc4694-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6120a22c-a6cc-4cf9-a478-7566b8bc4694-istiod-ca-cert\") pod \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\" (UID: \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\") " Nov 01 16:30:43.020629 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.019084 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6120a22c-a6cc-4cf9-a478-7566b8bc4694-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6120a22c-a6cc-4cf9-a478-7566b8bc4694-workload-certs\") pod \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\" (UID: \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\") " Nov 01 16:30:43.020629 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.019276 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6120a22c-a6cc-4cf9-a478-7566b8bc4694-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6120a22c-a6cc-4cf9-a478-7566b8bc4694-workload-socket\") pod \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\" (UID: \"6120a22c-a6cc-4cf9-a478-7566b8bc4694\") " Nov 01 16:30:43.021269 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.020998 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6120a22c-a6cc-4cf9-a478-7566b8bc4694-workload-socket" pod "6120a22c-a6cc-4cf9-a478-7566b8bc4694" (UID: "6120a22c-a6cc-4cf9-a478-7566b8bc4694"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:30:43.024206 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.023847 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6120a22c-a6cc-4cf9-a478-7566b8bc4694-istio-podinfo" pod "6120a22c-a6cc-4cf9-a478-7566b8bc4694" (UID: "6120a22c-a6cc-4cf9-a478-7566b8bc4694"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:30:43.024206 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.024097 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6120a22c-a6cc-4cf9-a478-7566b8bc4694-credential-socket" pod "6120a22c-a6cc-4cf9-a478-7566b8bc4694" (UID: "6120a22c-a6cc-4cf9-a478-7566b8bc4694"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:30:43.027267 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.027238 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6120a22c-a6cc-4cf9-a478-7566b8bc4694-istiod-ca-cert" pod "6120a22c-a6cc-4cf9-a478-7566b8bc4694" (UID: "6120a22c-a6cc-4cf9-a478-7566b8bc4694"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:30:43.027675 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.027507 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6120a22c-a6cc-4cf9-a478-7566b8bc4694-istio-token" pod "6120a22c-a6cc-4cf9-a478-7566b8bc4694" (UID: "6120a22c-a6cc-4cf9-a478-7566b8bc4694"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:43.027798 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.027762 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6120a22c-a6cc-4cf9-a478-7566b8bc4694-istio-data" pod "6120a22c-a6cc-4cf9-a478-7566b8bc4694" (UID: "6120a22c-a6cc-4cf9-a478-7566b8bc4694"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:30:43.027947 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.027927 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6120a22c-a6cc-4cf9-a478-7566b8bc4694-workload-certs" pod "6120a22c-a6cc-4cf9-a478-7566b8bc4694" (UID: "6120a22c-a6cc-4cf9-a478-7566b8bc4694"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:30:43.031225 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.031197 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6120a22c-a6cc-4cf9-a478-7566b8bc4694-istio-envoy" pod "6120a22c-a6cc-4cf9-a478-7566b8bc4694" (UID: "6120a22c-a6cc-4cf9-a478-7566b8bc4694"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:30:43.031779 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.031741 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6120a22c-a6cc-4cf9-a478-7566b8bc4694-kube-api-access-lq94m" pod "6120a22c-a6cc-4cf9-a478-7566b8bc4694" (UID: "6120a22c-a6cc-4cf9-a478-7566b8bc4694"). InnerVolumeSpecName "kube-api-access-lq94m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:43.046208 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.046180 4363 generic.go:356] "Generic (PLEG): container finished" podID="6120a22c-a6cc-4cf9-a478-7566b8bc4694" containerID="e04e57a9a871051c678b60133c6707b3c87ab05bda4280e70f540283e547b80f" exitCode=0 Nov 01 16:30:43.046335 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.046238 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" Nov 01 16:30:43.046335 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.046278 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" event={"ID":"6120a22c-a6cc-4cf9-a478-7566b8bc4694","Type":"ContainerDied","Data":"e04e57a9a871051c678b60133c6707b3c87ab05bda4280e70f540283e547b80f"} Nov 01 16:30:43.046335 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.046326 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl" event={"ID":"6120a22c-a6cc-4cf9-a478-7566b8bc4694","Type":"ContainerDied","Data":"f8a65071ca4a23c7208256dfaace2eaf850f5815d6d068a5c285e6e01f2ce072"} Nov 01 16:30:43.046438 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.046352 4363 scope.go:122] "RemoveContainer" containerID="e04e57a9a871051c678b60133c6707b3c87ab05bda4280e70f540283e547b80f" Nov 01 16:30:43.049111 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.048533 4363 generic.go:356] "Generic (PLEG): container finished" podID="55c3d81b-79de-44cb-9838-8b07da7c911b" containerID="176a0024fc3496d320e703914d7fca1fb729234e11b273152fa97d284fe177f9" exitCode=0 Nov 01 16:30:43.049111 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.048804 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d121-5595856559-g7tk9" event={"ID":"55c3d81b-79de-44cb-9838-8b07da7c911b","Type":"ContainerDied","Data":"176a0024fc3496d320e703914d7fca1fb729234e11b273152fa97d284fe177f9"} Nov 01 16:30:43.049111 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.048848 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d121-5595856559-g7tk9" event={"ID":"55c3d81b-79de-44cb-9838-8b07da7c911b","Type":"ContainerDied","Data":"43b8978f4bc7a41ff6ea4452e8ccde65472fedc9a3ed3ae26bccb6a70e6ca595"} Nov 01 16:30:43.049111 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.048913 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d121-5595856559-g7tk9" Nov 01 16:30:43.064958 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.064937 4363 scope.go:122] "RemoveContainer" containerID="e04e57a9a871051c678b60133c6707b3c87ab05bda4280e70f540283e547b80f" Nov 01 16:30:43.065304 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:43.065283 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e04e57a9a871051c678b60133c6707b3c87ab05bda4280e70f540283e547b80f\": container with ID starting with e04e57a9a871051c678b60133c6707b3c87ab05bda4280e70f540283e547b80f not found: ID does not exist" containerID="e04e57a9a871051c678b60133c6707b3c87ab05bda4280e70f540283e547b80f" Nov 01 16:30:43.065402 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.065313 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e04e57a9a871051c678b60133c6707b3c87ab05bda4280e70f540283e547b80f"} err="failed to get container status \"e04e57a9a871051c678b60133c6707b3c87ab05bda4280e70f540283e547b80f\": rpc error: code = NotFound desc = could not find container \"e04e57a9a871051c678b60133c6707b3c87ab05bda4280e70f540283e547b80f\": container with ID starting with e04e57a9a871051c678b60133c6707b3c87ab05bda4280e70f540283e547b80f not found: ID does not exist" Nov 01 16:30:43.065402 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.065335 4363 scope.go:122] "RemoveContainer" containerID="176a0024fc3496d320e703914d7fca1fb729234e11b273152fa97d284fe177f9" Nov 01 16:30:43.072779 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.072758 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d121-5595856559-g7tk9"] Nov 01 16:30:43.136609 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.124819 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6120a22c-a6cc-4cf9-a478-7566b8bc4694-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:43.136609 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.124854 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6120a22c-a6cc-4cf9-a478-7566b8bc4694-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:43.136609 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.124873 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6120a22c-a6cc-4cf9-a478-7566b8bc4694-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:43.136609 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.124890 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq94m\" (UniqueName: \"kubernetes.io/projected/6120a22c-a6cc-4cf9-a478-7566b8bc4694-kube-api-access-lq94m\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:43.136609 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.124904 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6120a22c-a6cc-4cf9-a478-7566b8bc4694-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:43.136609 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.124918 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6120a22c-a6cc-4cf9-a478-7566b8bc4694-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:43.136609 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.124931 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6120a22c-a6cc-4cf9-a478-7566b8bc4694-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:43.136609 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.124946 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6120a22c-a6cc-4cf9-a478-7566b8bc4694-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:43.136609 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.124959 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6120a22c-a6cc-4cf9-a478-7566b8bc4694-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:43.188057 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.142823 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d121-5595856559-g7tk9"] Nov 01 16:30:43.211120 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.210457 4363 scope.go:122] "RemoveContainer" containerID="176a0024fc3496d320e703914d7fca1fb729234e11b273152fa97d284fe177f9" Nov 01 16:30:43.227117 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:43.211819 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"176a0024fc3496d320e703914d7fca1fb729234e11b273152fa97d284fe177f9\": container with ID starting with 176a0024fc3496d320e703914d7fca1fb729234e11b273152fa97d284fe177f9 not found: ID does not exist" containerID="176a0024fc3496d320e703914d7fca1fb729234e11b273152fa97d284fe177f9" Nov 01 16:30:43.227117 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.212059 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"176a0024fc3496d320e703914d7fca1fb729234e11b273152fa97d284fe177f9"} err="failed to get container status \"176a0024fc3496d320e703914d7fca1fb729234e11b273152fa97d284fe177f9\": rpc error: code = NotFound desc = could not find container \"176a0024fc3496d320e703914d7fca1fb729234e11b273152fa97d284fe177f9\": container with ID starting with 176a0024fc3496d320e703914d7fca1fb729234e11b273152fa97d284fe177f9 not found: ID does not exist" Nov 01 16:30:43.227117 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.225843 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl"] Nov 01 16:30:43.231058 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:43.231032 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-jhtvl"] Nov 01 16:30:44.327077 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:44.327043 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r"] Nov 01 16:30:44.328078 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:44.328044 4363 cpu_manager.go:414] "containerMap: removing container" podUID="55c3d81b-79de-44cb-9838-8b07da7c911b" containerName="payload-pre-processing" Nov 01 16:30:44.328251 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:44.328226 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55c3d81b-79de-44cb-9838-8b07da7c911b" containerName="payload-pre-processing" Nov 01 16:30:44.328403 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:44.328267 4363 cpu_manager.go:414] "containerMap: removing container" podUID="6120a22c-a6cc-4cf9-a478-7566b8bc4694" containerName="istio-proxy" Nov 01 16:30:44.328403 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:44.328279 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6120a22c-a6cc-4cf9-a478-7566b8bc4694" containerName="istio-proxy" Nov 01 16:30:44.328521 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:44.328510 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="55c3d81b-79de-44cb-9838-8b07da7c911b" containerName="payload-pre-processing" Nov 01 16:30:44.328583 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:44.328530 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="6120a22c-a6cc-4cf9-a478-7566b8bc4694" containerName="istio-proxy" Nov 01 16:30:44.332946 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:44.332921 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" Nov 01 16:30:44.355421 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:44.354720 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r"] Nov 01 16:30:44.450422 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:44.450383 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4995f2f9-c830-41b4-b501-d8aacb690731-workload-socket\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r\" (UID: \"4995f2f9-c830-41b4-b501-d8aacb690731\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" Nov 01 16:30:44.450570 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:44.450440 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4995f2f9-c830-41b4-b501-d8aacb690731-istio-podinfo\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r\" (UID: \"4995f2f9-c830-41b4-b501-d8aacb690731\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" Nov 01 16:30:44.450570 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:44.450543 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4995f2f9-c830-41b4-b501-d8aacb690731-istio-token\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r\" (UID: \"4995f2f9-c830-41b4-b501-d8aacb690731\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" Nov 01 16:30:44.450668 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:44.450614 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4995f2f9-c830-41b4-b501-d8aacb690731-workload-certs\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r\" (UID: \"4995f2f9-c830-41b4-b501-d8aacb690731\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" Nov 01 16:30:44.450828 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:44.450805 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4995f2f9-c830-41b4-b501-d8aacb690731-istio-data\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r\" (UID: \"4995f2f9-c830-41b4-b501-d8aacb690731\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" Nov 01 16:30:44.450903 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:44.450884 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4995f2f9-c830-41b4-b501-d8aacb690731-istiod-ca-cert\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r\" (UID: \"4995f2f9-c830-41b4-b501-d8aacb690731\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" Nov 01 16:30:44.450962 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:44.450947 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt5z7\" (UniqueName: \"kubernetes.io/projected/4995f2f9-c830-41b4-b501-d8aacb690731-kube-api-access-xt5z7\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r\" (UID: \"4995f2f9-c830-41b4-b501-d8aacb690731\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" Nov 01 16:30:44.451006 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:44.450992 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4995f2f9-c830-41b4-b501-d8aacb690731-istio-envoy\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r\" (UID: \"4995f2f9-c830-41b4-b501-d8aacb690731\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" Nov 01 16:30:44.451102 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:44.451086 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4995f2f9-c830-41b4-b501-d8aacb690731-credential-socket\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r\" (UID: \"4995f2f9-c830-41b4-b501-d8aacb690731\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" Nov 01 16:30:44.553449 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:44.552281 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4995f2f9-c830-41b4-b501-d8aacb690731-workload-socket\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r\" (UID: \"4995f2f9-c830-41b4-b501-d8aacb690731\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" Nov 01 16:30:44.553449 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:44.552348 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4995f2f9-c830-41b4-b501-d8aacb690731-istio-podinfo\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r\" (UID: \"4995f2f9-c830-41b4-b501-d8aacb690731\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" Nov 01 16:30:44.553449 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:44.552393 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4995f2f9-c830-41b4-b501-d8aacb690731-istio-token\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r\" (UID: \"4995f2f9-c830-41b4-b501-d8aacb690731\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" Nov 01 16:30:44.553449 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:44.552472 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4995f2f9-c830-41b4-b501-d8aacb690731-workload-certs\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r\" (UID: \"4995f2f9-c830-41b4-b501-d8aacb690731\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" Nov 01 16:30:44.553449 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:44.552610 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4995f2f9-c830-41b4-b501-d8aacb690731-istio-data\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r\" (UID: \"4995f2f9-c830-41b4-b501-d8aacb690731\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" Nov 01 16:30:44.553449 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:44.552670 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4995f2f9-c830-41b4-b501-d8aacb690731-istiod-ca-cert\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r\" (UID: \"4995f2f9-c830-41b4-b501-d8aacb690731\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" Nov 01 16:30:44.553449 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:44.552724 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt5z7\" (UniqueName: \"kubernetes.io/projected/4995f2f9-c830-41b4-b501-d8aacb690731-kube-api-access-xt5z7\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r\" (UID: \"4995f2f9-c830-41b4-b501-d8aacb690731\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" Nov 01 16:30:44.553449 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:44.552760 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4995f2f9-c830-41b4-b501-d8aacb690731-istio-envoy\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r\" (UID: \"4995f2f9-c830-41b4-b501-d8aacb690731\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" Nov 01 16:30:44.553449 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:44.552800 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4995f2f9-c830-41b4-b501-d8aacb690731-credential-socket\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r\" (UID: \"4995f2f9-c830-41b4-b501-d8aacb690731\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" Nov 01 16:30:44.553449 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:44.553366 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4995f2f9-c830-41b4-b501-d8aacb690731-credential-socket\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r\" (UID: \"4995f2f9-c830-41b4-b501-d8aacb690731\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" Nov 01 16:30:44.554262 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:44.553633 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4995f2f9-c830-41b4-b501-d8aacb690731-workload-socket\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r\" (UID: \"4995f2f9-c830-41b4-b501-d8aacb690731\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" Nov 01 16:30:44.555257 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:44.554737 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4995f2f9-c830-41b4-b501-d8aacb690731-workload-certs\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r\" (UID: \"4995f2f9-c830-41b4-b501-d8aacb690731\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" Nov 01 16:30:44.555257 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:44.555212 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4995f2f9-c830-41b4-b501-d8aacb690731-istiod-ca-cert\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r\" (UID: \"4995f2f9-c830-41b4-b501-d8aacb690731\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" Nov 01 16:30:44.555257 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:44.555189 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4995f2f9-c830-41b4-b501-d8aacb690731-istio-data\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r\" (UID: \"4995f2f9-c830-41b4-b501-d8aacb690731\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" Nov 01 16:30:44.557509 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:44.557484 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4995f2f9-c830-41b4-b501-d8aacb690731-istio-podinfo\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r\" (UID: \"4995f2f9-c830-41b4-b501-d8aacb690731\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" Nov 01 16:30:44.557747 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:44.557722 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4995f2f9-c830-41b4-b501-d8aacb690731-istio-envoy\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r\" (UID: \"4995f2f9-c830-41b4-b501-d8aacb690731\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" Nov 01 16:30:44.562755 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:44.562717 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt5z7\" (UniqueName: \"kubernetes.io/projected/4995f2f9-c830-41b4-b501-d8aacb690731-kube-api-access-xt5z7\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r\" (UID: \"4995f2f9-c830-41b4-b501-d8aacb690731\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" Nov 01 16:30:44.564104 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:44.564044 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4995f2f9-c830-41b4-b501-d8aacb690731-istio-token\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r\" (UID: \"4995f2f9-c830-41b4-b501-d8aacb690731\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" Nov 01 16:30:44.669314 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:44.669281 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" Nov 01 16:30:44.803333 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:44.803298 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55c3d81b-79de-44cb-9838-8b07da7c911b" path="/var/lib/kubelet/pods/55c3d81b-79de-44cb-9838-8b07da7c911b/volumes" Nov 01 16:30:44.803908 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:44.803890 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6120a22c-a6cc-4cf9-a478-7566b8bc4694" path="/var/lib/kubelet/pods/6120a22c-a6cc-4cf9-a478-7566b8bc4694/volumes" Nov 01 16:30:44.961038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:44.960980 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r"] Nov 01 16:30:44.961533 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:30:44.961493 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4995f2f9_c830_41b4_b501_d8aacb690731.slice/crio-d49e4640d2068b20cab3099b6b8357858f47ca66ccce90fed993fa73944c21ba WatchSource:0}: Error finding container d49e4640d2068b20cab3099b6b8357858f47ca66ccce90fed993fa73944c21ba: Status 404 returned error can't find the container with id d49e4640d2068b20cab3099b6b8357858f47ca66ccce90fed993fa73944c21ba Nov 01 16:30:44.966369 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:44.966311 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:30:44.966494 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:44.966410 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:30:44.966494 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:44.966455 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:30:45.071135 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:45.071094 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" event={"ID":"4995f2f9-c830-41b4-b501-d8aacb690731","Type":"ContainerStarted","Data":"375710f4d1c2055c11f357a67a5833fa64df78c2deea66dac61991b8484b52ed"} Nov 01 16:30:45.071311 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:45.071176 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" event={"ID":"4995f2f9-c830-41b4-b501-d8aacb690731","Type":"ContainerStarted","Data":"d49e4640d2068b20cab3099b6b8357858f47ca66ccce90fed993fa73944c21ba"} Nov 01 16:30:45.096160 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:45.096109 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" podStartSLOduration=1.096091639 podStartE2EDuration="1.096091639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:30:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:30:45.089156902 +0000 UTC m=+2578.420894552" watchObservedRunningTime="2026-11-01 16:30:45.096091639 +0000 UTC m=+2578.427829292" Nov 01 16:30:45.391272 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:45.391234 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-6f8b7f8fdc-qfwn9"] Nov 01 16:30:45.678325 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:45.678162 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" Nov 01 16:30:46.671955 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:46.671918 4363 patch_prober.go:29] interesting pod/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body= Nov 01 16:30:46.672430 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:46.671976 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" podUID="4995f2f9-c830-41b4-b501-d8aacb690731" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" Nov 01 16:30:47.670405 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:47.670368 4363 patch_prober.go:29] interesting pod/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body= Nov 01 16:30:47.670570 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:47.670446 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" podUID="4995f2f9-c830-41b4-b501-d8aacb690731" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" Nov 01 16:30:48.277244 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:48.277207 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4a083d-84c576c9b7-jlscm"] Nov 01 16:30:48.371159 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:48.370740 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:30:48.679043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:48.676905 4363 patch_prober.go:29] interesting pod/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body= Nov 01 16:30:48.679043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:48.676993 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" podUID="4995f2f9-c830-41b4-b501-d8aacb690731" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" Nov 01 16:30:48.711160 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:48.697366 4363 patch_prober.go:29] interesting pod/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:30:48.711160 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:48.697500 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" podUID="f3655831-2619-45f1-9da3-5bbe7dca87f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:30:49.669977 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:49.669940 4363 patch_prober.go:29] interesting pod/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body= Nov 01 16:30:49.671193 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:49.670000 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" podUID="4995f2f9-c830-41b4-b501-d8aacb690731" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" Nov 01 16:30:49.975095 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:49.975059 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-028c46-9f59dd876-9bxwr" Nov 01 16:30:50.162452 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:50.162422 4363 generic.go:356] "Generic (PLEG): container finished" podID="f3655831-2619-45f1-9da3-5bbe7dca87f4" containerID="1c053fc029571cf2587d21ea99f73ad2fdc6c84e9249f49578275898ea8d521e" exitCode=0 Nov 01 16:30:50.162639 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:50.162498 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" event={"ID":"f3655831-2619-45f1-9da3-5bbe7dca87f4","Type":"ContainerDied","Data":"1c053fc029571cf2587d21ea99f73ad2fdc6c84e9249f49578275898ea8d521e"} Nov 01 16:30:50.162982 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:50.162966 4363 scope.go:122] "RemoveContainer" containerID="1c053fc029571cf2587d21ea99f73ad2fdc6c84e9249f49578275898ea8d521e" Nov 01 16:30:50.164962 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:50.164933 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:30:50.165063 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:50.165006 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:30:50.165111 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:50.165075 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:30:50.406657 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:50.406601 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-6f8b7f8fdc-qfwn9" podUID="2432fb41-4632-4522-a4c1-cdfcfcd9ea43" containerName="payload-pre-processing" containerID="cri-o://2763726f208d5318ded4fb777659b7530dd1480f44445a9d988c76ae598d41ef" gracePeriod=25 Nov 01 16:30:50.709917 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:50.673508 4363 patch_prober.go:29] interesting pod/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body= Nov 01 16:30:50.709917 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:50.673567 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" podUID="4995f2f9-c830-41b4-b501-d8aacb690731" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" Nov 01 16:30:50.939732 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:50.939293 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-6f8b7f8fdc-qfwn9" Nov 01 16:30:51.064611 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:51.064495 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2432fb41-4632-4522-a4c1-cdfcfcd9ea43-kube-api-access-j2cls\" (UniqueName: \"kubernetes.io/projected/2432fb41-4632-4522-a4c1-cdfcfcd9ea43-kube-api-access-j2cls\") pod \"2432fb41-4632-4522-a4c1-cdfcfcd9ea43\" (UID: \"2432fb41-4632-4522-a4c1-cdfcfcd9ea43\") " Nov 01 16:30:51.064611 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:51.064567 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2432fb41-4632-4522-a4c1-cdfcfcd9ea43-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2432fb41-4632-4522-a4c1-cdfcfcd9ea43-plugins-config-volume\") pod \"2432fb41-4632-4522-a4c1-cdfcfcd9ea43\" (UID: \"2432fb41-4632-4522-a4c1-cdfcfcd9ea43\") " Nov 01 16:30:51.064948 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:51.064921 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2432fb41-4632-4522-a4c1-cdfcfcd9ea43-plugins-config-volume" pod "2432fb41-4632-4522-a4c1-cdfcfcd9ea43" (UID: "2432fb41-4632-4522-a4c1-cdfcfcd9ea43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:30:51.065269 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:51.065251 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2432fb41-4632-4522-a4c1-cdfcfcd9ea43-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:51.067544 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:51.066668 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2432fb41-4632-4522-a4c1-cdfcfcd9ea43-kube-api-access-j2cls" pod "2432fb41-4632-4522-a4c1-cdfcfcd9ea43" (UID: "2432fb41-4632-4522-a4c1-cdfcfcd9ea43"). InnerVolumeSpecName "kube-api-access-j2cls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:51.166562 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:51.166525 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2cls\" (UniqueName: \"kubernetes.io/projected/2432fb41-4632-4522-a4c1-cdfcfcd9ea43-kube-api-access-j2cls\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:51.172948 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:51.172916 4363 generic.go:356] "Generic (PLEG): container finished" podID="2432fb41-4632-4522-a4c1-cdfcfcd9ea43" containerID="2763726f208d5318ded4fb777659b7530dd1480f44445a9d988c76ae598d41ef" exitCode=0 Nov 01 16:30:51.173128 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:51.172969 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-6f8b7f8fdc-qfwn9" Nov 01 16:30:51.173128 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:51.172994 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-6f8b7f8fdc-qfwn9" event={"ID":"2432fb41-4632-4522-a4c1-cdfcfcd9ea43","Type":"ContainerDied","Data":"2763726f208d5318ded4fb777659b7530dd1480f44445a9d988c76ae598d41ef"} Nov 01 16:30:51.173128 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:51.173076 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-6f8b7f8fdc-qfwn9" event={"ID":"2432fb41-4632-4522-a4c1-cdfcfcd9ea43","Type":"ContainerDied","Data":"a1699ccb618f7144e4594c4142dabbddd4edd592423304b6b6154aca999224c9"} Nov 01 16:30:51.173128 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:51.173112 4363 scope.go:122] "RemoveContainer" containerID="2763726f208d5318ded4fb777659b7530dd1480f44445a9d988c76ae598d41ef" Nov 01 16:30:51.175449 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:51.175417 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" event={"ID":"f3655831-2619-45f1-9da3-5bbe7dca87f4","Type":"ContainerStarted","Data":"82772b8f1c16ec6807099169d4a16b294ecc89b48490df7a20150da84c0d3beb"} Nov 01 16:30:51.222287 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:51.222259 4363 scope.go:122] "RemoveContainer" containerID="2763726f208d5318ded4fb777659b7530dd1480f44445a9d988c76ae598d41ef" Nov 01 16:30:51.222620 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:51.222595 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2763726f208d5318ded4fb777659b7530dd1480f44445a9d988c76ae598d41ef\": container with ID starting with 2763726f208d5318ded4fb777659b7530dd1480f44445a9d988c76ae598d41ef not found: ID does not exist" containerID="2763726f208d5318ded4fb777659b7530dd1480f44445a9d988c76ae598d41ef" Nov 01 16:30:51.222753 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:51.222730 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2763726f208d5318ded4fb777659b7530dd1480f44445a9d988c76ae598d41ef"} err="failed to get container status \"2763726f208d5318ded4fb777659b7530dd1480f44445a9d988c76ae598d41ef\": rpc error: code = NotFound desc = could not find container \"2763726f208d5318ded4fb777659b7530dd1480f44445a9d988c76ae598d41ef\": container with ID starting with 2763726f208d5318ded4fb777659b7530dd1480f44445a9d988c76ae598d41ef not found: ID does not exist" Nov 01 16:30:51.233497 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:51.233443 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-6f8b7f8fdc-qfwn9"] Nov 01 16:30:51.237385 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:51.237361 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-6f8b7f8fdc-qfwn9"] Nov 01 16:30:51.718063 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:51.713442 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" Nov 01 16:30:51.718063 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:51.715213 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" Nov 01 16:30:51.723131 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:51.721660 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" Nov 01 16:30:51.844124 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:51.844072 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" Nov 01 16:30:51.844353 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:51.844312 4363 patch_prober.go:29] interesting pod/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Nov 01 16:30:51.844433 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:51.844370 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" podUID="f3655831-2619-45f1-9da3-5bbe7dca87f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Nov 01 16:30:52.027951 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:52.027114 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96e10868-s6xp2"] Nov 01 16:30:52.028396 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:52.028344 4363 cpu_manager.go:414] "containerMap: removing container" podUID="2432fb41-4632-4522-a4c1-cdfcfcd9ea43" containerName="payload-pre-processing" Nov 01 16:30:52.028396 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:52.028373 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2432fb41-4632-4522-a4c1-cdfcfcd9ea43" containerName="payload-pre-processing" Nov 01 16:30:52.029102 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:52.028644 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="2432fb41-4632-4522-a4c1-cdfcfcd9ea43" containerName="payload-pre-processing" Nov 01 16:30:52.034826 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:52.034802 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96e10868-s6xp2"] Nov 01 16:30:52.035009 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:52.034914 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e10868-s6xp2" Nov 01 16:30:52.114363 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:52.114319 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:30:52.114555 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:52.114480 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:30:52.134772 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:52.134738 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6cdf1c8-f85e-4abf-af8a-8a695f7302de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96e10868-s6xp2\" (UID: \"e6cdf1c8-f85e-4abf-af8a-8a695f7302de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e10868-s6xp2" Nov 01 16:30:52.134973 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:52.134808 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6cdf1c8-f85e-4abf-af8a-8a695f7302de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96e10868-s6xp2\" (UID: \"e6cdf1c8-f85e-4abf-af8a-8a695f7302de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e10868-s6xp2" Nov 01 16:30:52.135080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:52.135056 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m79h\" (UniqueName: \"kubernetes.io/projected/e6cdf1c8-f85e-4abf-af8a-8a695f7302de-kube-api-access-9m79h\") pod \"maas-api-revoke-sub-96e10868-s6xp2\" (UID: \"e6cdf1c8-f85e-4abf-af8a-8a695f7302de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e10868-s6xp2" Nov 01 16:30:52.236467 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:52.236429 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m79h\" (UniqueName: \"kubernetes.io/projected/e6cdf1c8-f85e-4abf-af8a-8a695f7302de-kube-api-access-9m79h\") pod \"maas-api-revoke-sub-96e10868-s6xp2\" (UID: \"e6cdf1c8-f85e-4abf-af8a-8a695f7302de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e10868-s6xp2" Nov 01 16:30:52.236730 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:52.236711 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6cdf1c8-f85e-4abf-af8a-8a695f7302de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96e10868-s6xp2\" (UID: \"e6cdf1c8-f85e-4abf-af8a-8a695f7302de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e10868-s6xp2" Nov 01 16:30:52.236907 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:52.236887 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6cdf1c8-f85e-4abf-af8a-8a695f7302de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96e10868-s6xp2\" (UID: \"e6cdf1c8-f85e-4abf-af8a-8a695f7302de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e10868-s6xp2" Nov 01 16:30:52.237363 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:52.237340 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6cdf1c8-f85e-4abf-af8a-8a695f7302de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96e10868-s6xp2\" (UID: \"e6cdf1c8-f85e-4abf-af8a-8a695f7302de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e10868-s6xp2" Nov 01 16:30:52.239183 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:52.239166 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6cdf1c8-f85e-4abf-af8a-8a695f7302de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96e10868-s6xp2\" (UID: \"e6cdf1c8-f85e-4abf-af8a-8a695f7302de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e10868-s6xp2" Nov 01 16:30:52.243493 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:52.243469 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m79h\" (UniqueName: \"kubernetes.io/projected/e6cdf1c8-f85e-4abf-af8a-8a695f7302de-kube-api-access-9m79h\") pod \"maas-api-revoke-sub-96e10868-s6xp2\" (UID: \"e6cdf1c8-f85e-4abf-af8a-8a695f7302de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e10868-s6xp2" Nov 01 16:30:52.355040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:52.350858 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e10868-s6xp2" Nov 01 16:30:52.717514 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:52.717346 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96e10868-s6xp2"] Nov 01 16:30:52.717801 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:30:52.717757 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6cdf1c8_f85e_4abf_af8a_8a695f7302de.slice/crio-e869d99fd04aac1aff5d2868e9a62df42c9891d8b291039fe9c6a3476cdfeca2 WatchSource:0}: Error finding container e869d99fd04aac1aff5d2868e9a62df42c9891d8b291039fe9c6a3476cdfeca2: Status 404 returned error can't find the container with id e869d99fd04aac1aff5d2868e9a62df42c9891d8b291039fe9c6a3476cdfeca2 Nov 01 16:30:52.819247 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:52.819212 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2432fb41-4632-4522-a4c1-cdfcfcd9ea43" path="/var/lib/kubelet/pods/2432fb41-4632-4522-a4c1-cdfcfcd9ea43/volumes" Nov 01 16:30:52.845633 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:52.845596 4363 patch_prober.go:29] interesting pod/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Nov 01 16:30:52.845826 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:52.845689 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" podUID="f3655831-2619-45f1-9da3-5bbe7dca87f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Nov 01 16:30:53.318094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:53.299733 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e10868-s6xp2" event={"ID":"e6cdf1c8-f85e-4abf-af8a-8a695f7302de","Type":"ContainerStarted","Data":"e869d99fd04aac1aff5d2868e9a62df42c9891d8b291039fe9c6a3476cdfeca2"} Nov 01 16:30:53.336724 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:53.336683 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-4a083d-84c576c9b7-jlscm" podUID="a1a8daa3-0ee1-4164-9dab-1fdba6d81073" containerName="payload-processing" containerID="cri-o://f2fe23e94e94984fa2d17809d5fad34c4e7fe6cf3bfc2fc5bf17b5d2bb0df0c0" gracePeriod=25 Nov 01 16:30:53.718766 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:53.718736 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-4a083d-84c576c9b7-jlscm" Nov 01 16:30:53.745504 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:53.745471 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1a8daa3-0ee1-4164-9dab-1fdba6d81073-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a8daa3-0ee1-4164-9dab-1fdba6d81073-plugins-config-volume\") pod \"a1a8daa3-0ee1-4164-9dab-1fdba6d81073\" (UID: \"a1a8daa3-0ee1-4164-9dab-1fdba6d81073\") " Nov 01 16:30:53.745670 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:53.745650 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1a8daa3-0ee1-4164-9dab-1fdba6d81073-kube-api-access-7ptws\" (UniqueName: \"kubernetes.io/projected/a1a8daa3-0ee1-4164-9dab-1fdba6d81073-kube-api-access-7ptws\") pod \"a1a8daa3-0ee1-4164-9dab-1fdba6d81073\" (UID: \"a1a8daa3-0ee1-4164-9dab-1fdba6d81073\") " Nov 01 16:30:53.745844 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:53.745822 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1a8daa3-0ee1-4164-9dab-1fdba6d81073-plugins-config-volume" pod "a1a8daa3-0ee1-4164-9dab-1fdba6d81073" (UID: "a1a8daa3-0ee1-4164-9dab-1fdba6d81073"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:30:53.746139 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:53.746122 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a8daa3-0ee1-4164-9dab-1fdba6d81073-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:53.747829 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:53.747806 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1a8daa3-0ee1-4164-9dab-1fdba6d81073-kube-api-access-7ptws" pod "a1a8daa3-0ee1-4164-9dab-1fdba6d81073" (UID: "a1a8daa3-0ee1-4164-9dab-1fdba6d81073"). InnerVolumeSpecName "kube-api-access-7ptws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:53.845053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:53.844999 4363 patch_prober.go:29] interesting pod/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Nov 01 16:30:53.845436 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:53.845085 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" podUID="f3655831-2619-45f1-9da3-5bbe7dca87f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Nov 01 16:30:53.848217 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:53.848187 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ptws\" (UniqueName: \"kubernetes.io/projected/a1a8daa3-0ee1-4164-9dab-1fdba6d81073-kube-api-access-7ptws\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:54.318843 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:54.318806 4363 generic.go:356] "Generic (PLEG): container finished" podID="a1a8daa3-0ee1-4164-9dab-1fdba6d81073" containerID="f2fe23e94e94984fa2d17809d5fad34c4e7fe6cf3bfc2fc5bf17b5d2bb0df0c0" exitCode=0 Nov 01 16:30:54.318843 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:54.318847 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-4a083d-84c576c9b7-jlscm" Nov 01 16:30:54.319144 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:54.318891 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-4a083d-84c576c9b7-jlscm" event={"ID":"a1a8daa3-0ee1-4164-9dab-1fdba6d81073","Type":"ContainerDied","Data":"f2fe23e94e94984fa2d17809d5fad34c4e7fe6cf3bfc2fc5bf17b5d2bb0df0c0"} Nov 01 16:30:54.319144 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:54.318956 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-4a083d-84c576c9b7-jlscm" event={"ID":"a1a8daa3-0ee1-4164-9dab-1fdba6d81073","Type":"ContainerDied","Data":"1a61272de52eb30083f5e818c2578cc11a9f3a414bb14db4ae236e48ccf8491f"} Nov 01 16:30:54.319144 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:54.318984 4363 scope.go:122] "RemoveContainer" containerID="f2fe23e94e94984fa2d17809d5fad34c4e7fe6cf3bfc2fc5bf17b5d2bb0df0c0" Nov 01 16:30:54.323887 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:54.323859 4363 generic.go:356] "Generic (PLEG): container finished" podID="e6cdf1c8-f85e-4abf-af8a-8a695f7302de" containerID="61022769ba07beb383d453bb2975b8e961d5508c7b62aa20ba254df1b3b270f5" exitCode=0 Nov 01 16:30:54.324095 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:54.323926 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e10868-s6xp2" event={"ID":"e6cdf1c8-f85e-4abf-af8a-8a695f7302de","Type":"ContainerDied","Data":"61022769ba07beb383d453bb2975b8e961d5508c7b62aa20ba254df1b3b270f5"} Nov 01 16:30:54.338583 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:54.338568 4363 scope.go:122] "RemoveContainer" containerID="f2fe23e94e94984fa2d17809d5fad34c4e7fe6cf3bfc2fc5bf17b5d2bb0df0c0" Nov 01 16:30:54.338911 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:30:54.338886 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2fe23e94e94984fa2d17809d5fad34c4e7fe6cf3bfc2fc5bf17b5d2bb0df0c0\": container with ID starting with f2fe23e94e94984fa2d17809d5fad34c4e7fe6cf3bfc2fc5bf17b5d2bb0df0c0 not found: ID does not exist" containerID="f2fe23e94e94984fa2d17809d5fad34c4e7fe6cf3bfc2fc5bf17b5d2bb0df0c0" Nov 01 16:30:54.339010 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:54.338921 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2fe23e94e94984fa2d17809d5fad34c4e7fe6cf3bfc2fc5bf17b5d2bb0df0c0"} err="failed to get container status \"f2fe23e94e94984fa2d17809d5fad34c4e7fe6cf3bfc2fc5bf17b5d2bb0df0c0\": rpc error: code = NotFound desc = could not find container \"f2fe23e94e94984fa2d17809d5fad34c4e7fe6cf3bfc2fc5bf17b5d2bb0df0c0\": container with ID starting with f2fe23e94e94984fa2d17809d5fad34c4e7fe6cf3bfc2fc5bf17b5d2bb0df0c0 not found: ID does not exist" Nov 01 16:30:54.355680 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:54.355659 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4a083d-84c576c9b7-jlscm"] Nov 01 16:30:54.361240 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:54.361219 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4a083d-84c576c9b7-jlscm"] Nov 01 16:30:54.831906 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:54.831869 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1a8daa3-0ee1-4164-9dab-1fdba6d81073" path="/var/lib/kubelet/pods/a1a8daa3-0ee1-4164-9dab-1fdba6d81073/volumes" Nov 01 16:30:54.844277 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:54.844247 4363 patch_prober.go:29] interesting pod/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Nov 01 16:30:54.844452 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:54.844295 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" podUID="f3655831-2619-45f1-9da3-5bbe7dca87f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Nov 01 16:30:55.648889 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:55.648865 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e10868-s6xp2" Nov 01 16:30:55.650316 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:55.650288 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-e1e630/ipp-route-e1e630-kserve-85dbb46dd5-qqv5q" podUID="474f6a6d-0ab2-4f46-b9b8-f29f577f7787" containerName="main" containerID="cri-o://6ba8a563e37b7d8b4e8f0a5869b2f43fed73a62556405e59548bfb3ffa3b15ef" gracePeriod=45 Nov 01 16:30:55.750857 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:55.750826 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h"] Nov 01 16:30:55.751698 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:55.751670 4363 cpu_manager.go:414] "containerMap: removing container" podUID="a1a8daa3-0ee1-4164-9dab-1fdba6d81073" containerName="payload-processing" Nov 01 16:30:55.751698 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:55.751698 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1a8daa3-0ee1-4164-9dab-1fdba6d81073" containerName="payload-processing" Nov 01 16:30:55.751917 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:55.751726 4363 cpu_manager.go:414] "containerMap: removing container" podUID="e6cdf1c8-f85e-4abf-af8a-8a695f7302de" containerName="revoke-keys" Nov 01 16:30:55.751917 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:55.751733 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6cdf1c8-f85e-4abf-af8a-8a695f7302de" containerName="revoke-keys" Nov 01 16:30:55.752051 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:55.751952 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1a8daa3-0ee1-4164-9dab-1fdba6d81073" containerName="payload-processing" Nov 01 16:30:55.752051 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:55.751971 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6cdf1c8-f85e-4abf-af8a-8a695f7302de" containerName="revoke-keys" Nov 01 16:30:55.755817 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:55.755797 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" Nov 01 16:30:55.765389 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:55.765369 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-aa6303-openshift-default-dockercfg-pmj8v\"" Nov 01 16:30:55.797054 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:55.789662 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h"] Nov 01 16:30:55.827887 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:55.827855 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6cdf1c8-f85e-4abf-af8a-8a695f7302de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6cdf1c8-f85e-4abf-af8a-8a695f7302de-maas-api-xks-ca\") pod \"e6cdf1c8-f85e-4abf-af8a-8a695f7302de\" (UID: \"e6cdf1c8-f85e-4abf-af8a-8a695f7302de\") " Nov 01 16:30:55.828183 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:55.828169 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6cdf1c8-f85e-4abf-af8a-8a695f7302de-kube-api-access-9m79h\" (UniqueName: \"kubernetes.io/projected/e6cdf1c8-f85e-4abf-af8a-8a695f7302de-kube-api-access-9m79h\") pod \"e6cdf1c8-f85e-4abf-af8a-8a695f7302de\" (UID: \"e6cdf1c8-f85e-4abf-af8a-8a695f7302de\") " Nov 01 16:30:55.828320 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:55.828284 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6cdf1c8-f85e-4abf-af8a-8a695f7302de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6cdf1c8-f85e-4abf-af8a-8a695f7302de-maas-api-openshift-ca\") pod \"e6cdf1c8-f85e-4abf-af8a-8a695f7302de\" (UID: \"e6cdf1c8-f85e-4abf-af8a-8a695f7302de\") " Nov 01 16:30:55.828607 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:55.828562 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c97faba-9728-4c13-b99d-19fa20b7e85b-istio-envoy\") pod \"e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h\" (UID: \"7c97faba-9728-4c13-b99d-19fa20b7e85b\") " pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" Nov 01 16:30:55.828757 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:55.828643 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c97faba-9728-4c13-b99d-19fa20b7e85b-istio-token\") pod \"e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h\" (UID: \"7c97faba-9728-4c13-b99d-19fa20b7e85b\") " pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" Nov 01 16:30:55.828757 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:55.828744 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c97faba-9728-4c13-b99d-19fa20b7e85b-istio-data\") pod \"e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h\" (UID: \"7c97faba-9728-4c13-b99d-19fa20b7e85b\") " pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" Nov 01 16:30:55.828933 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:55.828836 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fszcr\" (UniqueName: \"kubernetes.io/projected/7c97faba-9728-4c13-b99d-19fa20b7e85b-kube-api-access-fszcr\") pod \"e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h\" (UID: \"7c97faba-9728-4c13-b99d-19fa20b7e85b\") " pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" Nov 01 16:30:55.829066 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:55.829044 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c97faba-9728-4c13-b99d-19fa20b7e85b-credential-socket\") pod \"e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h\" (UID: \"7c97faba-9728-4c13-b99d-19fa20b7e85b\") " pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" Nov 01 16:30:55.829169 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:55.829137 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6cdf1c8-f85e-4abf-af8a-8a695f7302de-maas-api-openshift-ca" pod "e6cdf1c8-f85e-4abf-af8a-8a695f7302de" (UID: "e6cdf1c8-f85e-4abf-af8a-8a695f7302de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:30:55.829585 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:55.829316 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c97faba-9728-4c13-b99d-19fa20b7e85b-workload-socket\") pod \"e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h\" (UID: \"7c97faba-9728-4c13-b99d-19fa20b7e85b\") " pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" Nov 01 16:30:55.829585 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:55.829389 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c97faba-9728-4c13-b99d-19fa20b7e85b-workload-certs\") pod \"e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h\" (UID: \"7c97faba-9728-4c13-b99d-19fa20b7e85b\") " pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" Nov 01 16:30:55.829585 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:55.829428 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c97faba-9728-4c13-b99d-19fa20b7e85b-istio-podinfo\") pod \"e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h\" (UID: \"7c97faba-9728-4c13-b99d-19fa20b7e85b\") " pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" Nov 01 16:30:55.829585 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:55.829558 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c97faba-9728-4c13-b99d-19fa20b7e85b-istiod-ca-cert\") pod \"e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h\" (UID: \"7c97faba-9728-4c13-b99d-19fa20b7e85b\") " pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" Nov 01 16:30:55.829939 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:55.829764 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6cdf1c8-f85e-4abf-af8a-8a695f7302de-maas-api-openshift-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:55.831511 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:55.831483 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6cdf1c8-f85e-4abf-af8a-8a695f7302de-kube-api-access-9m79h" pod "e6cdf1c8-f85e-4abf-af8a-8a695f7302de" (UID: "e6cdf1c8-f85e-4abf-af8a-8a695f7302de"). InnerVolumeSpecName "kube-api-access-9m79h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:55.831977 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:55.831955 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6cdf1c8-f85e-4abf-af8a-8a695f7302de-maas-api-xks-ca" pod "e6cdf1c8-f85e-4abf-af8a-8a695f7302de" (UID: "e6cdf1c8-f85e-4abf-af8a-8a695f7302de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:30:55.846261 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:55.846234 4363 patch_prober.go:29] interesting pod/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Nov 01 16:30:55.846367 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:55.846295 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" podUID="f3655831-2619-45f1-9da3-5bbe7dca87f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Nov 01 16:30:56.270055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.268643 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c97faba-9728-4c13-b99d-19fa20b7e85b-istio-token\") pod \"e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h\" (UID: \"7c97faba-9728-4c13-b99d-19fa20b7e85b\") " pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" Nov 01 16:30:56.270055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.268791 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c97faba-9728-4c13-b99d-19fa20b7e85b-istio-data\") pod \"e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h\" (UID: \"7c97faba-9728-4c13-b99d-19fa20b7e85b\") " pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" Nov 01 16:30:56.270055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.268841 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fszcr\" (UniqueName: \"kubernetes.io/projected/7c97faba-9728-4c13-b99d-19fa20b7e85b-kube-api-access-fszcr\") pod \"e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h\" (UID: \"7c97faba-9728-4c13-b99d-19fa20b7e85b\") " pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" Nov 01 16:30:56.270055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.268893 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c97faba-9728-4c13-b99d-19fa20b7e85b-credential-socket\") pod \"e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h\" (UID: \"7c97faba-9728-4c13-b99d-19fa20b7e85b\") " pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" Nov 01 16:30:56.270055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.269009 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c97faba-9728-4c13-b99d-19fa20b7e85b-workload-socket\") pod \"e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h\" (UID: \"7c97faba-9728-4c13-b99d-19fa20b7e85b\") " pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" Nov 01 16:30:56.270055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.269098 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c97faba-9728-4c13-b99d-19fa20b7e85b-workload-certs\") pod \"e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h\" (UID: \"7c97faba-9728-4c13-b99d-19fa20b7e85b\") " pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" Nov 01 16:30:56.270055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.269122 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c97faba-9728-4c13-b99d-19fa20b7e85b-istio-podinfo\") pod \"e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h\" (UID: \"7c97faba-9728-4c13-b99d-19fa20b7e85b\") " pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" Nov 01 16:30:56.270055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.269214 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c97faba-9728-4c13-b99d-19fa20b7e85b-istiod-ca-cert\") pod \"e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h\" (UID: \"7c97faba-9728-4c13-b99d-19fa20b7e85b\") " pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" Nov 01 16:30:56.270055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.269388 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c97faba-9728-4c13-b99d-19fa20b7e85b-istio-envoy\") pod \"e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h\" (UID: \"7c97faba-9728-4c13-b99d-19fa20b7e85b\") " pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" Nov 01 16:30:56.270988 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.270656 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c97faba-9728-4c13-b99d-19fa20b7e85b-credential-socket\") pod \"e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h\" (UID: \"7c97faba-9728-4c13-b99d-19fa20b7e85b\") " pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" Nov 01 16:30:56.271078 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.270988 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m79h\" (UniqueName: \"kubernetes.io/projected/e6cdf1c8-f85e-4abf-af8a-8a695f7302de-kube-api-access-9m79h\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:56.271078 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.271042 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6cdf1c8-f85e-4abf-af8a-8a695f7302de-maas-api-xks-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:56.272677 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.271872 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c97faba-9728-4c13-b99d-19fa20b7e85b-istio-data\") pod \"e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h\" (UID: \"7c97faba-9728-4c13-b99d-19fa20b7e85b\") " pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" Nov 01 16:30:56.272677 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.272170 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c97faba-9728-4c13-b99d-19fa20b7e85b-workload-certs\") pod \"e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h\" (UID: \"7c97faba-9728-4c13-b99d-19fa20b7e85b\") " pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" Nov 01 16:30:56.272677 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.272427 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c97faba-9728-4c13-b99d-19fa20b7e85b-workload-socket\") pod \"e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h\" (UID: \"7c97faba-9728-4c13-b99d-19fa20b7e85b\") " pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" Nov 01 16:30:56.274313 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.273506 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c97faba-9728-4c13-b99d-19fa20b7e85b-istiod-ca-cert\") pod \"e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h\" (UID: \"7c97faba-9728-4c13-b99d-19fa20b7e85b\") " pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" Nov 01 16:30:56.325242 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.298911 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c97faba-9728-4c13-b99d-19fa20b7e85b-istio-envoy\") pod \"e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h\" (UID: \"7c97faba-9728-4c13-b99d-19fa20b7e85b\") " pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" Nov 01 16:30:56.325242 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.301590 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c97faba-9728-4c13-b99d-19fa20b7e85b-istio-podinfo\") pod \"e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h\" (UID: \"7c97faba-9728-4c13-b99d-19fa20b7e85b\") " pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" Nov 01 16:30:56.325242 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.303186 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c97faba-9728-4c13-b99d-19fa20b7e85b-istio-token\") pod \"e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h\" (UID: \"7c97faba-9728-4c13-b99d-19fa20b7e85b\") " pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" Nov 01 16:30:56.325242 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.305983 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fszcr\" (UniqueName: \"kubernetes.io/projected/7c97faba-9728-4c13-b99d-19fa20b7e85b-kube-api-access-fszcr\") pod \"e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h\" (UID: \"7c97faba-9728-4c13-b99d-19fa20b7e85b\") " pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" Nov 01 16:30:56.345230 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.345193 4363 generic.go:356] "Generic (PLEG): container finished" podID="474f6a6d-0ab2-4f46-b9b8-f29f577f7787" containerID="6ba8a563e37b7d8b4e8f0a5869b2f43fed73a62556405e59548bfb3ffa3b15ef" exitCode=0 Nov 01 16:30:56.345392 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.345262 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e1e630/ipp-route-e1e630-kserve-85dbb46dd5-qqv5q" event={"ID":"474f6a6d-0ab2-4f46-b9b8-f29f577f7787","Type":"ContainerDied","Data":"6ba8a563e37b7d8b4e8f0a5869b2f43fed73a62556405e59548bfb3ffa3b15ef"} Nov 01 16:30:56.345392 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.345329 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e1e630/ipp-route-e1e630-kserve-85dbb46dd5-qqv5q" event={"ID":"474f6a6d-0ab2-4f46-b9b8-f29f577f7787","Type":"ContainerDied","Data":"d46c8c3ec19d6852059e405167e5134e1a7885010468ef6516a38a59502ec8ad"} Nov 01 16:30:56.345392 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.345345 4363 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d46c8c3ec19d6852059e405167e5134e1a7885010468ef6516a38a59502ec8ad" Nov 01 16:30:56.346777 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.346759 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e1e630/ipp-route-e1e630-kserve-85dbb46dd5-qqv5q" Nov 01 16:30:56.346886 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.346829 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e10868-s6xp2" event={"ID":"e6cdf1c8-f85e-4abf-af8a-8a695f7302de","Type":"ContainerDied","Data":"e869d99fd04aac1aff5d2868e9a62df42c9891d8b291039fe9c6a3476cdfeca2"} Nov 01 16:30:56.346886 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.346873 4363 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e869d99fd04aac1aff5d2868e9a62df42c9891d8b291039fe9c6a3476cdfeca2" Nov 01 16:30:56.346971 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.346936 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e10868-s6xp2" Nov 01 16:30:56.469364 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.469141 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" Nov 01 16:30:56.475329 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.473931 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/474f6a6d-0ab2-4f46-b9b8-f29f577f7787-home\" (UniqueName: \"kubernetes.io/empty-dir/474f6a6d-0ab2-4f46-b9b8-f29f577f7787-home\") pod \"474f6a6d-0ab2-4f46-b9b8-f29f577f7787\" (UID: \"474f6a6d-0ab2-4f46-b9b8-f29f577f7787\") " Nov 01 16:30:56.475329 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.473977 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/474f6a6d-0ab2-4f46-b9b8-f29f577f7787-tls-certs\" (UniqueName: \"kubernetes.io/secret/474f6a6d-0ab2-4f46-b9b8-f29f577f7787-tls-certs\") pod \"474f6a6d-0ab2-4f46-b9b8-f29f577f7787\" (UID: \"474f6a6d-0ab2-4f46-b9b8-f29f577f7787\") " Nov 01 16:30:56.475329 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.474085 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/474f6a6d-0ab2-4f46-b9b8-f29f577f7787-dshm\" (UniqueName: \"kubernetes.io/empty-dir/474f6a6d-0ab2-4f46-b9b8-f29f577f7787-dshm\") pod \"474f6a6d-0ab2-4f46-b9b8-f29f577f7787\" (UID: \"474f6a6d-0ab2-4f46-b9b8-f29f577f7787\") " Nov 01 16:30:56.475329 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.474143 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/474f6a6d-0ab2-4f46-b9b8-f29f577f7787-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/474f6a6d-0ab2-4f46-b9b8-f29f577f7787-model-cache\") pod \"474f6a6d-0ab2-4f46-b9b8-f29f577f7787\" (UID: \"474f6a6d-0ab2-4f46-b9b8-f29f577f7787\") " Nov 01 16:30:56.475329 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.474318 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/474f6a6d-0ab2-4f46-b9b8-f29f577f7787-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/474f6a6d-0ab2-4f46-b9b8-f29f577f7787-tmp-dir\") pod \"474f6a6d-0ab2-4f46-b9b8-f29f577f7787\" (UID: \"474f6a6d-0ab2-4f46-b9b8-f29f577f7787\") " Nov 01 16:30:56.475329 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.474366 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/474f6a6d-0ab2-4f46-b9b8-f29f577f7787-kube-api-access-wg6sf\" (UniqueName: \"kubernetes.io/projected/474f6a6d-0ab2-4f46-b9b8-f29f577f7787-kube-api-access-wg6sf\") pod \"474f6a6d-0ab2-4f46-b9b8-f29f577f7787\" (UID: \"474f6a6d-0ab2-4f46-b9b8-f29f577f7787\") " Nov 01 16:30:56.475329 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.474702 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/474f6a6d-0ab2-4f46-b9b8-f29f577f7787-model-cache" pod "474f6a6d-0ab2-4f46-b9b8-f29f577f7787" (UID: "474f6a6d-0ab2-4f46-b9b8-f29f577f7787"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:30:56.475329 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.474979 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/474f6a6d-0ab2-4f46-b9b8-f29f577f7787-home" pod "474f6a6d-0ab2-4f46-b9b8-f29f577f7787" (UID: "474f6a6d-0ab2-4f46-b9b8-f29f577f7787"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:30:56.475329 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.475248 4363 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/474f6a6d-0ab2-4f46-b9b8-f29f577f7787-home\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:56.475329 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.475259 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/474f6a6d-0ab2-4f46-b9b8-f29f577f7787-tmp-dir" pod "474f6a6d-0ab2-4f46-b9b8-f29f577f7787" (UID: "474f6a6d-0ab2-4f46-b9b8-f29f577f7787"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:30:56.475329 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.475269 4363 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/474f6a6d-0ab2-4f46-b9b8-f29f577f7787-model-cache\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:56.495066 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.494978 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/474f6a6d-0ab2-4f46-b9b8-f29f577f7787-dshm" pod "474f6a6d-0ab2-4f46-b9b8-f29f577f7787" (UID: "474f6a6d-0ab2-4f46-b9b8-f29f577f7787"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:30:56.495280 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.495207 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/474f6a6d-0ab2-4f46-b9b8-f29f577f7787-kube-api-access-wg6sf" pod "474f6a6d-0ab2-4f46-b9b8-f29f577f7787" (UID: "474f6a6d-0ab2-4f46-b9b8-f29f577f7787"). InnerVolumeSpecName "kube-api-access-wg6sf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:56.520040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.499760 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/474f6a6d-0ab2-4f46-b9b8-f29f577f7787-tls-certs" pod "474f6a6d-0ab2-4f46-b9b8-f29f577f7787" (UID: "474f6a6d-0ab2-4f46-b9b8-f29f577f7787"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:30:56.604547 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.577489 4363 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/474f6a6d-0ab2-4f46-b9b8-f29f577f7787-tmp-dir\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:56.604547 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.577518 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg6sf\" (UniqueName: \"kubernetes.io/projected/474f6a6d-0ab2-4f46-b9b8-f29f577f7787-kube-api-access-wg6sf\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:56.604547 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.577533 4363 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/474f6a6d-0ab2-4f46-b9b8-f29f577f7787-tls-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:56.604547 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.577548 4363 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/474f6a6d-0ab2-4f46-b9b8-f29f577f7787-dshm\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:56.856190 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.846272 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h"] Nov 01 16:30:56.856190 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.855098 4363 patch_prober.go:29] interesting pod/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Nov 01 16:30:56.856190 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.855155 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" podUID="f3655831-2619-45f1-9da3-5bbe7dca87f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Nov 01 16:30:56.867499 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:30:56.867084 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c97faba_9728_4c13_b99d_19fa20b7e85b.slice/crio-fdf0b4bdd4f6944f0868bdc636ba984adb0d494fe79c181b74370d8d23ac46ac WatchSource:0}: Error finding container fdf0b4bdd4f6944f0868bdc636ba984adb0d494fe79c181b74370d8d23ac46ac: Status 404 returned error can't find the container with id fdf0b4bdd4f6944f0868bdc636ba984adb0d494fe79c181b74370d8d23ac46ac Nov 01 16:30:56.873954 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.873206 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:30:56.874418 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.874305 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:30:56.874418 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:56.874388 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:30:57.064045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:57.063471 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-779fd7cf75-tnrpx"] Nov 01 16:30:57.180041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:57.172419 4363 cpu_manager.go:414] "containerMap: removing container" podUID="474f6a6d-0ab2-4f46-b9b8-f29f577f7787" containerName="main" Nov 01 16:30:57.180041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:57.177069 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="474f6a6d-0ab2-4f46-b9b8-f29f577f7787" containerName="main" Nov 01 16:30:57.180041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:57.177387 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="474f6a6d-0ab2-4f46-b9b8-f29f577f7787" containerName="main" Nov 01 16:30:57.235038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:57.229498 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-779fd7cf75-tnrpx"] Nov 01 16:30:57.235038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:57.229627 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-779fd7cf75-tnrpx" Nov 01 16:30:57.293282 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:57.256172 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-6b6cfc\"" Nov 01 16:30:57.313805 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:57.312727 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-6b6cfc-dockercfg-fwtx6\"" Nov 01 16:30:57.324045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:57.318059 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cee70f14-4206-4670-b671-82a568db2cc8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6b6cfc-779fd7cf75-tnrpx\" (UID: \"cee70f14-4206-4670-b671-82a568db2cc8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-779fd7cf75-tnrpx" Nov 01 16:30:57.324045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:57.318141 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85hkz\" (UniqueName: \"kubernetes.io/projected/cee70f14-4206-4670-b671-82a568db2cc8-kube-api-access-85hkz\") pod \"payload-pre-processing-e2e-ipp-b-6b6cfc-779fd7cf75-tnrpx\" (UID: \"cee70f14-4206-4670-b671-82a568db2cc8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-779fd7cf75-tnrpx" Nov 01 16:30:57.365179 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:57.365148 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" event={"ID":"7c97faba-9728-4c13-b99d-19fa20b7e85b","Type":"ContainerStarted","Data":"fdf0b4bdd4f6944f0868bdc636ba984adb0d494fe79c181b74370d8d23ac46ac"} Nov 01 16:30:57.365477 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:57.365460 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e1e630/ipp-route-e1e630-kserve-85dbb46dd5-qqv5q" Nov 01 16:30:57.421512 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:57.419072 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-e1e630/ipp-route-e1e630-kserve-85dbb46dd5-qqv5q"] Nov 01 16:30:57.424314 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:57.423195 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-e1e630/ipp-route-e1e630-kserve-85dbb46dd5-qqv5q"] Nov 01 16:30:57.425211 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:57.425185 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cee70f14-4206-4670-b671-82a568db2cc8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6b6cfc-779fd7cf75-tnrpx\" (UID: \"cee70f14-4206-4670-b671-82a568db2cc8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-779fd7cf75-tnrpx" Nov 01 16:30:57.425341 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:57.425227 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85hkz\" (UniqueName: \"kubernetes.io/projected/cee70f14-4206-4670-b671-82a568db2cc8-kube-api-access-85hkz\") pod \"payload-pre-processing-e2e-ipp-b-6b6cfc-779fd7cf75-tnrpx\" (UID: \"cee70f14-4206-4670-b671-82a568db2cc8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-779fd7cf75-tnrpx" Nov 01 16:30:57.426405 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:57.426381 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cee70f14-4206-4670-b671-82a568db2cc8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6b6cfc-779fd7cf75-tnrpx\" (UID: \"cee70f14-4206-4670-b671-82a568db2cc8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-779fd7cf75-tnrpx" Nov 01 16:30:57.434095 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:57.433511 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85hkz\" (UniqueName: \"kubernetes.io/projected/cee70f14-4206-4670-b671-82a568db2cc8-kube-api-access-85hkz\") pod \"payload-pre-processing-e2e-ipp-b-6b6cfc-779fd7cf75-tnrpx\" (UID: \"cee70f14-4206-4670-b671-82a568db2cc8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-779fd7cf75-tnrpx" Nov 01 16:30:57.662361 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:57.662326 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-779fd7cf75-tnrpx" Nov 01 16:30:58.001241 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:58.001189 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-779fd7cf75-tnrpx"] Nov 01 16:30:58.051501 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:58.051468 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" containerID="cri-o://2df13e314347272901587d10fdc1ea1131d787ea6a562cd60bbfd562130826ea" gracePeriod=45 Nov 01 16:30:58.153059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:58.150595 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" Nov 01 16:30:58.153059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:58.151442 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" Nov 01 16:30:58.154374 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:58.154350 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" Nov 01 16:30:58.402050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:58.396945 4363 generic.go:356] "Generic (PLEG): container finished" podID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerID="2df13e314347272901587d10fdc1ea1131d787ea6a562cd60bbfd562130826ea" exitCode=0 Nov 01 16:30:58.402050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:58.397116 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" event={"ID":"0049d3d4-16f8-4b77-87fa-dacf0921ad20","Type":"ContainerDied","Data":"2df13e314347272901587d10fdc1ea1131d787ea6a562cd60bbfd562130826ea"} Nov 01 16:30:58.415610 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:58.414985 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" event={"ID":"7c97faba-9728-4c13-b99d-19fa20b7e85b","Type":"ContainerStarted","Data":"be5a939df28fce2552f2b99c55ede881a6d3f0bdbfd0604f45cfd72a5ee4eaa3"} Nov 01 16:30:58.421857 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:58.420463 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-779fd7cf75-tnrpx" event={"ID":"cee70f14-4206-4670-b671-82a568db2cc8","Type":"ContainerStarted","Data":"9756db03eed437c23547e5d5bd8ddb6bf4171b17251a853537108b2cdf679d26"} Nov 01 16:30:58.421857 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:58.420520 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-779fd7cf75-tnrpx" Nov 01 16:30:58.421857 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:58.420547 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-779fd7cf75-tnrpx" event={"ID":"cee70f14-4206-4670-b671-82a568db2cc8","Type":"ContainerStarted","Data":"dad16c4960a03c533653658469d28d0720581b0b94d02483321ae7d4265482fc"} Nov 01 16:30:58.443267 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:58.442145 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" podStartSLOduration=3.442124211 podStartE2EDuration="3.442124211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:30:58.43695617 +0000 UTC m=+2591.768693822" watchObservedRunningTime="2026-11-01 16:30:58.442124211 +0000 UTC m=+2591.773861862" Nov 01 16:30:58.451231 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:58.451207 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:30:58.467604 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:58.467541 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-779fd7cf75-tnrpx" podStartSLOduration=2.467521645 podStartE2EDuration="2.467521645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:30:58.45347427 +0000 UTC m=+2591.785211920" watchObservedRunningTime="2026-11-01 16:30:58.467521645 +0000 UTC m=+2591.799259296" Nov 01 16:30:58.469562 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:58.469531 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" Nov 01 16:30:58.488276 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:58.488248 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" Nov 01 16:30:58.591131 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:58.582536 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0049d3d4-16f8-4b77-87fa-dacf0921ad20-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0049d3d4-16f8-4b77-87fa-dacf0921ad20-model-cache\") pod \"0049d3d4-16f8-4b77-87fa-dacf0921ad20\" (UID: \"0049d3d4-16f8-4b77-87fa-dacf0921ad20\") " Nov 01 16:30:58.591131 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:58.582586 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0049d3d4-16f8-4b77-87fa-dacf0921ad20-kube-api-access-p6j9q\" (UniqueName: \"kubernetes.io/projected/0049d3d4-16f8-4b77-87fa-dacf0921ad20-kube-api-access-p6j9q\") pod \"0049d3d4-16f8-4b77-87fa-dacf0921ad20\" (UID: \"0049d3d4-16f8-4b77-87fa-dacf0921ad20\") " Nov 01 16:30:58.591131 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:58.582649 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0049d3d4-16f8-4b77-87fa-dacf0921ad20-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0049d3d4-16f8-4b77-87fa-dacf0921ad20-tmp-dir\") pod \"0049d3d4-16f8-4b77-87fa-dacf0921ad20\" (UID: \"0049d3d4-16f8-4b77-87fa-dacf0921ad20\") " Nov 01 16:30:58.591131 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:58.582705 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0049d3d4-16f8-4b77-87fa-dacf0921ad20-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0049d3d4-16f8-4b77-87fa-dacf0921ad20-dshm\") pod \"0049d3d4-16f8-4b77-87fa-dacf0921ad20\" (UID: \"0049d3d4-16f8-4b77-87fa-dacf0921ad20\") " Nov 01 16:30:58.591131 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:58.583028 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0049d3d4-16f8-4b77-87fa-dacf0921ad20-tls-certs\" (UniqueName: \"kubernetes.io/secret/0049d3d4-16f8-4b77-87fa-dacf0921ad20-tls-certs\") pod \"0049d3d4-16f8-4b77-87fa-dacf0921ad20\" (UID: \"0049d3d4-16f8-4b77-87fa-dacf0921ad20\") " Nov 01 16:30:58.591131 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:58.583065 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0049d3d4-16f8-4b77-87fa-dacf0921ad20-home\" (UniqueName: \"kubernetes.io/empty-dir/0049d3d4-16f8-4b77-87fa-dacf0921ad20-home\") pod \"0049d3d4-16f8-4b77-87fa-dacf0921ad20\" (UID: \"0049d3d4-16f8-4b77-87fa-dacf0921ad20\") " Nov 01 16:30:58.608362 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:58.606344 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0049d3d4-16f8-4b77-87fa-dacf0921ad20-tmp-dir" pod "0049d3d4-16f8-4b77-87fa-dacf0921ad20" (UID: "0049d3d4-16f8-4b77-87fa-dacf0921ad20"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:30:58.608362 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:58.606575 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0049d3d4-16f8-4b77-87fa-dacf0921ad20-model-cache" pod "0049d3d4-16f8-4b77-87fa-dacf0921ad20" (UID: "0049d3d4-16f8-4b77-87fa-dacf0921ad20"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:30:58.620035 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:58.613822 4363 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0049d3d4-16f8-4b77-87fa-dacf0921ad20-model-cache\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:58.620035 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:58.613855 4363 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0049d3d4-16f8-4b77-87fa-dacf0921ad20-tmp-dir\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:58.620035 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:58.616147 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0049d3d4-16f8-4b77-87fa-dacf0921ad20-home" pod "0049d3d4-16f8-4b77-87fa-dacf0921ad20" (UID: "0049d3d4-16f8-4b77-87fa-dacf0921ad20"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:30:58.630330 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:58.630292 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0049d3d4-16f8-4b77-87fa-dacf0921ad20-kube-api-access-p6j9q" pod "0049d3d4-16f8-4b77-87fa-dacf0921ad20" (UID: "0049d3d4-16f8-4b77-87fa-dacf0921ad20"). InnerVolumeSpecName "kube-api-access-p6j9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:58.663351 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:58.663300 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0049d3d4-16f8-4b77-87fa-dacf0921ad20-tls-certs" pod "0049d3d4-16f8-4b77-87fa-dacf0921ad20" (UID: "0049d3d4-16f8-4b77-87fa-dacf0921ad20"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:30:58.663676 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:58.663652 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0049d3d4-16f8-4b77-87fa-dacf0921ad20-dshm" pod "0049d3d4-16f8-4b77-87fa-dacf0921ad20" (UID: "0049d3d4-16f8-4b77-87fa-dacf0921ad20"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:30:58.715530 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:58.715494 4363 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0049d3d4-16f8-4b77-87fa-dacf0921ad20-tls-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:58.715697 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:58.715528 4363 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0049d3d4-16f8-4b77-87fa-dacf0921ad20-home\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:58.715697 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:58.715559 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6j9q\" (UniqueName: \"kubernetes.io/projected/0049d3d4-16f8-4b77-87fa-dacf0921ad20-kube-api-access-p6j9q\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:58.715697 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:58.715573 4363 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0049d3d4-16f8-4b77-87fa-dacf0921ad20-dshm\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:58.815333 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:58.815293 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="474f6a6d-0ab2-4f46-b9b8-f29f577f7787" path="/var/lib/kubelet/pods/474f6a6d-0ab2-4f46-b9b8-f29f577f7787/volumes" Nov 01 16:30:59.431515 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:59.431482 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" Nov 01 16:30:59.431515 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:59.431500 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t" event={"ID":"0049d3d4-16f8-4b77-87fa-dacf0921ad20","Type":"ContainerDied","Data":"e3722284e77368ee55ec735b3e3537005497281011e353a1dec6ce5d59f9b3af"} Nov 01 16:30:59.432225 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:59.431548 4363 scope.go:122] "RemoveContainer" containerID="2df13e314347272901587d10fdc1ea1131d787ea6a562cd60bbfd562130826ea" Nov 01 16:30:59.432225 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:59.432130 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" Nov 01 16:30:59.433072 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:59.432927 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" Nov 01 16:30:59.505038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:59.503091 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t"] Nov 01 16:30:59.530520 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:30:59.530479 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-4a083d/distinct-w0-4a083d-kserve-785988c485-2z65t"] Nov 01 16:31:00.801291 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:00.801257 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" path="/var/lib/kubelet/pods/0049d3d4-16f8-4b77-87fa-dacf0921ad20/volumes" Nov 01 16:31:09.433778 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:09.433746 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-779fd7cf75-tnrpx" Nov 01 16:31:17.606288 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:17.606250 4363 generic.go:356] "Generic (PLEG): container finished" podID="69113cc0-84e0-4de6-8078-aa79ad5cad66" containerID="23fa7d7a5213c3f856678ea3caab08f627ff8f5dd3bc2456e2f45811ed5a5ab0" exitCode=0 Nov 01 16:31:17.606761 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:17.606326 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" event={"ID":"69113cc0-84e0-4de6-8078-aa79ad5cad66","Type":"ContainerDied","Data":"23fa7d7a5213c3f856678ea3caab08f627ff8f5dd3bc2456e2f45811ed5a5ab0"} Nov 01 16:31:17.606946 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:17.606928 4363 scope.go:122] "RemoveContainer" containerID="23fa7d7a5213c3f856678ea3caab08f627ff8f5dd3bc2456e2f45811ed5a5ab0" Nov 01 16:31:17.609288 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:17.609254 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:31:17.609413 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:17.609345 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:31:17.609413 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:17.609391 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:31:18.325471 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:31:18.325429 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:31:18.620850 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:18.620806 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" event={"ID":"69113cc0-84e0-4de6-8078-aa79ad5cad66","Type":"ContainerStarted","Data":"2d98c1ea18ed165181d0193fc3ef20f590882d103df04962c57309157bc17ef7"} Nov 01 16:31:19.041298 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:19.041216 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" Nov 01 16:31:19.042113 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:19.042086 4363 patch_prober.go:29] interesting pod/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" start-of-body= Nov 01 16:31:19.042238 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:19.042145 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" podUID="69113cc0-84e0-4de6-8078-aa79ad5cad66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" Nov 01 16:31:20.038040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:20.037044 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj"] Nov 01 16:31:20.038040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:20.037312 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" podUID="d4b55ba5-b1af-492d-abe2-1baa85947260" containerName="istio-proxy" containerID="cri-o://b7529db873a8893bcca0e736639bd1bd76e4cc7538633cae8d240320c5cc1fce" gracePeriod=30 Nov 01 16:31:20.042061 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:20.041668 4363 patch_prober.go:29] interesting pod/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" start-of-body= Nov 01 16:31:20.042061 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:20.041732 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" podUID="69113cc0-84e0-4de6-8078-aa79ad5cad66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" Nov 01 16:31:21.041865 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:21.041827 4363 patch_prober.go:29] interesting pod/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" start-of-body= Nov 01 16:31:21.042412 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:21.041884 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" podUID="69113cc0-84e0-4de6-8078-aa79ad5cad66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" Nov 01 16:31:22.041132 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:22.041095 4363 patch_prober.go:29] interesting pod/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" start-of-body= Nov 01 16:31:22.041361 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:22.041161 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" podUID="69113cc0-84e0-4de6-8078-aa79ad5cad66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" Nov 01 16:31:23.042117 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:23.042082 4363 patch_prober.go:29] interesting pod/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" start-of-body= Nov 01 16:31:23.042623 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:23.042146 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" podUID="69113cc0-84e0-4de6-8078-aa79ad5cad66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" Nov 01 16:31:23.206152 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:23.206097 4363 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:31:23.206368 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:23.206177 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" podUID="d4b55ba5-b1af-492d-abe2-1baa85947260" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:31:23.746548 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:23.746511 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" Nov 01 16:31:24.041742 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:24.041648 4363 patch_prober.go:29] interesting pod/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" start-of-body= Nov 01 16:31:24.041742 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:24.041706 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" podUID="69113cc0-84e0-4de6-8078-aa79ad5cad66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" Nov 01 16:31:24.736286 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:24.736255 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-9c5644f64-xrw75"] Nov 01 16:31:24.736837 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:24.736512 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-9c5644f64-xrw75" podUID="d60400a5-9792-41ec-90c2-f1e76bee301b" containerName="payload-processing" containerID="cri-o://2366b7adb832230aaabbaf349f114b32073a6512c4bd3e0253a1cf6ed09cdc8d" gracePeriod=30 Nov 01 16:31:25.122037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.118593 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" Nov 01 16:31:25.150704 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.140203 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" Nov 01 16:31:25.347601 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.347560 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn"] Nov 01 16:31:25.348154 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.348118 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" podUID="9a4c04ae-eac9-46b2-b5fb-b36585723722" containerName="istio-proxy" containerID="cri-o://658d6ed056d3a59ace04f1b865ede340f35cf6a281e0637a358929aa22ab9038" gracePeriod=30 Nov 01 16:31:25.562417 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.562354 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-9c5644f64-xrw75" Nov 01 16:31:25.571558 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.571524 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d60400a5-9792-41ec-90c2-f1e76bee301b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d60400a5-9792-41ec-90c2-f1e76bee301b-plugins-config-volume\") pod \"d60400a5-9792-41ec-90c2-f1e76bee301b\" (UID: \"d60400a5-9792-41ec-90c2-f1e76bee301b\") " Nov 01 16:31:25.571825 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.571806 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d60400a5-9792-41ec-90c2-f1e76bee301b-kube-api-access-6v9cg\" (UniqueName: \"kubernetes.io/projected/d60400a5-9792-41ec-90c2-f1e76bee301b-kube-api-access-6v9cg\") pod \"d60400a5-9792-41ec-90c2-f1e76bee301b\" (UID: \"d60400a5-9792-41ec-90c2-f1e76bee301b\") " Nov 01 16:31:25.574055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.572778 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d60400a5-9792-41ec-90c2-f1e76bee301b-plugins-config-volume" pod "d60400a5-9792-41ec-90c2-f1e76bee301b" (UID: "d60400a5-9792-41ec-90c2-f1e76bee301b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:31:25.612555 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.612365 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d60400a5-9792-41ec-90c2-f1e76bee301b-kube-api-access-6v9cg" pod "d60400a5-9792-41ec-90c2-f1e76bee301b" (UID: "d60400a5-9792-41ec-90c2-f1e76bee301b"). InnerVolumeSpecName "kube-api-access-6v9cg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:31:25.677605 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.677505 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v9cg\" (UniqueName: \"kubernetes.io/projected/d60400a5-9792-41ec-90c2-f1e76bee301b-kube-api-access-6v9cg\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:25.678805 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.678109 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d60400a5-9792-41ec-90c2-f1e76bee301b-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:25.742706 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.742675 4363 generic.go:356] "Generic (PLEG): container finished" podID="d60400a5-9792-41ec-90c2-f1e76bee301b" containerID="2366b7adb832230aaabbaf349f114b32073a6512c4bd3e0253a1cf6ed09cdc8d" exitCode=0 Nov 01 16:31:25.743127 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.742777 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-9c5644f64-xrw75" Nov 01 16:31:25.743127 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.742816 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-9c5644f64-xrw75" event={"ID":"d60400a5-9792-41ec-90c2-f1e76bee301b","Type":"ContainerDied","Data":"2366b7adb832230aaabbaf349f114b32073a6512c4bd3e0253a1cf6ed09cdc8d"} Nov 01 16:31:25.743127 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.742878 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-9c5644f64-xrw75" event={"ID":"d60400a5-9792-41ec-90c2-f1e76bee301b","Type":"ContainerDied","Data":"aa66c5f735fc82cdfec95e70fef6434e6276464214d69e6f3803ee93eb097f41"} Nov 01 16:31:25.743127 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.742920 4363 scope.go:122] "RemoveContainer" containerID="2366b7adb832230aaabbaf349f114b32073a6512c4bd3e0253a1cf6ed09cdc8d" Nov 01 16:31:25.744837 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.744814 4363 generic.go:356] "Generic (PLEG): container finished" podID="d4b55ba5-b1af-492d-abe2-1baa85947260" containerID="b7529db873a8893bcca0e736639bd1bd76e4cc7538633cae8d240320c5cc1fce" exitCode=0 Nov 01 16:31:25.745188 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.745160 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" event={"ID":"d4b55ba5-b1af-492d-abe2-1baa85947260","Type":"ContainerDied","Data":"b7529db873a8893bcca0e736639bd1bd76e4cc7538633cae8d240320c5cc1fce"} Nov 01 16:31:25.749293 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.749274 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" Nov 01 16:31:25.755314 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.755298 4363 scope.go:122] "RemoveContainer" containerID="2366b7adb832230aaabbaf349f114b32073a6512c4bd3e0253a1cf6ed09cdc8d" Nov 01 16:31:25.755573 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:31:25.755554 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2366b7adb832230aaabbaf349f114b32073a6512c4bd3e0253a1cf6ed09cdc8d\": container with ID starting with 2366b7adb832230aaabbaf349f114b32073a6512c4bd3e0253a1cf6ed09cdc8d not found: ID does not exist" containerID="2366b7adb832230aaabbaf349f114b32073a6512c4bd3e0253a1cf6ed09cdc8d" Nov 01 16:31:25.755630 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.755582 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2366b7adb832230aaabbaf349f114b32073a6512c4bd3e0253a1cf6ed09cdc8d"} err="failed to get container status \"2366b7adb832230aaabbaf349f114b32073a6512c4bd3e0253a1cf6ed09cdc8d\": rpc error: code = NotFound desc = could not find container \"2366b7adb832230aaabbaf349f114b32073a6512c4bd3e0253a1cf6ed09cdc8d\": container with ID starting with 2366b7adb832230aaabbaf349f114b32073a6512c4bd3e0253a1cf6ed09cdc8d not found: ID does not exist" Nov 01 16:31:25.791429 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.791384 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-9c5644f64-xrw75"] Nov 01 16:31:25.798171 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.798148 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-9c5644f64-xrw75"] Nov 01 16:31:25.880796 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.880750 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4b55ba5-b1af-492d-abe2-1baa85947260-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4b55ba5-b1af-492d-abe2-1baa85947260-istio-envoy\") pod \"d4b55ba5-b1af-492d-abe2-1baa85947260\" (UID: \"d4b55ba5-b1af-492d-abe2-1baa85947260\") " Nov 01 16:31:25.881267 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.881234 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4b55ba5-b1af-492d-abe2-1baa85947260-kube-api-access-srx75\" (UniqueName: \"kubernetes.io/projected/d4b55ba5-b1af-492d-abe2-1baa85947260-kube-api-access-srx75\") pod \"d4b55ba5-b1af-492d-abe2-1baa85947260\" (UID: \"d4b55ba5-b1af-492d-abe2-1baa85947260\") " Nov 01 16:31:25.881394 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.881284 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4b55ba5-b1af-492d-abe2-1baa85947260-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b55ba5-b1af-492d-abe2-1baa85947260-credential-socket\") pod \"d4b55ba5-b1af-492d-abe2-1baa85947260\" (UID: \"d4b55ba5-b1af-492d-abe2-1baa85947260\") " Nov 01 16:31:25.881394 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.881320 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4b55ba5-b1af-492d-abe2-1baa85947260-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4b55ba5-b1af-492d-abe2-1baa85947260-istio-data\") pod \"d4b55ba5-b1af-492d-abe2-1baa85947260\" (UID: \"d4b55ba5-b1af-492d-abe2-1baa85947260\") " Nov 01 16:31:25.881394 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.881362 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4b55ba5-b1af-492d-abe2-1baa85947260-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4b55ba5-b1af-492d-abe2-1baa85947260-workload-certs\") pod \"d4b55ba5-b1af-492d-abe2-1baa85947260\" (UID: \"d4b55ba5-b1af-492d-abe2-1baa85947260\") " Nov 01 16:31:25.881575 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.881395 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4b55ba5-b1af-492d-abe2-1baa85947260-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4b55ba5-b1af-492d-abe2-1baa85947260-istiod-ca-cert\") pod \"d4b55ba5-b1af-492d-abe2-1baa85947260\" (UID: \"d4b55ba5-b1af-492d-abe2-1baa85947260\") " Nov 01 16:31:25.881575 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.881481 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4b55ba5-b1af-492d-abe2-1baa85947260-istio-token\" (UniqueName: \"kubernetes.io/projected/d4b55ba5-b1af-492d-abe2-1baa85947260-istio-token\") pod \"d4b55ba5-b1af-492d-abe2-1baa85947260\" (UID: \"d4b55ba5-b1af-492d-abe2-1baa85947260\") " Nov 01 16:31:25.881809 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.881786 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b55ba5-b1af-492d-abe2-1baa85947260-credential-socket" pod "d4b55ba5-b1af-492d-abe2-1baa85947260" (UID: "d4b55ba5-b1af-492d-abe2-1baa85947260"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:31:25.881865 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.881800 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b55ba5-b1af-492d-abe2-1baa85947260-istio-data" pod "d4b55ba5-b1af-492d-abe2-1baa85947260" (UID: "d4b55ba5-b1af-492d-abe2-1baa85947260"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:31:25.881998 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.881975 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4b55ba5-b1af-492d-abe2-1baa85947260-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4b55ba5-b1af-492d-abe2-1baa85947260-istio-podinfo\") pod \"d4b55ba5-b1af-492d-abe2-1baa85947260\" (UID: \"d4b55ba5-b1af-492d-abe2-1baa85947260\") " Nov 01 16:31:25.882166 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.882080 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4b55ba5-b1af-492d-abe2-1baa85947260-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b55ba5-b1af-492d-abe2-1baa85947260-workload-socket\") pod \"d4b55ba5-b1af-492d-abe2-1baa85947260\" (UID: \"d4b55ba5-b1af-492d-abe2-1baa85947260\") " Nov 01 16:31:25.882166 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.881985 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b55ba5-b1af-492d-abe2-1baa85947260-workload-certs" pod "d4b55ba5-b1af-492d-abe2-1baa85947260" (UID: "d4b55ba5-b1af-492d-abe2-1baa85947260"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:31:25.882310 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.882186 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4b55ba5-b1af-492d-abe2-1baa85947260-istiod-ca-cert" pod "d4b55ba5-b1af-492d-abe2-1baa85947260" (UID: "d4b55ba5-b1af-492d-abe2-1baa85947260"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:31:25.882310 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.882270 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b55ba5-b1af-492d-abe2-1baa85947260-workload-socket" pod "d4b55ba5-b1af-492d-abe2-1baa85947260" (UID: "d4b55ba5-b1af-492d-abe2-1baa85947260"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:31:25.882808 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.882788 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b55ba5-b1af-492d-abe2-1baa85947260-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:25.882901 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.882813 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4b55ba5-b1af-492d-abe2-1baa85947260-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:25.882901 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.882828 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4b55ba5-b1af-492d-abe2-1baa85947260-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:25.882901 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.882841 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4b55ba5-b1af-492d-abe2-1baa85947260-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:25.882901 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.882853 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b55ba5-b1af-492d-abe2-1baa85947260-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:25.885053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.884598 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4b55ba5-b1af-492d-abe2-1baa85947260-istio-token" pod "d4b55ba5-b1af-492d-abe2-1baa85947260" (UID: "d4b55ba5-b1af-492d-abe2-1baa85947260"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:31:25.885053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.884738 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4b55ba5-b1af-492d-abe2-1baa85947260-istio-podinfo" pod "d4b55ba5-b1af-492d-abe2-1baa85947260" (UID: "d4b55ba5-b1af-492d-abe2-1baa85947260"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:31:25.885053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.884855 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b55ba5-b1af-492d-abe2-1baa85947260-istio-envoy" pod "d4b55ba5-b1af-492d-abe2-1baa85947260" (UID: "d4b55ba5-b1af-492d-abe2-1baa85947260"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:31:25.886082 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.885986 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4b55ba5-b1af-492d-abe2-1baa85947260-kube-api-access-srx75" pod "d4b55ba5-b1af-492d-abe2-1baa85947260" (UID: "d4b55ba5-b1af-492d-abe2-1baa85947260"). InnerVolumeSpecName "kube-api-access-srx75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:31:25.984540 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.984466 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4b55ba5-b1af-492d-abe2-1baa85947260-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:25.984540 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.984497 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srx75\" (UniqueName: \"kubernetes.io/projected/d4b55ba5-b1af-492d-abe2-1baa85947260-kube-api-access-srx75\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:25.984540 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.984508 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4b55ba5-b1af-492d-abe2-1baa85947260-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:25.984540 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:25.984517 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4b55ba5-b1af-492d-abe2-1baa85947260-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:26.754668 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:26.754635 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" Nov 01 16:31:26.754668 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:26.754650 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj" event={"ID":"d4b55ba5-b1af-492d-abe2-1baa85947260","Type":"ContainerDied","Data":"68a1fccdbb906aa5efc7d432639854440eec39c75ca1b92dc4dd9961de2b04e1"} Nov 01 16:31:26.755261 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:26.754736 4363 scope.go:122] "RemoveContainer" containerID="b7529db873a8893bcca0e736639bd1bd76e4cc7538633cae8d240320c5cc1fce" Nov 01 16:31:26.781514 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:26.781197 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj"] Nov 01 16:31:26.785737 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:26.785708 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-z5wwj"] Nov 01 16:31:26.801839 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:26.801809 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4b55ba5-b1af-492d-abe2-1baa85947260" path="/var/lib/kubelet/pods/d4b55ba5-b1af-492d-abe2-1baa85947260/volumes" Nov 01 16:31:26.802276 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:26.802258 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d60400a5-9792-41ec-90c2-f1e76bee301b" path="/var/lib/kubelet/pods/d60400a5-9792-41ec-90c2-f1e76bee301b/volumes" Nov 01 16:31:30.735147 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:30.735120 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" Nov 01 16:31:30.794856 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:30.794820 4363 generic.go:356] "Generic (PLEG): container finished" podID="9a4c04ae-eac9-46b2-b5fb-b36585723722" containerID="658d6ed056d3a59ace04f1b865ede340f35cf6a281e0637a358929aa22ab9038" exitCode=0 Nov 01 16:31:30.795105 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:30.794932 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" Nov 01 16:31:30.800327 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:30.800297 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a4c04ae-eac9-46b2-b5fb-b36585723722-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a4c04ae-eac9-46b2-b5fb-b36585723722-credential-socket\") pod \"9a4c04ae-eac9-46b2-b5fb-b36585723722\" (UID: \"9a4c04ae-eac9-46b2-b5fb-b36585723722\") " Nov 01 16:31:30.800500 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:30.800476 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a4c04ae-eac9-46b2-b5fb-b36585723722-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a4c04ae-eac9-46b2-b5fb-b36585723722-istio-envoy\") pod \"9a4c04ae-eac9-46b2-b5fb-b36585723722\" (UID: \"9a4c04ae-eac9-46b2-b5fb-b36585723722\") " Nov 01 16:31:30.800575 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:30.800517 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a4c04ae-eac9-46b2-b5fb-b36585723722-istio-token\" (UniqueName: \"kubernetes.io/projected/9a4c04ae-eac9-46b2-b5fb-b36585723722-istio-token\") pod \"9a4c04ae-eac9-46b2-b5fb-b36585723722\" (UID: \"9a4c04ae-eac9-46b2-b5fb-b36585723722\") " Nov 01 16:31:30.800575 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:30.800561 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a4c04ae-eac9-46b2-b5fb-b36585723722-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a4c04ae-eac9-46b2-b5fb-b36585723722-istio-data\") pod \"9a4c04ae-eac9-46b2-b5fb-b36585723722\" (UID: \"9a4c04ae-eac9-46b2-b5fb-b36585723722\") " Nov 01 16:31:30.800701 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:30.800597 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a4c04ae-eac9-46b2-b5fb-b36585723722-kube-api-access-wpfs5\" (UniqueName: \"kubernetes.io/projected/9a4c04ae-eac9-46b2-b5fb-b36585723722-kube-api-access-wpfs5\") pod \"9a4c04ae-eac9-46b2-b5fb-b36585723722\" (UID: \"9a4c04ae-eac9-46b2-b5fb-b36585723722\") " Nov 01 16:31:30.800701 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:30.800642 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a4c04ae-eac9-46b2-b5fb-b36585723722-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a4c04ae-eac9-46b2-b5fb-b36585723722-istio-podinfo\") pod \"9a4c04ae-eac9-46b2-b5fb-b36585723722\" (UID: \"9a4c04ae-eac9-46b2-b5fb-b36585723722\") " Nov 01 16:31:30.800701 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:30.800684 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a4c04ae-eac9-46b2-b5fb-b36585723722-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a4c04ae-eac9-46b2-b5fb-b36585723722-istiod-ca-cert\") pod \"9a4c04ae-eac9-46b2-b5fb-b36585723722\" (UID: \"9a4c04ae-eac9-46b2-b5fb-b36585723722\") " Nov 01 16:31:30.801534 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:30.800739 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a4c04ae-eac9-46b2-b5fb-b36585723722-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a4c04ae-eac9-46b2-b5fb-b36585723722-workload-certs\") pod \"9a4c04ae-eac9-46b2-b5fb-b36585723722\" (UID: \"9a4c04ae-eac9-46b2-b5fb-b36585723722\") " Nov 01 16:31:30.801534 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:30.800871 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a4c04ae-eac9-46b2-b5fb-b36585723722-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a4c04ae-eac9-46b2-b5fb-b36585723722-workload-socket\") pod \"9a4c04ae-eac9-46b2-b5fb-b36585723722\" (UID: \"9a4c04ae-eac9-46b2-b5fb-b36585723722\") " Nov 01 16:31:30.801534 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:30.800996 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a4c04ae-eac9-46b2-b5fb-b36585723722-istio-data" pod "9a4c04ae-eac9-46b2-b5fb-b36585723722" (UID: "9a4c04ae-eac9-46b2-b5fb-b36585723722"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:31:30.801534 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:30.801507 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a4c04ae-eac9-46b2-b5fb-b36585723722-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:30.801816 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:30.801749 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a4c04ae-eac9-46b2-b5fb-b36585723722-workload-socket" pod "9a4c04ae-eac9-46b2-b5fb-b36585723722" (UID: "9a4c04ae-eac9-46b2-b5fb-b36585723722"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:31:30.802309 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:30.802274 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a4c04ae-eac9-46b2-b5fb-b36585723722-workload-certs" pod "9a4c04ae-eac9-46b2-b5fb-b36585723722" (UID: "9a4c04ae-eac9-46b2-b5fb-b36585723722"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:31:30.802905 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:30.802885 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a4c04ae-eac9-46b2-b5fb-b36585723722-istiod-ca-cert" pod "9a4c04ae-eac9-46b2-b5fb-b36585723722" (UID: "9a4c04ae-eac9-46b2-b5fb-b36585723722"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:31:30.803407 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:30.803385 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a4c04ae-eac9-46b2-b5fb-b36585723722-credential-socket" pod "9a4c04ae-eac9-46b2-b5fb-b36585723722" (UID: "9a4c04ae-eac9-46b2-b5fb-b36585723722"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:31:30.811355 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:30.811330 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a4c04ae-eac9-46b2-b5fb-b36585723722-istio-podinfo" pod "9a4c04ae-eac9-46b2-b5fb-b36585723722" (UID: "9a4c04ae-eac9-46b2-b5fb-b36585723722"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:31:30.811716 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:30.811688 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a4c04ae-eac9-46b2-b5fb-b36585723722-istio-token" pod "9a4c04ae-eac9-46b2-b5fb-b36585723722" (UID: "9a4c04ae-eac9-46b2-b5fb-b36585723722"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:31:30.811820 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:30.811784 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a4c04ae-eac9-46b2-b5fb-b36585723722-kube-api-access-wpfs5" pod "9a4c04ae-eac9-46b2-b5fb-b36585723722" (UID: "9a4c04ae-eac9-46b2-b5fb-b36585723722"). InnerVolumeSpecName "kube-api-access-wpfs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:31:30.812051 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:30.812003 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a4c04ae-eac9-46b2-b5fb-b36585723722-istio-envoy" pod "9a4c04ae-eac9-46b2-b5fb-b36585723722" (UID: "9a4c04ae-eac9-46b2-b5fb-b36585723722"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:31:30.837708 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:30.837683 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" event={"ID":"9a4c04ae-eac9-46b2-b5fb-b36585723722","Type":"ContainerDied","Data":"658d6ed056d3a59ace04f1b865ede340f35cf6a281e0637a358929aa22ab9038"} Nov 01 16:31:30.837799 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:30.837742 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn" event={"ID":"9a4c04ae-eac9-46b2-b5fb-b36585723722","Type":"ContainerDied","Data":"1d49f97fd7b2941c4fc39580ddd3b4a7cb95abae3194fec1693cd9e9a8da1877"} Nov 01 16:31:30.837799 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:30.837765 4363 scope.go:122] "RemoveContainer" containerID="658d6ed056d3a59ace04f1b865ede340f35cf6a281e0637a358929aa22ab9038" Nov 01 16:31:30.848216 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:30.848178 4363 scope.go:122] "RemoveContainer" containerID="02be29b57b17261b3ec54fb84642abd5c33b125e81098185a462576cbf9b4c9b" Nov 01 16:31:30.857635 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:30.857612 4363 scope.go:122] "RemoveContainer" containerID="658d6ed056d3a59ace04f1b865ede340f35cf6a281e0637a358929aa22ab9038" Nov 01 16:31:30.857922 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:31:30.857899 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"658d6ed056d3a59ace04f1b865ede340f35cf6a281e0637a358929aa22ab9038\": container with ID starting with 658d6ed056d3a59ace04f1b865ede340f35cf6a281e0637a358929aa22ab9038 not found: ID does not exist" containerID="658d6ed056d3a59ace04f1b865ede340f35cf6a281e0637a358929aa22ab9038" Nov 01 16:31:30.857997 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:30.857929 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"658d6ed056d3a59ace04f1b865ede340f35cf6a281e0637a358929aa22ab9038"} err="failed to get container status \"658d6ed056d3a59ace04f1b865ede340f35cf6a281e0637a358929aa22ab9038\": rpc error: code = NotFound desc = could not find container \"658d6ed056d3a59ace04f1b865ede340f35cf6a281e0637a358929aa22ab9038\": container with ID starting with 658d6ed056d3a59ace04f1b865ede340f35cf6a281e0637a358929aa22ab9038 not found: ID does not exist" Nov 01 16:31:30.857997 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:30.857946 4363 scope.go:122] "RemoveContainer" containerID="02be29b57b17261b3ec54fb84642abd5c33b125e81098185a462576cbf9b4c9b" Nov 01 16:31:30.858230 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:31:30.858208 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02be29b57b17261b3ec54fb84642abd5c33b125e81098185a462576cbf9b4c9b\": container with ID starting with 02be29b57b17261b3ec54fb84642abd5c33b125e81098185a462576cbf9b4c9b not found: ID does not exist" containerID="02be29b57b17261b3ec54fb84642abd5c33b125e81098185a462576cbf9b4c9b" Nov 01 16:31:30.858286 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:30.858236 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02be29b57b17261b3ec54fb84642abd5c33b125e81098185a462576cbf9b4c9b"} err="failed to get container status \"02be29b57b17261b3ec54fb84642abd5c33b125e81098185a462576cbf9b4c9b\": rpc error: code = NotFound desc = could not find container \"02be29b57b17261b3ec54fb84642abd5c33b125e81098185a462576cbf9b4c9b\": container with ID starting with 02be29b57b17261b3ec54fb84642abd5c33b125e81098185a462576cbf9b4c9b not found: ID does not exist" Nov 01 16:31:30.904034 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:30.903198 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a4c04ae-eac9-46b2-b5fb-b36585723722-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:30.904034 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:30.903229 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a4c04ae-eac9-46b2-b5fb-b36585723722-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:30.904034 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:30.903244 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a4c04ae-eac9-46b2-b5fb-b36585723722-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:30.904034 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:30.903260 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a4c04ae-eac9-46b2-b5fb-b36585723722-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:30.904034 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:30.903275 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpfs5\" (UniqueName: \"kubernetes.io/projected/9a4c04ae-eac9-46b2-b5fb-b36585723722-kube-api-access-wpfs5\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:30.904034 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:30.903292 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a4c04ae-eac9-46b2-b5fb-b36585723722-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:30.904034 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:30.903308 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a4c04ae-eac9-46b2-b5fb-b36585723722-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:30.904034 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:30.903323 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a4c04ae-eac9-46b2-b5fb-b36585723722-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:31.119916 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:31.119872 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn"] Nov 01 16:31:31.123953 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:31.123922 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-e1e630-openshift-default-6576d59bc5-ggbwn"] Nov 01 16:31:32.571934 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:32.571892 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr"] Nov 01 16:31:32.572450 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:32.572138 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" podUID="c8ea572a-0005-426a-9017-de3331cbcbd7" containerName="istio-proxy" containerID="cri-o://cd303c7ee68e6eb367148892c128c853bc6edeca3af7cdc2bde9fc3e9ac6fe81" gracePeriod=30 Nov 01 16:31:32.824094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:32.820634 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a4c04ae-eac9-46b2-b5fb-b36585723722" path="/var/lib/kubelet/pods/9a4c04ae-eac9-46b2-b5fb-b36585723722/volumes" Nov 01 16:31:33.045081 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:33.045044 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6b6cfc-kmm9h"] Nov 01 16:31:33.045775 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:33.045758 4363 cpu_manager.go:414] "containerMap: removing container" podUID="d60400a5-9792-41ec-90c2-f1e76bee301b" containerName="payload-processing" Nov 01 16:31:33.045820 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:33.045776 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d60400a5-9792-41ec-90c2-f1e76bee301b" containerName="payload-processing" Nov 01 16:31:33.045820 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:33.045803 4363 cpu_manager.go:414] "containerMap: removing container" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" Nov 01 16:31:33.045820 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:33.045808 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" Nov 01 16:31:33.045924 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:33.045821 4363 cpu_manager.go:414] "containerMap: removing container" podUID="9a4c04ae-eac9-46b2-b5fb-b36585723722" containerName="istio-proxy" Nov 01 16:31:33.045924 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:33.045827 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a4c04ae-eac9-46b2-b5fb-b36585723722" containerName="istio-proxy" Nov 01 16:31:33.045924 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:33.045835 4363 cpu_manager.go:414] "containerMap: removing container" podUID="d4b55ba5-b1af-492d-abe2-1baa85947260" containerName="istio-proxy" Nov 01 16:31:33.045924 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:33.045840 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4b55ba5-b1af-492d-abe2-1baa85947260" containerName="istio-proxy" Nov 01 16:31:33.045924 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:33.045848 4363 cpu_manager.go:414] "containerMap: removing container" podUID="9a4c04ae-eac9-46b2-b5fb-b36585723722" containerName="istio-proxy" Nov 01 16:31:33.045924 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:33.045852 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a4c04ae-eac9-46b2-b5fb-b36585723722" containerName="istio-proxy" Nov 01 16:31:33.046133 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:33.045951 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="0049d3d4-16f8-4b77-87fa-dacf0921ad20" containerName="main" Nov 01 16:31:33.046133 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:33.045960 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a4c04ae-eac9-46b2-b5fb-b36585723722" containerName="istio-proxy" Nov 01 16:31:33.046133 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:33.045966 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4b55ba5-b1af-492d-abe2-1baa85947260" containerName="istio-proxy" Nov 01 16:31:33.046133 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:33.045974 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a4c04ae-eac9-46b2-b5fb-b36585723722" containerName="istio-proxy" Nov 01 16:31:33.046133 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:33.045984 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="d60400a5-9792-41ec-90c2-f1e76bee301b" containerName="payload-processing" Nov 01 16:31:33.050696 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:33.050676 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6b6cfc-kmm9h" Nov 01 16:31:33.057648 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:33.057622 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6b6cfc-kmm9h"] Nov 01 16:31:33.096705 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:33.096673 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:31:33.096993 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:33.096974 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:31:33.164187 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:33.164155 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32192888-1077-4e65-ab4d-673ca10210a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6b6cfc-kmm9h\" (UID: \"32192888-1077-4e65-ab4d-673ca10210a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6b6cfc-kmm9h" Nov 01 16:31:33.164370 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:33.164208 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32192888-1077-4e65-ab4d-673ca10210a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6b6cfc-kmm9h\" (UID: \"32192888-1077-4e65-ab4d-673ca10210a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6b6cfc-kmm9h" Nov 01 16:31:33.164370 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:33.164327 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrvwg\" (UniqueName: \"kubernetes.io/projected/32192888-1077-4e65-ab4d-673ca10210a7-kube-api-access-xrvwg\") pod \"maas-api-revoke-keys-e2e-ipp-b-6b6cfc-kmm9h\" (UID: \"32192888-1077-4e65-ab4d-673ca10210a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6b6cfc-kmm9h" Nov 01 16:31:33.266264 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:33.266236 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32192888-1077-4e65-ab4d-673ca10210a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6b6cfc-kmm9h\" (UID: \"32192888-1077-4e65-ab4d-673ca10210a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6b6cfc-kmm9h" Nov 01 16:31:33.266460 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:33.266284 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32192888-1077-4e65-ab4d-673ca10210a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6b6cfc-kmm9h\" (UID: \"32192888-1077-4e65-ab4d-673ca10210a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6b6cfc-kmm9h" Nov 01 16:31:33.266460 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:33.266358 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrvwg\" (UniqueName: \"kubernetes.io/projected/32192888-1077-4e65-ab4d-673ca10210a7-kube-api-access-xrvwg\") pod \"maas-api-revoke-keys-e2e-ipp-b-6b6cfc-kmm9h\" (UID: \"32192888-1077-4e65-ab4d-673ca10210a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6b6cfc-kmm9h" Nov 01 16:31:33.267310 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:33.267250 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32192888-1077-4e65-ab4d-673ca10210a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6b6cfc-kmm9h\" (UID: \"32192888-1077-4e65-ab4d-673ca10210a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6b6cfc-kmm9h" Nov 01 16:31:33.268992 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:33.268965 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32192888-1077-4e65-ab4d-673ca10210a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6b6cfc-kmm9h\" (UID: \"32192888-1077-4e65-ab4d-673ca10210a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6b6cfc-kmm9h" Nov 01 16:31:33.274689 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:33.274667 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrvwg\" (UniqueName: \"kubernetes.io/projected/32192888-1077-4e65-ab4d-673ca10210a7-kube-api-access-xrvwg\") pod \"maas-api-revoke-keys-e2e-ipp-b-6b6cfc-kmm9h\" (UID: \"32192888-1077-4e65-ab4d-673ca10210a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6b6cfc-kmm9h" Nov 01 16:31:33.370454 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:33.370381 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6b6cfc-kmm9h" Nov 01 16:31:33.656462 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:33.656430 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6b6cfc-kmm9h"] Nov 01 16:31:33.658928 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:31:33.658885 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32192888_1077_4e65_ab4d_673ca10210a7.slice/crio-25fef69f8974354dd9f6772996b0e871ff30af7dc847062a6876771c39693def WatchSource:0}: Error finding container 25fef69f8974354dd9f6772996b0e871ff30af7dc847062a6876771c39693def: Status 404 returned error can't find the container with id 25fef69f8974354dd9f6772996b0e871ff30af7dc847062a6876771c39693def Nov 01 16:31:33.853849 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:33.853816 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6b6cfc-kmm9h" event={"ID":"32192888-1077-4e65-ab4d-673ca10210a7","Type":"ContainerStarted","Data":"25fef69f8974354dd9f6772996b0e871ff30af7dc847062a6876771c39693def"} Nov 01 16:31:34.862473 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:34.862438 4363 generic.go:356] "Generic (PLEG): container finished" podID="32192888-1077-4e65-ab4d-673ca10210a7" containerID="b3593911a0b22644699a9af8e38d6f655dc34a0915679e91809926a5a6cc4102" exitCode=0 Nov 01 16:31:34.862864 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:34.862482 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6b6cfc-kmm9h" event={"ID":"32192888-1077-4e65-ab4d-673ca10210a7","Type":"ContainerDied","Data":"b3593911a0b22644699a9af8e38d6f655dc34a0915679e91809926a5a6cc4102"} Nov 01 16:31:36.066458 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:36.066430 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6b6cfc-kmm9h" Nov 01 16:31:36.111956 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:36.110788 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32192888-1077-4e65-ab4d-673ca10210a7-kube-api-access-xrvwg\" (UniqueName: \"kubernetes.io/projected/32192888-1077-4e65-ab4d-673ca10210a7-kube-api-access-xrvwg\") pod \"32192888-1077-4e65-ab4d-673ca10210a7\" (UID: \"32192888-1077-4e65-ab4d-673ca10210a7\") " Nov 01 16:31:36.111956 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:36.110975 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32192888-1077-4e65-ab4d-673ca10210a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32192888-1077-4e65-ab4d-673ca10210a7-maas-api-openshift-ca\") pod \"32192888-1077-4e65-ab4d-673ca10210a7\" (UID: \"32192888-1077-4e65-ab4d-673ca10210a7\") " Nov 01 16:31:36.111956 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:36.111286 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32192888-1077-4e65-ab4d-673ca10210a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32192888-1077-4e65-ab4d-673ca10210a7-maas-api-xks-ca\") pod \"32192888-1077-4e65-ab4d-673ca10210a7\" (UID: \"32192888-1077-4e65-ab4d-673ca10210a7\") " Nov 01 16:31:36.111956 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:36.111412 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32192888-1077-4e65-ab4d-673ca10210a7-maas-api-openshift-ca" pod "32192888-1077-4e65-ab4d-673ca10210a7" (UID: "32192888-1077-4e65-ab4d-673ca10210a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:31:36.111956 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:36.111895 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32192888-1077-4e65-ab4d-673ca10210a7-maas-api-openshift-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:36.122081 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:36.121974 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32192888-1077-4e65-ab4d-673ca10210a7-maas-api-xks-ca" pod "32192888-1077-4e65-ab4d-673ca10210a7" (UID: "32192888-1077-4e65-ab4d-673ca10210a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:31:36.122366 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:36.122327 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32192888-1077-4e65-ab4d-673ca10210a7-kube-api-access-xrvwg" pod "32192888-1077-4e65-ab4d-673ca10210a7" (UID: "32192888-1077-4e65-ab4d-673ca10210a7"). InnerVolumeSpecName "kube-api-access-xrvwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:31:36.213313 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:36.213274 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrvwg\" (UniqueName: \"kubernetes.io/projected/32192888-1077-4e65-ab4d-673ca10210a7-kube-api-access-xrvwg\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:36.213313 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:36.213309 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32192888-1077-4e65-ab4d-673ca10210a7-maas-api-xks-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:36.878513 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:36.878487 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6b6cfc-kmm9h" Nov 01 16:31:36.878734 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:36.878483 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6b6cfc-kmm9h" event={"ID":"32192888-1077-4e65-ab4d-673ca10210a7","Type":"ContainerDied","Data":"25fef69f8974354dd9f6772996b0e871ff30af7dc847062a6876771c39693def"} Nov 01 16:31:36.878734 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:36.878612 4363 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25fef69f8974354dd9f6772996b0e871ff30af7dc847062a6876771c39693def" Nov 01 16:31:37.860791 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:37.860765 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" Nov 01 16:31:37.914959 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:37.914918 4363 generic.go:356] "Generic (PLEG): container finished" podID="c8ea572a-0005-426a-9017-de3331cbcbd7" containerID="cd303c7ee68e6eb367148892c128c853bc6edeca3af7cdc2bde9fc3e9ac6fe81" exitCode=0 Nov 01 16:31:37.915178 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:37.914970 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" event={"ID":"c8ea572a-0005-426a-9017-de3331cbcbd7","Type":"ContainerDied","Data":"cd303c7ee68e6eb367148892c128c853bc6edeca3af7cdc2bde9fc3e9ac6fe81"} Nov 01 16:31:37.915178 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:37.915039 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" event={"ID":"c8ea572a-0005-426a-9017-de3331cbcbd7","Type":"ContainerDied","Data":"50fd26e93f89366e49b99a0680857a333869dba52669a2787bff7d34bbd0b062"} Nov 01 16:31:37.915178 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:37.915070 4363 scope.go:122] "RemoveContainer" containerID="cd303c7ee68e6eb367148892c128c853bc6edeca3af7cdc2bde9fc3e9ac6fe81" Nov 01 16:31:37.915382 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:37.915231 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr" Nov 01 16:31:37.932269 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:37.932165 4363 scope.go:122] "RemoveContainer" containerID="4f2463064534c9e5b955d6af551a388d04d3253128b18b6059ce214a47d6f1e6" Nov 01 16:31:37.942577 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:37.942555 4363 scope.go:122] "RemoveContainer" containerID="cd303c7ee68e6eb367148892c128c853bc6edeca3af7cdc2bde9fc3e9ac6fe81" Nov 01 16:31:37.942887 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:31:37.942859 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd303c7ee68e6eb367148892c128c853bc6edeca3af7cdc2bde9fc3e9ac6fe81\": container with ID starting with cd303c7ee68e6eb367148892c128c853bc6edeca3af7cdc2bde9fc3e9ac6fe81 not found: ID does not exist" containerID="cd303c7ee68e6eb367148892c128c853bc6edeca3af7cdc2bde9fc3e9ac6fe81" Nov 01 16:31:37.942931 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:37.942896 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd303c7ee68e6eb367148892c128c853bc6edeca3af7cdc2bde9fc3e9ac6fe81"} err="failed to get container status \"cd303c7ee68e6eb367148892c128c853bc6edeca3af7cdc2bde9fc3e9ac6fe81\": rpc error: code = NotFound desc = could not find container \"cd303c7ee68e6eb367148892c128c853bc6edeca3af7cdc2bde9fc3e9ac6fe81\": container with ID starting with cd303c7ee68e6eb367148892c128c853bc6edeca3af7cdc2bde9fc3e9ac6fe81 not found: ID does not exist" Nov 01 16:31:37.942931 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:37.942915 4363 scope.go:122] "RemoveContainer" containerID="4f2463064534c9e5b955d6af551a388d04d3253128b18b6059ce214a47d6f1e6" Nov 01 16:31:37.943192 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:31:37.943174 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f2463064534c9e5b955d6af551a388d04d3253128b18b6059ce214a47d6f1e6\": container with ID starting with 4f2463064534c9e5b955d6af551a388d04d3253128b18b6059ce214a47d6f1e6 not found: ID does not exist" containerID="4f2463064534c9e5b955d6af551a388d04d3253128b18b6059ce214a47d6f1e6" Nov 01 16:31:37.943243 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:37.943199 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f2463064534c9e5b955d6af551a388d04d3253128b18b6059ce214a47d6f1e6"} err="failed to get container status \"4f2463064534c9e5b955d6af551a388d04d3253128b18b6059ce214a47d6f1e6\": rpc error: code = NotFound desc = could not find container \"4f2463064534c9e5b955d6af551a388d04d3253128b18b6059ce214a47d6f1e6\": container with ID starting with 4f2463064534c9e5b955d6af551a388d04d3253128b18b6059ce214a47d6f1e6 not found: ID does not exist" Nov 01 16:31:37.954212 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:37.953944 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8ea572a-0005-426a-9017-de3331cbcbd7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ea572a-0005-426a-9017-de3331cbcbd7-workload-socket\") pod \"c8ea572a-0005-426a-9017-de3331cbcbd7\" (UID: \"c8ea572a-0005-426a-9017-de3331cbcbd7\") " Nov 01 16:31:37.954212 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:37.953984 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8ea572a-0005-426a-9017-de3331cbcbd7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8ea572a-0005-426a-9017-de3331cbcbd7-istio-podinfo\") pod \"c8ea572a-0005-426a-9017-de3331cbcbd7\" (UID: \"c8ea572a-0005-426a-9017-de3331cbcbd7\") " Nov 01 16:31:37.954212 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:37.954062 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8ea572a-0005-426a-9017-de3331cbcbd7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ea572a-0005-426a-9017-de3331cbcbd7-credential-socket\") pod \"c8ea572a-0005-426a-9017-de3331cbcbd7\" (UID: \"c8ea572a-0005-426a-9017-de3331cbcbd7\") " Nov 01 16:31:37.954212 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:37.954146 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8ea572a-0005-426a-9017-de3331cbcbd7-istio-token\" (UniqueName: \"kubernetes.io/projected/c8ea572a-0005-426a-9017-de3331cbcbd7-istio-token\") pod \"c8ea572a-0005-426a-9017-de3331cbcbd7\" (UID: \"c8ea572a-0005-426a-9017-de3331cbcbd7\") " Nov 01 16:31:37.954212 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:37.954199 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8ea572a-0005-426a-9017-de3331cbcbd7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8ea572a-0005-426a-9017-de3331cbcbd7-istio-envoy\") pod \"c8ea572a-0005-426a-9017-de3331cbcbd7\" (UID: \"c8ea572a-0005-426a-9017-de3331cbcbd7\") " Nov 01 16:31:37.954550 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:37.954241 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8ea572a-0005-426a-9017-de3331cbcbd7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8ea572a-0005-426a-9017-de3331cbcbd7-istio-data\") pod \"c8ea572a-0005-426a-9017-de3331cbcbd7\" (UID: \"c8ea572a-0005-426a-9017-de3331cbcbd7\") " Nov 01 16:31:37.954550 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:37.954312 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8ea572a-0005-426a-9017-de3331cbcbd7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8ea572a-0005-426a-9017-de3331cbcbd7-istiod-ca-cert\") pod \"c8ea572a-0005-426a-9017-de3331cbcbd7\" (UID: \"c8ea572a-0005-426a-9017-de3331cbcbd7\") " Nov 01 16:31:37.954550 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:37.954333 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8ea572a-0005-426a-9017-de3331cbcbd7-kube-api-access-6k2sk\" (UniqueName: \"kubernetes.io/projected/c8ea572a-0005-426a-9017-de3331cbcbd7-kube-api-access-6k2sk\") pod \"c8ea572a-0005-426a-9017-de3331cbcbd7\" (UID: \"c8ea572a-0005-426a-9017-de3331cbcbd7\") " Nov 01 16:31:37.954550 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:37.954329 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ea572a-0005-426a-9017-de3331cbcbd7-workload-socket" pod "c8ea572a-0005-426a-9017-de3331cbcbd7" (UID: "c8ea572a-0005-426a-9017-de3331cbcbd7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:31:37.954550 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:37.954379 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8ea572a-0005-426a-9017-de3331cbcbd7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8ea572a-0005-426a-9017-de3331cbcbd7-workload-certs\") pod \"c8ea572a-0005-426a-9017-de3331cbcbd7\" (UID: \"c8ea572a-0005-426a-9017-de3331cbcbd7\") " Nov 01 16:31:37.954822 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:37.954552 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ea572a-0005-426a-9017-de3331cbcbd7-istio-data" pod "c8ea572a-0005-426a-9017-de3331cbcbd7" (UID: "c8ea572a-0005-426a-9017-de3331cbcbd7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:31:37.954884 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:37.954836 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8ea572a-0005-426a-9017-de3331cbcbd7-istiod-ca-cert" pod "c8ea572a-0005-426a-9017-de3331cbcbd7" (UID: "c8ea572a-0005-426a-9017-de3331cbcbd7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:31:37.954884 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:37.954846 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ea572a-0005-426a-9017-de3331cbcbd7-workload-certs" pod "c8ea572a-0005-426a-9017-de3331cbcbd7" (UID: "c8ea572a-0005-426a-9017-de3331cbcbd7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:31:37.955156 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:37.955102 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ea572a-0005-426a-9017-de3331cbcbd7-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:37.955156 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:37.955006 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ea572a-0005-426a-9017-de3331cbcbd7-credential-socket" pod "c8ea572a-0005-426a-9017-de3331cbcbd7" (UID: "c8ea572a-0005-426a-9017-de3331cbcbd7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:31:37.955156 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:37.955128 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8ea572a-0005-426a-9017-de3331cbcbd7-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:37.955314 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:37.955179 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8ea572a-0005-426a-9017-de3331cbcbd7-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:37.955314 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:37.955200 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8ea572a-0005-426a-9017-de3331cbcbd7-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:37.956370 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:37.956347 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8ea572a-0005-426a-9017-de3331cbcbd7-istio-podinfo" pod "c8ea572a-0005-426a-9017-de3331cbcbd7" (UID: "c8ea572a-0005-426a-9017-de3331cbcbd7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:31:37.956774 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:37.956751 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8ea572a-0005-426a-9017-de3331cbcbd7-istio-token" pod "c8ea572a-0005-426a-9017-de3331cbcbd7" (UID: "c8ea572a-0005-426a-9017-de3331cbcbd7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:31:37.956940 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:37.956916 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8ea572a-0005-426a-9017-de3331cbcbd7-kube-api-access-6k2sk" pod "c8ea572a-0005-426a-9017-de3331cbcbd7" (UID: "c8ea572a-0005-426a-9017-de3331cbcbd7"). InnerVolumeSpecName "kube-api-access-6k2sk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:31:37.956994 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:37.956920 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ea572a-0005-426a-9017-de3331cbcbd7-istio-envoy" pod "c8ea572a-0005-426a-9017-de3331cbcbd7" (UID: "c8ea572a-0005-426a-9017-de3331cbcbd7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:31:38.056336 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:38.056311 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8ea572a-0005-426a-9017-de3331cbcbd7-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:38.056336 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:38.056335 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8ea572a-0005-426a-9017-de3331cbcbd7-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:38.056514 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:38.056347 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k2sk\" (UniqueName: \"kubernetes.io/projected/c8ea572a-0005-426a-9017-de3331cbcbd7-kube-api-access-6k2sk\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:38.056514 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:38.056357 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8ea572a-0005-426a-9017-de3331cbcbd7-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:38.056514 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:38.056366 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ea572a-0005-426a-9017-de3331cbcbd7-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:38.242268 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:38.242116 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr"] Nov 01 16:31:38.246116 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:38.246087 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3fd73f-openshift-default-5d5658744f-zdxwr"] Nov 01 16:31:38.800750 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:38.800713 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8ea572a-0005-426a-9017-de3331cbcbd7" path="/var/lib/kubelet/pods/c8ea572a-0005-426a-9017-de3331cbcbd7/volumes" Nov 01 16:31:45.828957 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:45.828914 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-cd5bc765f-9f5tw"] Nov 01 16:31:45.829657 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:45.829639 4363 cpu_manager.go:414] "containerMap: removing container" podUID="c8ea572a-0005-426a-9017-de3331cbcbd7" containerName="istio-proxy" Nov 01 16:31:45.829657 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:45.829656 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8ea572a-0005-426a-9017-de3331cbcbd7" containerName="istio-proxy" Nov 01 16:31:45.829800 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:45.829700 4363 cpu_manager.go:414] "containerMap: removing container" podUID="32192888-1077-4e65-ab4d-673ca10210a7" containerName="revoke-keys" Nov 01 16:31:45.829800 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:45.829705 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32192888-1077-4e65-ab4d-673ca10210a7" containerName="revoke-keys" Nov 01 16:31:45.829925 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:45.829800 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="32192888-1077-4e65-ab4d-673ca10210a7" containerName="revoke-keys" Nov 01 16:31:45.829925 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:45.829810 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8ea572a-0005-426a-9017-de3331cbcbd7" containerName="istio-proxy" Nov 01 16:31:45.829925 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:45.829819 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8ea572a-0005-426a-9017-de3331cbcbd7" containerName="istio-proxy" Nov 01 16:31:45.835891 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:45.835864 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-cd5bc765f-9f5tw" Nov 01 16:31:45.840407 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:45.840372 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b8aa932-6702-4c56-acaf-2bebb3d5074f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-aa6303-cd5bc765f-9f5tw\" (UID: \"0b8aa932-6702-4c56-acaf-2bebb3d5074f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-cd5bc765f-9f5tw" Nov 01 16:31:45.840594 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:45.840421 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbw68\" (UniqueName: \"kubernetes.io/projected/0b8aa932-6702-4c56-acaf-2bebb3d5074f-kube-api-access-mbw68\") pod \"payload-pre-processing-e2e-ipp-b-aa6303-cd5bc765f-9f5tw\" (UID: \"0b8aa932-6702-4c56-acaf-2bebb3d5074f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-cd5bc765f-9f5tw" Nov 01 16:31:45.841617 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:45.841102 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-cd5bc765f-9f5tw"] Nov 01 16:31:45.841732 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:45.841675 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-aa6303\"" Nov 01 16:31:45.858913 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:45.858887 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa6303-5bcf66496-8dckq"] Nov 01 16:31:45.859854 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:45.859828 4363 cpu_manager.go:414] "containerMap: removing container" podUID="c8ea572a-0005-426a-9017-de3331cbcbd7" containerName="istio-proxy" Nov 01 16:31:45.859962 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:45.859855 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8ea572a-0005-426a-9017-de3331cbcbd7" containerName="istio-proxy" Nov 01 16:31:45.866065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:45.864612 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa6303-5bcf66496-8dckq" Nov 01 16:31:45.875705 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:45.875675 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa6303-5bcf66496-8dckq"] Nov 01 16:31:45.875958 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:45.875941 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-aa6303-dockercfg-75f9h\"" Nov 01 16:31:45.983069 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:45.964413 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b8aa932-6702-4c56-acaf-2bebb3d5074f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-aa6303-cd5bc765f-9f5tw\" (UID: \"0b8aa932-6702-4c56-acaf-2bebb3d5074f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-cd5bc765f-9f5tw" Nov 01 16:31:45.983069 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:45.964467 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbw68\" (UniqueName: \"kubernetes.io/projected/0b8aa932-6702-4c56-acaf-2bebb3d5074f-kube-api-access-mbw68\") pod \"payload-pre-processing-e2e-ipp-b-aa6303-cd5bc765f-9f5tw\" (UID: \"0b8aa932-6702-4c56-acaf-2bebb3d5074f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-cd5bc765f-9f5tw" Nov 01 16:31:45.983069 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:45.964652 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc9aad52-e26b-4e45-9241-ad27218724d7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-aa6303-5bcf66496-8dckq\" (UID: \"bc9aad52-e26b-4e45-9241-ad27218724d7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa6303-5bcf66496-8dckq" Nov 01 16:31:45.983069 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:45.964683 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n4kg\" (UniqueName: \"kubernetes.io/projected/bc9aad52-e26b-4e45-9241-ad27218724d7-kube-api-access-7n4kg\") pod \"payload-processing-e2e-ipp-b-aa6303-5bcf66496-8dckq\" (UID: \"bc9aad52-e26b-4e45-9241-ad27218724d7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa6303-5bcf66496-8dckq" Nov 01 16:31:45.983069 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:45.965843 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b8aa932-6702-4c56-acaf-2bebb3d5074f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-aa6303-cd5bc765f-9f5tw\" (UID: \"0b8aa932-6702-4c56-acaf-2bebb3d5074f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-cd5bc765f-9f5tw" Nov 01 16:31:46.106097 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:46.066378 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc9aad52-e26b-4e45-9241-ad27218724d7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-aa6303-5bcf66496-8dckq\" (UID: \"bc9aad52-e26b-4e45-9241-ad27218724d7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa6303-5bcf66496-8dckq" Nov 01 16:31:46.106097 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:46.066421 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n4kg\" (UniqueName: \"kubernetes.io/projected/bc9aad52-e26b-4e45-9241-ad27218724d7-kube-api-access-7n4kg\") pod \"payload-processing-e2e-ipp-b-aa6303-5bcf66496-8dckq\" (UID: \"bc9aad52-e26b-4e45-9241-ad27218724d7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa6303-5bcf66496-8dckq" Nov 01 16:31:46.121048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:46.106869 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc9aad52-e26b-4e45-9241-ad27218724d7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-aa6303-5bcf66496-8dckq\" (UID: \"bc9aad52-e26b-4e45-9241-ad27218724d7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa6303-5bcf66496-8dckq" Nov 01 16:31:46.245038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:46.233054 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n4kg\" (UniqueName: \"kubernetes.io/projected/bc9aad52-e26b-4e45-9241-ad27218724d7-kube-api-access-7n4kg\") pod \"payload-processing-e2e-ipp-b-aa6303-5bcf66496-8dckq\" (UID: \"bc9aad52-e26b-4e45-9241-ad27218724d7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa6303-5bcf66496-8dckq" Nov 01 16:31:46.255819 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:46.255743 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbw68\" (UniqueName: \"kubernetes.io/projected/0b8aa932-6702-4c56-acaf-2bebb3d5074f-kube-api-access-mbw68\") pod \"payload-pre-processing-e2e-ipp-b-aa6303-cd5bc765f-9f5tw\" (UID: \"0b8aa932-6702-4c56-acaf-2bebb3d5074f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-cd5bc765f-9f5tw" Nov 01 16:31:46.458181 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:46.453550 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa6303-5bcf66496-8dckq" Nov 01 16:31:46.465223 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:46.464257 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-cd5bc765f-9f5tw" Nov 01 16:31:46.866996 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:31:46.866970 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b8aa932_6702_4c56_acaf_2bebb3d5074f.slice/crio-0260126030b8ab1baf44bcf85db597a46f2ecb1e741d7cfebadcb5d80a134c06 WatchSource:0}: Error finding container 0260126030b8ab1baf44bcf85db597a46f2ecb1e741d7cfebadcb5d80a134c06: Status 404 returned error can't find the container with id 0260126030b8ab1baf44bcf85db597a46f2ecb1e741d7cfebadcb5d80a134c06 Nov 01 16:31:46.868158 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:46.868137 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-cd5bc765f-9f5tw"] Nov 01 16:31:46.869686 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:31:46.869659 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc9aad52_e26b_4e45_9241_ad27218724d7.slice/crio-c8fdc4b2fe7e9e20ee972fa719f5496b6e118163dc68bf042fd04466b3f1ffbb WatchSource:0}: Error finding container c8fdc4b2fe7e9e20ee972fa719f5496b6e118163dc68bf042fd04466b3f1ffbb: Status 404 returned error can't find the container with id c8fdc4b2fe7e9e20ee972fa719f5496b6e118163dc68bf042fd04466b3f1ffbb Nov 01 16:31:46.873279 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:46.873253 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa6303-5bcf66496-8dckq"] Nov 01 16:31:47.305105 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:47.304126 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa6303-5bcf66496-8dckq" event={"ID":"bc9aad52-e26b-4e45-9241-ad27218724d7","Type":"ContainerStarted","Data":"0522aa7f097281170b331bd8ca4019e7a50fc9a3f04307f17e6757c943dd4a30"} Nov 01 16:31:47.305105 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:47.304192 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa6303-5bcf66496-8dckq" event={"ID":"bc9aad52-e26b-4e45-9241-ad27218724d7","Type":"ContainerStarted","Data":"c8fdc4b2fe7e9e20ee972fa719f5496b6e118163dc68bf042fd04466b3f1ffbb"} Nov 01 16:31:47.309047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:47.305352 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa6303-5bcf66496-8dckq" Nov 01 16:31:47.313033 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:47.312064 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-cd5bc765f-9f5tw" event={"ID":"0b8aa932-6702-4c56-acaf-2bebb3d5074f","Type":"ContainerStarted","Data":"e76892e8178156c4582f94d49a3f13ce62b305b082826857f36c6da96a2ba152"} Nov 01 16:31:47.313033 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:47.312115 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-cd5bc765f-9f5tw" event={"ID":"0b8aa932-6702-4c56-acaf-2bebb3d5074f","Type":"ContainerStarted","Data":"0260126030b8ab1baf44bcf85db597a46f2ecb1e741d7cfebadcb5d80a134c06"} Nov 01 16:31:47.313033 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:47.312876 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-cd5bc765f-9f5tw" Nov 01 16:31:47.351802 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:47.351527 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa6303-5bcf66496-8dckq" podStartSLOduration=2.351515141 podStartE2EDuration="2.351515141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:31:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:31:47.348480647 +0000 UTC m=+2640.680218300" watchObservedRunningTime="2026-11-01 16:31:47.351515141 +0000 UTC m=+2640.683252792" Nov 01 16:31:48.319202 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:31:48.319163 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:31:53.377643 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:53.377607 4363 generic.go:356] "Generic (PLEG): container finished" podID="7c97faba-9728-4c13-b99d-19fa20b7e85b" containerID="be5a939df28fce2552f2b99c55ede881a6d3f0bdbfd0604f45cfd72a5ee4eaa3" exitCode=0 Nov 01 16:31:53.378141 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:53.377688 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" event={"ID":"7c97faba-9728-4c13-b99d-19fa20b7e85b","Type":"ContainerDied","Data":"be5a939df28fce2552f2b99c55ede881a6d3f0bdbfd0604f45cfd72a5ee4eaa3"} Nov 01 16:31:53.378210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:53.378191 4363 scope.go:122] "RemoveContainer" containerID="be5a939df28fce2552f2b99c55ede881a6d3f0bdbfd0604f45cfd72a5ee4eaa3" Nov 01 16:31:53.380268 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:53.380237 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:31:53.380368 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:53.380328 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:31:53.380407 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:53.380375 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:31:53.403008 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:53.402955 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-cd5bc765f-9f5tw" podStartSLOduration=8.402938881 podStartE2EDuration="8.402938881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:31:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:31:47.407962703 +0000 UTC m=+2640.739700354" watchObservedRunningTime="2026-11-01 16:31:53.402938881 +0000 UTC m=+2646.734676532" Nov 01 16:31:54.394386 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:54.394346 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" event={"ID":"7c97faba-9728-4c13-b99d-19fa20b7e85b","Type":"ContainerStarted","Data":"5a9a235ac5ca4fc6a6d4ce8b7b3d1a1a240b519dded5c4dd5de5e264c334ee9e"} Nov 01 16:31:54.470084 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:54.470053 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" Nov 01 16:31:54.470263 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:54.470245 4363 patch_prober.go:29] interesting pod/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" start-of-body= Nov 01 16:31:54.470314 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:54.470298 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" podUID="7c97faba-9728-4c13-b99d-19fa20b7e85b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" Nov 01 16:31:55.365161 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:55.365126 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6b6cfc-5c65865b75-87k56"] Nov 01 16:31:55.365403 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:55.365375 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6b6cfc-5c65865b75-87k56" podUID="3bae9e9c-d6f5-4b49-be31-c793e0b293ef" containerName="maas-api" containerID="cri-o://55adff084fad0057b633b6bafbf4af965c421fe749797d5d5e514c2e56f26596" gracePeriod=30 Nov 01 16:31:55.468677 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:55.445338 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-779fd7cf75-tnrpx"] Nov 01 16:31:55.468677 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:55.445560 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-779fd7cf75-tnrpx" podUID="cee70f14-4206-4670-b671-82a568db2cc8" containerName="payload-pre-processing" containerID="cri-o://9756db03eed437c23547e5d5bd8ddb6bf4171b17251a853537108b2cdf679d26" gracePeriod=30 Nov 01 16:31:55.469955 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:55.469927 4363 patch_prober.go:29] interesting pod/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" start-of-body= Nov 01 16:31:55.470102 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:55.469987 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" podUID="7c97faba-9728-4c13-b99d-19fa20b7e85b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" Nov 01 16:31:55.674730 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:55.674653 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6b6cfc-kmm9h"] Nov 01 16:31:55.720084 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:55.717800 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6b6cfc-kmm9h"] Nov 01 16:31:56.152484 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:56.152450 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-779fd7cf75-tnrpx" Nov 01 16:31:56.159606 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:56.159579 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6b6cfc-5c65865b75-87k56" Nov 01 16:31:56.328091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:56.315003 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bae9e9c-d6f5-4b49-be31-c793e0b293ef-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bae9e9c-d6f5-4b49-be31-c793e0b293ef-maas-api-tls\") pod \"3bae9e9c-d6f5-4b49-be31-c793e0b293ef\" (UID: \"3bae9e9c-d6f5-4b49-be31-c793e0b293ef\") " Nov 01 16:31:56.328091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:56.315162 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bae9e9c-d6f5-4b49-be31-c793e0b293ef-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bae9e9c-d6f5-4b49-be31-c793e0b293ef-metrics-certs\") pod \"3bae9e9c-d6f5-4b49-be31-c793e0b293ef\" (UID: \"3bae9e9c-d6f5-4b49-be31-c793e0b293ef\") " Nov 01 16:31:56.328091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:56.315249 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bae9e9c-d6f5-4b49-be31-c793e0b293ef-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bae9e9c-d6f5-4b49-be31-c793e0b293ef-openshift-service-ca\") pod \"3bae9e9c-d6f5-4b49-be31-c793e0b293ef\" (UID: \"3bae9e9c-d6f5-4b49-be31-c793e0b293ef\") " Nov 01 16:31:56.328091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:56.315373 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cee70f14-4206-4670-b671-82a568db2cc8-kube-api-access-85hkz\" (UniqueName: \"kubernetes.io/projected/cee70f14-4206-4670-b671-82a568db2cc8-kube-api-access-85hkz\") pod \"cee70f14-4206-4670-b671-82a568db2cc8\" (UID: \"cee70f14-4206-4670-b671-82a568db2cc8\") " Nov 01 16:31:56.328091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:56.315458 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bae9e9c-d6f5-4b49-be31-c793e0b293ef-kube-api-access-5vtf9\" (UniqueName: \"kubernetes.io/projected/3bae9e9c-d6f5-4b49-be31-c793e0b293ef-kube-api-access-5vtf9\") pod \"3bae9e9c-d6f5-4b49-be31-c793e0b293ef\" (UID: \"3bae9e9c-d6f5-4b49-be31-c793e0b293ef\") " Nov 01 16:31:56.328091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:56.315537 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cee70f14-4206-4670-b671-82a568db2cc8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cee70f14-4206-4670-b671-82a568db2cc8-plugins-config-volume\") pod \"cee70f14-4206-4670-b671-82a568db2cc8\" (UID: \"cee70f14-4206-4670-b671-82a568db2cc8\") " Nov 01 16:31:56.328091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:56.324785 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bae9e9c-d6f5-4b49-be31-c793e0b293ef-openshift-service-ca" pod "3bae9e9c-d6f5-4b49-be31-c793e0b293ef" (UID: "3bae9e9c-d6f5-4b49-be31-c793e0b293ef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:31:56.328091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:56.326466 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cee70f14-4206-4670-b671-82a568db2cc8-plugins-config-volume" pod "cee70f14-4206-4670-b671-82a568db2cc8" (UID: "cee70f14-4206-4670-b671-82a568db2cc8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:31:56.328091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:56.327666 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cee70f14-4206-4670-b671-82a568db2cc8-kube-api-access-85hkz" pod "cee70f14-4206-4670-b671-82a568db2cc8" (UID: "cee70f14-4206-4670-b671-82a568db2cc8"). InnerVolumeSpecName "kube-api-access-85hkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:31:56.328091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:56.328033 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bae9e9c-d6f5-4b49-be31-c793e0b293ef-maas-api-tls" pod "3bae9e9c-d6f5-4b49-be31-c793e0b293ef" (UID: "3bae9e9c-d6f5-4b49-be31-c793e0b293ef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:31:56.329988 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:56.329110 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bae9e9c-d6f5-4b49-be31-c793e0b293ef-metrics-certs" pod "3bae9e9c-d6f5-4b49-be31-c793e0b293ef" (UID: "3bae9e9c-d6f5-4b49-be31-c793e0b293ef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:31:56.330458 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:56.330432 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bae9e9c-d6f5-4b49-be31-c793e0b293ef-kube-api-access-5vtf9" pod "3bae9e9c-d6f5-4b49-be31-c793e0b293ef" (UID: "3bae9e9c-d6f5-4b49-be31-c793e0b293ef"). InnerVolumeSpecName "kube-api-access-5vtf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:31:56.418633 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:56.418592 4363 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bae9e9c-d6f5-4b49-be31-c793e0b293ef-metrics-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:56.418633 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:56.418629 4363 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bae9e9c-d6f5-4b49-be31-c793e0b293ef-openshift-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:56.418843 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:56.418646 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85hkz\" (UniqueName: \"kubernetes.io/projected/cee70f14-4206-4670-b671-82a568db2cc8-kube-api-access-85hkz\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:56.418843 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:56.418662 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vtf9\" (UniqueName: \"kubernetes.io/projected/3bae9e9c-d6f5-4b49-be31-c793e0b293ef-kube-api-access-5vtf9\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:56.418843 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:56.418675 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cee70f14-4206-4670-b671-82a568db2cc8-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:56.418843 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:56.418691 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bae9e9c-d6f5-4b49-be31-c793e0b293ef-maas-api-tls\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:56.444979 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:56.444949 4363 generic.go:356] "Generic (PLEG): container finished" podID="3bae9e9c-d6f5-4b49-be31-c793e0b293ef" containerID="55adff084fad0057b633b6bafbf4af965c421fe749797d5d5e514c2e56f26596" exitCode=0 Nov 01 16:31:56.445151 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:56.445002 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6b6cfc-5c65865b75-87k56" Nov 01 16:31:56.445151 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:56.445044 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6b6cfc-5c65865b75-87k56" event={"ID":"3bae9e9c-d6f5-4b49-be31-c793e0b293ef","Type":"ContainerDied","Data":"55adff084fad0057b633b6bafbf4af965c421fe749797d5d5e514c2e56f26596"} Nov 01 16:31:56.445151 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:56.445098 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6b6cfc-5c65865b75-87k56" event={"ID":"3bae9e9c-d6f5-4b49-be31-c793e0b293ef","Type":"ContainerDied","Data":"844ca0723e25930379ab6d1d1e03d6a52e58c0efaa86f8e4b2d693abf1c81748"} Nov 01 16:31:56.445151 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:56.445123 4363 scope.go:122] "RemoveContainer" containerID="55adff084fad0057b633b6bafbf4af965c421fe749797d5d5e514c2e56f26596" Nov 01 16:31:56.446525 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:56.446450 4363 generic.go:356] "Generic (PLEG): container finished" podID="cee70f14-4206-4670-b671-82a568db2cc8" containerID="9756db03eed437c23547e5d5bd8ddb6bf4171b17251a853537108b2cdf679d26" exitCode=0 Nov 01 16:31:56.446525 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:56.446477 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-779fd7cf75-tnrpx" event={"ID":"cee70f14-4206-4670-b671-82a568db2cc8","Type":"ContainerDied","Data":"9756db03eed437c23547e5d5bd8ddb6bf4171b17251a853537108b2cdf679d26"} Nov 01 16:31:56.446525 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:56.446498 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-779fd7cf75-tnrpx" Nov 01 16:31:56.446688 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:56.446531 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-779fd7cf75-tnrpx" event={"ID":"cee70f14-4206-4670-b671-82a568db2cc8","Type":"ContainerDied","Data":"dad16c4960a03c533653658469d28d0720581b0b94d02483321ae7d4265482fc"} Nov 01 16:31:56.458493 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:56.458474 4363 scope.go:122] "RemoveContainer" containerID="55adff084fad0057b633b6bafbf4af965c421fe749797d5d5e514c2e56f26596" Nov 01 16:31:56.458817 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:31:56.458792 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55adff084fad0057b633b6bafbf4af965c421fe749797d5d5e514c2e56f26596\": container with ID starting with 55adff084fad0057b633b6bafbf4af965c421fe749797d5d5e514c2e56f26596 not found: ID does not exist" containerID="55adff084fad0057b633b6bafbf4af965c421fe749797d5d5e514c2e56f26596" Nov 01 16:31:56.458938 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:56.458828 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55adff084fad0057b633b6bafbf4af965c421fe749797d5d5e514c2e56f26596"} err="failed to get container status \"55adff084fad0057b633b6bafbf4af965c421fe749797d5d5e514c2e56f26596\": rpc error: code = NotFound desc = could not find container \"55adff084fad0057b633b6bafbf4af965c421fe749797d5d5e514c2e56f26596\": container with ID starting with 55adff084fad0057b633b6bafbf4af965c421fe749797d5d5e514c2e56f26596 not found: ID does not exist" Nov 01 16:31:56.458938 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:56.458850 4363 scope.go:122] "RemoveContainer" containerID="9756db03eed437c23547e5d5bd8ddb6bf4171b17251a853537108b2cdf679d26" Nov 01 16:31:56.468469 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:56.468454 4363 scope.go:122] "RemoveContainer" containerID="9756db03eed437c23547e5d5bd8ddb6bf4171b17251a853537108b2cdf679d26" Nov 01 16:31:56.468751 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:31:56.468734 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9756db03eed437c23547e5d5bd8ddb6bf4171b17251a853537108b2cdf679d26\": container with ID starting with 9756db03eed437c23547e5d5bd8ddb6bf4171b17251a853537108b2cdf679d26 not found: ID does not exist" containerID="9756db03eed437c23547e5d5bd8ddb6bf4171b17251a853537108b2cdf679d26" Nov 01 16:31:56.469100 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:56.468762 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9756db03eed437c23547e5d5bd8ddb6bf4171b17251a853537108b2cdf679d26"} err="failed to get container status \"9756db03eed437c23547e5d5bd8ddb6bf4171b17251a853537108b2cdf679d26\": rpc error: code = NotFound desc = could not find container \"9756db03eed437c23547e5d5bd8ddb6bf4171b17251a853537108b2cdf679d26\": container with ID starting with 9756db03eed437c23547e5d5bd8ddb6bf4171b17251a853537108b2cdf679d26 not found: ID does not exist" Nov 01 16:31:56.470212 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:56.470189 4363 patch_prober.go:29] interesting pod/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" start-of-body= Nov 01 16:31:56.470314 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:56.470238 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" podUID="7c97faba-9728-4c13-b99d-19fa20b7e85b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" Nov 01 16:31:56.474108 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:56.474085 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6b6cfc-5c65865b75-87k56"] Nov 01 16:31:56.511484 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:56.511439 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6b6cfc-5c65865b75-87k56"] Nov 01 16:31:56.515633 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:56.515604 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-779fd7cf75-tnrpx"] Nov 01 16:31:56.538220 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:56.538193 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6b6cfc-779fd7cf75-tnrpx"] Nov 01 16:31:56.816774 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:56.816737 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32192888-1077-4e65-ab4d-673ca10210a7" path="/var/lib/kubelet/pods/32192888-1077-4e65-ab4d-673ca10210a7/volumes" Nov 01 16:31:56.817418 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:56.817395 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bae9e9c-d6f5-4b49-be31-c793e0b293ef" path="/var/lib/kubelet/pods/3bae9e9c-d6f5-4b49-be31-c793e0b293ef/volumes" Nov 01 16:31:56.817816 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:56.817797 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cee70f14-4206-4670-b671-82a568db2cc8" path="/var/lib/kubelet/pods/cee70f14-4206-4670-b671-82a568db2cc8/volumes" Nov 01 16:31:57.470396 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:57.470365 4363 patch_prober.go:29] interesting pod/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" start-of-body= Nov 01 16:31:57.470793 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:57.470416 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" podUID="7c97faba-9728-4c13-b99d-19fa20b7e85b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" Nov 01 16:31:58.700057 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:58.697121 4363 patch_prober.go:29] interesting pod/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" start-of-body= Nov 01 16:31:58.700057 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:58.697194 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" podUID="7c97faba-9728-4c13-b99d-19fa20b7e85b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" Nov 01 16:31:58.750041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:58.746475 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w"] Nov 01 16:31:58.750041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:58.746749 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" podUID="3b5725f6-5b76-4300-9886-6a8c2556e6ae" containerName="istio-proxy" containerID="cri-o://67be541f031c228b189f751595a8c551d35203f778f2bad6a1856122069c90e5" gracePeriod=30 Nov 01 16:31:59.016092 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:59.015257 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-43fac8-mr4wd"] Nov 01 16:31:59.019038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:59.016385 4363 cpu_manager.go:414] "containerMap: removing container" podUID="3bae9e9c-d6f5-4b49-be31-c793e0b293ef" containerName="maas-api" Nov 01 16:31:59.019038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:59.016409 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bae9e9c-d6f5-4b49-be31-c793e0b293ef" containerName="maas-api" Nov 01 16:31:59.019038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:59.016436 4363 cpu_manager.go:414] "containerMap: removing container" podUID="cee70f14-4206-4670-b671-82a568db2cc8" containerName="payload-pre-processing" Nov 01 16:31:59.019038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:59.016444 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cee70f14-4206-4670-b671-82a568db2cc8" containerName="payload-pre-processing" Nov 01 16:31:59.019038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:59.016640 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bae9e9c-d6f5-4b49-be31-c793e0b293ef" containerName="maas-api" Nov 01 16:31:59.019038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:59.016657 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="cee70f14-4206-4670-b671-82a568db2cc8" containerName="payload-pre-processing" Nov 01 16:31:59.022745 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:59.022727 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-43fac8-mr4wd" Nov 01 16:31:59.024059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:59.024032 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-43fac8-mr4wd"] Nov 01 16:31:59.026519 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:59.026466 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:31:59.026638 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:59.026580 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:31:59.119402 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:59.119364 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/857faaf7-ea2e-4d29-8b08-32e0d5b083e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-43fac8-mr4wd\" (UID: \"857faaf7-ea2e-4d29-8b08-32e0d5b083e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-43fac8-mr4wd" Nov 01 16:31:59.119647 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:59.119622 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/857faaf7-ea2e-4d29-8b08-32e0d5b083e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-43fac8-mr4wd\" (UID: \"857faaf7-ea2e-4d29-8b08-32e0d5b083e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-43fac8-mr4wd" Nov 01 16:31:59.119841 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:59.119822 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnnlj\" (UniqueName: \"kubernetes.io/projected/857faaf7-ea2e-4d29-8b08-32e0d5b083e8-kube-api-access-nnnlj\") pod \"maas-api-revoke-keys-e2e-worker-w6-43fac8-mr4wd\" (UID: \"857faaf7-ea2e-4d29-8b08-32e0d5b083e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-43fac8-mr4wd" Nov 01 16:31:59.221318 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:59.221276 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/857faaf7-ea2e-4d29-8b08-32e0d5b083e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-43fac8-mr4wd\" (UID: \"857faaf7-ea2e-4d29-8b08-32e0d5b083e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-43fac8-mr4wd" Nov 01 16:31:59.221520 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:59.221457 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnnlj\" (UniqueName: \"kubernetes.io/projected/857faaf7-ea2e-4d29-8b08-32e0d5b083e8-kube-api-access-nnnlj\") pod \"maas-api-revoke-keys-e2e-worker-w6-43fac8-mr4wd\" (UID: \"857faaf7-ea2e-4d29-8b08-32e0d5b083e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-43fac8-mr4wd" Nov 01 16:31:59.221588 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:59.221535 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/857faaf7-ea2e-4d29-8b08-32e0d5b083e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-43fac8-mr4wd\" (UID: \"857faaf7-ea2e-4d29-8b08-32e0d5b083e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-43fac8-mr4wd" Nov 01 16:31:59.222235 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:59.222208 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/857faaf7-ea2e-4d29-8b08-32e0d5b083e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-43fac8-mr4wd\" (UID: \"857faaf7-ea2e-4d29-8b08-32e0d5b083e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-43fac8-mr4wd" Nov 01 16:31:59.223996 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:59.223969 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/857faaf7-ea2e-4d29-8b08-32e0d5b083e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-43fac8-mr4wd\" (UID: \"857faaf7-ea2e-4d29-8b08-32e0d5b083e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-43fac8-mr4wd" Nov 01 16:31:59.229333 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:59.229310 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnnlj\" (UniqueName: \"kubernetes.io/projected/857faaf7-ea2e-4d29-8b08-32e0d5b083e8-kube-api-access-nnnlj\") pod \"maas-api-revoke-keys-e2e-worker-w6-43fac8-mr4wd\" (UID: \"857faaf7-ea2e-4d29-8b08-32e0d5b083e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-43fac8-mr4wd" Nov 01 16:31:59.334917 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:59.334830 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-cd5bc765f-9f5tw" Nov 01 16:31:59.335133 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:59.335010 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa6303-5bcf66496-8dckq" Nov 01 16:31:59.344445 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:59.344425 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-43fac8-mr4wd" Nov 01 16:31:59.432348 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:59.432244 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" Nov 01 16:31:59.469553 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:59.469521 4363 patch_prober.go:29] interesting pod/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" start-of-body= Nov 01 16:31:59.469758 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:59.469579 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" podUID="7c97faba-9728-4c13-b99d-19fa20b7e85b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" Nov 01 16:31:59.741263 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:31:59.741228 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod857faaf7_ea2e_4d29_8b08_32e0d5b083e8.slice/crio-8e512efda52c5629f626cba689577b6b536f0d2807b0f86bafca70ec6fabd5fe WatchSource:0}: Error finding container 8e512efda52c5629f626cba689577b6b536f0d2807b0f86bafca70ec6fabd5fe: Status 404 returned error can't find the container with id 8e512efda52c5629f626cba689577b6b536f0d2807b0f86bafca70ec6fabd5fe Nov 01 16:31:59.741590 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:31:59.741454 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-43fac8-mr4wd"] Nov 01 16:32:00.474742 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:00.474712 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" Nov 01 16:32:00.475723 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:00.475699 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" Nov 01 16:32:00.742410 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:00.742338 4363 generic.go:356] "Generic (PLEG): container finished" podID="857faaf7-ea2e-4d29-8b08-32e0d5b083e8" containerID="55f86b87d59b58a53b74749d7d0aaca3981acd6c125b9a17d6f01c0f08813daa" exitCode=0 Nov 01 16:32:00.742764 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:00.742427 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-43fac8-mr4wd" event={"ID":"857faaf7-ea2e-4d29-8b08-32e0d5b083e8","Type":"ContainerDied","Data":"55f86b87d59b58a53b74749d7d0aaca3981acd6c125b9a17d6f01c0f08813daa"} Nov 01 16:32:00.742764 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:00.742476 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-43fac8-mr4wd" event={"ID":"857faaf7-ea2e-4d29-8b08-32e0d5b083e8","Type":"ContainerStarted","Data":"8e512efda52c5629f626cba689577b6b536f0d2807b0f86bafca70ec6fabd5fe"} Nov 01 16:32:01.954788 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:01.954762 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-43fac8-mr4wd" Nov 01 16:32:02.003231 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:02.003193 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/857faaf7-ea2e-4d29-8b08-32e0d5b083e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/857faaf7-ea2e-4d29-8b08-32e0d5b083e8-maas-api-openshift-ca\") pod \"857faaf7-ea2e-4d29-8b08-32e0d5b083e8\" (UID: \"857faaf7-ea2e-4d29-8b08-32e0d5b083e8\") " Nov 01 16:32:02.003462 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:02.003442 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/857faaf7-ea2e-4d29-8b08-32e0d5b083e8-kube-api-access-nnnlj\" (UniqueName: \"kubernetes.io/projected/857faaf7-ea2e-4d29-8b08-32e0d5b083e8-kube-api-access-nnnlj\") pod \"857faaf7-ea2e-4d29-8b08-32e0d5b083e8\" (UID: \"857faaf7-ea2e-4d29-8b08-32e0d5b083e8\") " Nov 01 16:32:02.003631 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:02.003611 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/857faaf7-ea2e-4d29-8b08-32e0d5b083e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/857faaf7-ea2e-4d29-8b08-32e0d5b083e8-maas-api-xks-ca\") pod \"857faaf7-ea2e-4d29-8b08-32e0d5b083e8\" (UID: \"857faaf7-ea2e-4d29-8b08-32e0d5b083e8\") " Nov 01 16:32:02.004330 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:02.003611 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/857faaf7-ea2e-4d29-8b08-32e0d5b083e8-maas-api-openshift-ca" pod "857faaf7-ea2e-4d29-8b08-32e0d5b083e8" (UID: "857faaf7-ea2e-4d29-8b08-32e0d5b083e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:32:02.004330 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:02.004296 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/857faaf7-ea2e-4d29-8b08-32e0d5b083e8-maas-api-openshift-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:02.005837 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:02.005807 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/857faaf7-ea2e-4d29-8b08-32e0d5b083e8-kube-api-access-nnnlj" pod "857faaf7-ea2e-4d29-8b08-32e0d5b083e8" (UID: "857faaf7-ea2e-4d29-8b08-32e0d5b083e8"). InnerVolumeSpecName "kube-api-access-nnnlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:02.006278 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:02.006242 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/857faaf7-ea2e-4d29-8b08-32e0d5b083e8-maas-api-xks-ca" pod "857faaf7-ea2e-4d29-8b08-32e0d5b083e8" (UID: "857faaf7-ea2e-4d29-8b08-32e0d5b083e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:32:02.106283 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:02.106246 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnnlj\" (UniqueName: \"kubernetes.io/projected/857faaf7-ea2e-4d29-8b08-32e0d5b083e8-kube-api-access-nnnlj\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:02.106283 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:02.106278 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/857faaf7-ea2e-4d29-8b08-32e0d5b083e8-maas-api-xks-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:02.790306 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:02.790275 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-43fac8-mr4wd" Nov 01 16:32:02.801398 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:02.801367 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-43fac8-mr4wd" event={"ID":"857faaf7-ea2e-4d29-8b08-32e0d5b083e8","Type":"ContainerDied","Data":"8e512efda52c5629f626cba689577b6b536f0d2807b0f86bafca70ec6fabd5fe"} Nov 01 16:32:02.801573 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:02.801424 4363 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e512efda52c5629f626cba689577b6b536f0d2807b0f86bafca70ec6fabd5fe" Nov 01 16:32:03.165151 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:03.165114 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4"] Nov 01 16:32:03.165691 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:03.165378 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" podUID="69113cc0-84e0-4de6-8078-aa79ad5cad66" containerName="istio-proxy" containerID="cri-o://2d98c1ea18ed165181d0193fc3ef20f590882d103df04962c57309157bc17ef7" gracePeriod=30 Nov 01 16:32:03.708887 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:03.708849 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-028c46-rnvcs"] Nov 01 16:32:03.709605 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:03.709587 4363 cpu_manager.go:414] "containerMap: removing container" podUID="857faaf7-ea2e-4d29-8b08-32e0d5b083e8" containerName="revoke-keys" Nov 01 16:32:03.709605 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:03.709604 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="857faaf7-ea2e-4d29-8b08-32e0d5b083e8" containerName="revoke-keys" Nov 01 16:32:03.709790 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:03.709741 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="857faaf7-ea2e-4d29-8b08-32e0d5b083e8" containerName="revoke-keys" Nov 01 16:32:03.713586 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:03.713563 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-028c46-rnvcs" Nov 01 16:32:03.722544 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:03.722518 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-028c46-rnvcs"] Nov 01 16:32:03.737722 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:03.737694 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500072d0-7098-41b9-8498-1804ad8307a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-028c46-rnvcs\" (UID: \"500072d0-7098-41b9-8498-1804ad8307a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-028c46-rnvcs" Nov 01 16:32:03.737871 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:03.737842 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500072d0-7098-41b9-8498-1804ad8307a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-028c46-rnvcs\" (UID: \"500072d0-7098-41b9-8498-1804ad8307a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-028c46-rnvcs" Nov 01 16:32:03.737999 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:03.737979 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8llzw\" (UniqueName: \"kubernetes.io/projected/500072d0-7098-41b9-8498-1804ad8307a1-kube-api-access-8llzw\") pod \"maas-api-revoke-keys-e2e-ipp-a-028c46-rnvcs\" (UID: \"500072d0-7098-41b9-8498-1804ad8307a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-028c46-rnvcs" Nov 01 16:32:03.778111 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:03.778082 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:32:03.778111 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:03.778107 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:32:03.807586 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:03.807552 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql"] Nov 01 16:32:03.839209 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:03.839166 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500072d0-7098-41b9-8498-1804ad8307a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-028c46-rnvcs\" (UID: \"500072d0-7098-41b9-8498-1804ad8307a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-028c46-rnvcs" Nov 01 16:32:03.839400 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:03.839381 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500072d0-7098-41b9-8498-1804ad8307a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-028c46-rnvcs\" (UID: \"500072d0-7098-41b9-8498-1804ad8307a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-028c46-rnvcs" Nov 01 16:32:03.839704 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:03.839684 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8llzw\" (UniqueName: \"kubernetes.io/projected/500072d0-7098-41b9-8498-1804ad8307a1-kube-api-access-8llzw\") pod \"maas-api-revoke-keys-e2e-ipp-a-028c46-rnvcs\" (UID: \"500072d0-7098-41b9-8498-1804ad8307a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-028c46-rnvcs" Nov 01 16:32:03.839942 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:03.839914 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500072d0-7098-41b9-8498-1804ad8307a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-028c46-rnvcs\" (UID: \"500072d0-7098-41b9-8498-1804ad8307a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-028c46-rnvcs" Nov 01 16:32:03.841966 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:03.841944 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500072d0-7098-41b9-8498-1804ad8307a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-028c46-rnvcs\" (UID: \"500072d0-7098-41b9-8498-1804ad8307a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-028c46-rnvcs" Nov 01 16:32:03.852059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:03.852035 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8llzw\" (UniqueName: \"kubernetes.io/projected/500072d0-7098-41b9-8498-1804ad8307a1-kube-api-access-8llzw\") pod \"maas-api-revoke-keys-e2e-ipp-a-028c46-rnvcs\" (UID: \"500072d0-7098-41b9-8498-1804ad8307a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-028c46-rnvcs" Nov 01 16:32:04.039175 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.039139 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-028c46-rnvcs" Nov 01 16:32:04.116046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.115259 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" Nov 01 16:32:04.160918 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.159039 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b5725f6-5b76-4300-9886-6a8c2556e6ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b5725f6-5b76-4300-9886-6a8c2556e6ae-istio-data\") pod \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\" (UID: \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\") " Nov 01 16:32:04.160918 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.159201 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b5725f6-5b76-4300-9886-6a8c2556e6ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b5725f6-5b76-4300-9886-6a8c2556e6ae-istio-envoy\") pod \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\" (UID: \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\") " Nov 01 16:32:04.160918 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.159303 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b5725f6-5b76-4300-9886-6a8c2556e6ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b5725f6-5b76-4300-9886-6a8c2556e6ae-istio-podinfo\") pod \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\" (UID: \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\") " Nov 01 16:32:04.160918 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.159340 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b5725f6-5b76-4300-9886-6a8c2556e6ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b5725f6-5b76-4300-9886-6a8c2556e6ae-istiod-ca-cert\") pod \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\" (UID: \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\") " Nov 01 16:32:04.160918 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.159394 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b5725f6-5b76-4300-9886-6a8c2556e6ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b5725f6-5b76-4300-9886-6a8c2556e6ae-workload-certs\") pod \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\" (UID: \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\") " Nov 01 16:32:04.160918 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.159463 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b5725f6-5b76-4300-9886-6a8c2556e6ae-istio-token\" (UniqueName: \"kubernetes.io/projected/3b5725f6-5b76-4300-9886-6a8c2556e6ae-istio-token\") pod \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\" (UID: \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\") " Nov 01 16:32:04.160918 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.159549 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b5725f6-5b76-4300-9886-6a8c2556e6ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5725f6-5b76-4300-9886-6a8c2556e6ae-credential-socket\") pod \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\" (UID: \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\") " Nov 01 16:32:04.160918 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.159589 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b5725f6-5b76-4300-9886-6a8c2556e6ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5725f6-5b76-4300-9886-6a8c2556e6ae-workload-socket\") pod \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\" (UID: \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\") " Nov 01 16:32:04.160918 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.159651 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b5725f6-5b76-4300-9886-6a8c2556e6ae-kube-api-access-8pxgs\" (UniqueName: \"kubernetes.io/projected/3b5725f6-5b76-4300-9886-6a8c2556e6ae-kube-api-access-8pxgs\") pod \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\" (UID: \"3b5725f6-5b76-4300-9886-6a8c2556e6ae\") " Nov 01 16:32:04.168170 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.166143 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5725f6-5b76-4300-9886-6a8c2556e6ae-credential-socket" pod "3b5725f6-5b76-4300-9886-6a8c2556e6ae" (UID: "3b5725f6-5b76-4300-9886-6a8c2556e6ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:04.168170 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.166443 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5725f6-5b76-4300-9886-6a8c2556e6ae-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:04.168170 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.166671 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5725f6-5b76-4300-9886-6a8c2556e6ae-workload-certs" pod "3b5725f6-5b76-4300-9886-6a8c2556e6ae" (UID: "3b5725f6-5b76-4300-9886-6a8c2556e6ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:04.168170 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.166745 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5725f6-5b76-4300-9886-6a8c2556e6ae-istio-data" pod "3b5725f6-5b76-4300-9886-6a8c2556e6ae" (UID: "3b5725f6-5b76-4300-9886-6a8c2556e6ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:04.168170 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.166889 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b5725f6-5b76-4300-9886-6a8c2556e6ae-istiod-ca-cert" pod "3b5725f6-5b76-4300-9886-6a8c2556e6ae" (UID: "3b5725f6-5b76-4300-9886-6a8c2556e6ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:32:04.168170 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.167176 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5725f6-5b76-4300-9886-6a8c2556e6ae-workload-socket" pod "3b5725f6-5b76-4300-9886-6a8c2556e6ae" (UID: "3b5725f6-5b76-4300-9886-6a8c2556e6ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:04.175611 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.175585 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b5725f6-5b76-4300-9886-6a8c2556e6ae-istio-podinfo" pod "3b5725f6-5b76-4300-9886-6a8c2556e6ae" (UID: "3b5725f6-5b76-4300-9886-6a8c2556e6ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:32:04.183853 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.183821 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b5725f6-5b76-4300-9886-6a8c2556e6ae-istio-token" pod "3b5725f6-5b76-4300-9886-6a8c2556e6ae" (UID: "3b5725f6-5b76-4300-9886-6a8c2556e6ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:04.183974 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.183951 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5725f6-5b76-4300-9886-6a8c2556e6ae-istio-envoy" pod "3b5725f6-5b76-4300-9886-6a8c2556e6ae" (UID: "3b5725f6-5b76-4300-9886-6a8c2556e6ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:04.186937 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.186912 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b5725f6-5b76-4300-9886-6a8c2556e6ae-kube-api-access-8pxgs" pod "3b5725f6-5b76-4300-9886-6a8c2556e6ae" (UID: "3b5725f6-5b76-4300-9886-6a8c2556e6ae"). InnerVolumeSpecName "kube-api-access-8pxgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:04.267733 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.267654 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b5725f6-5b76-4300-9886-6a8c2556e6ae-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:04.267733 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.267686 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b5725f6-5b76-4300-9886-6a8c2556e6ae-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:04.267733 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.267696 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b5725f6-5b76-4300-9886-6a8c2556e6ae-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:04.267733 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.267706 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b5725f6-5b76-4300-9886-6a8c2556e6ae-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:04.267733 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.267716 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b5725f6-5b76-4300-9886-6a8c2556e6ae-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:04.267733 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.267726 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5725f6-5b76-4300-9886-6a8c2556e6ae-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:04.267733 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.267735 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pxgs\" (UniqueName: \"kubernetes.io/projected/3b5725f6-5b76-4300-9886-6a8c2556e6ae-kube-api-access-8pxgs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:04.268102 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.267746 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b5725f6-5b76-4300-9886-6a8c2556e6ae-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:04.377445 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.377416 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-028c46-rnvcs"] Nov 01 16:32:04.377668 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:32:04.377649 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod500072d0_7098_41b9_8498_1804ad8307a1.slice/crio-16df0b7c97b8e1e173d981a7350fe4c0da90cff431f04678c8a5dc2a3ef9cdb3 WatchSource:0}: Error finding container 16df0b7c97b8e1e173d981a7350fe4c0da90cff431f04678c8a5dc2a3ef9cdb3: Status 404 returned error can't find the container with id 16df0b7c97b8e1e173d981a7350fe4c0da90cff431f04678c8a5dc2a3ef9cdb3 Nov 01 16:32:04.807467 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.807430 4363 generic.go:356] "Generic (PLEG): container finished" podID="3b5725f6-5b76-4300-9886-6a8c2556e6ae" containerID="67be541f031c228b189f751595a8c551d35203f778f2bad6a1856122069c90e5" exitCode=0 Nov 01 16:32:04.807624 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.807493 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" Nov 01 16:32:04.807624 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.807511 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" event={"ID":"3b5725f6-5b76-4300-9886-6a8c2556e6ae","Type":"ContainerDied","Data":"67be541f031c228b189f751595a8c551d35203f778f2bad6a1856122069c90e5"} Nov 01 16:32:04.807624 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.807570 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w" event={"ID":"3b5725f6-5b76-4300-9886-6a8c2556e6ae","Type":"ContainerDied","Data":"b572eaf6ae4167fb1e67c25c052812cbee10afe38edd43ee181f85d903228602"} Nov 01 16:32:04.807624 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.807601 4363 scope.go:122] "RemoveContainer" containerID="67be541f031c228b189f751595a8c551d35203f778f2bad6a1856122069c90e5" Nov 01 16:32:04.812134 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.811751 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-028c46-rnvcs" event={"ID":"500072d0-7098-41b9-8498-1804ad8307a1","Type":"ContainerStarted","Data":"39ee3493f648383e85b8dc746b119e6cee1179fd6c64aa444d4dde884ab9da59"} Nov 01 16:32:04.812134 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.811797 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-028c46-rnvcs" event={"ID":"500072d0-7098-41b9-8498-1804ad8307a1","Type":"ContainerStarted","Data":"16df0b7c97b8e1e173d981a7350fe4c0da90cff431f04678c8a5dc2a3ef9cdb3"} Nov 01 16:32:04.820399 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.820381 4363 scope.go:122] "RemoveContainer" containerID="c4f9204e651009e25d691e291968fd2ba29f08a6db5f84e35abef7f4019f7a31" Nov 01 16:32:04.831824 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.831803 4363 scope.go:122] "RemoveContainer" containerID="67be541f031c228b189f751595a8c551d35203f778f2bad6a1856122069c90e5" Nov 01 16:32:04.832107 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:32:04.832085 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67be541f031c228b189f751595a8c551d35203f778f2bad6a1856122069c90e5\": container with ID starting with 67be541f031c228b189f751595a8c551d35203f778f2bad6a1856122069c90e5 not found: ID does not exist" containerID="67be541f031c228b189f751595a8c551d35203f778f2bad6a1856122069c90e5" Nov 01 16:32:04.832161 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.832117 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67be541f031c228b189f751595a8c551d35203f778f2bad6a1856122069c90e5"} err="failed to get container status \"67be541f031c228b189f751595a8c551d35203f778f2bad6a1856122069c90e5\": rpc error: code = NotFound desc = could not find container \"67be541f031c228b189f751595a8c551d35203f778f2bad6a1856122069c90e5\": container with ID starting with 67be541f031c228b189f751595a8c551d35203f778f2bad6a1856122069c90e5 not found: ID does not exist" Nov 01 16:32:04.832161 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.832152 4363 scope.go:122] "RemoveContainer" containerID="c4f9204e651009e25d691e291968fd2ba29f08a6db5f84e35abef7f4019f7a31" Nov 01 16:32:04.832469 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:32:04.832441 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4f9204e651009e25d691e291968fd2ba29f08a6db5f84e35abef7f4019f7a31\": container with ID starting with c4f9204e651009e25d691e291968fd2ba29f08a6db5f84e35abef7f4019f7a31 not found: ID does not exist" containerID="c4f9204e651009e25d691e291968fd2ba29f08a6db5f84e35abef7f4019f7a31" Nov 01 16:32:04.832568 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.832472 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4f9204e651009e25d691e291968fd2ba29f08a6db5f84e35abef7f4019f7a31"} err="failed to get container status \"c4f9204e651009e25d691e291968fd2ba29f08a6db5f84e35abef7f4019f7a31\": rpc error: code = NotFound desc = could not find container \"c4f9204e651009e25d691e291968fd2ba29f08a6db5f84e35abef7f4019f7a31\": container with ID starting with c4f9204e651009e25d691e291968fd2ba29f08a6db5f84e35abef7f4019f7a31 not found: ID does not exist" Nov 01 16:32:04.832868 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.832831 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-028c46-rnvcs" podStartSLOduration=1.832818195 podStartE2EDuration="1.832818195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:32:04.827204858 +0000 UTC m=+2658.158942515" watchObservedRunningTime="2026-11-01 16:32:04.832818195 +0000 UTC m=+2658.164555844" Nov 01 16:32:04.843314 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.843293 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w"] Nov 01 16:32:04.847690 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:04.847665 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-2b94b5-openshift-default-5b584bf549-h5b5w"] Nov 01 16:32:05.822722 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:05.822689 4363 generic.go:356] "Generic (PLEG): container finished" podID="500072d0-7098-41b9-8498-1804ad8307a1" containerID="39ee3493f648383e85b8dc746b119e6cee1179fd6c64aa444d4dde884ab9da59" exitCode=0 Nov 01 16:32:05.823170 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:05.822770 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-028c46-rnvcs" event={"ID":"500072d0-7098-41b9-8498-1804ad8307a1","Type":"ContainerDied","Data":"39ee3493f648383e85b8dc746b119e6cee1179fd6c64aa444d4dde884ab9da59"} Nov 01 16:32:06.800680 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:06.800642 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b5725f6-5b76-4300-9886-6a8c2556e6ae" path="/var/lib/kubelet/pods/3b5725f6-5b76-4300-9886-6a8c2556e6ae/volumes" Nov 01 16:32:07.001246 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:07.001217 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-028c46-rnvcs" Nov 01 16:32:07.143974 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:07.135656 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/500072d0-7098-41b9-8498-1804ad8307a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500072d0-7098-41b9-8498-1804ad8307a1-maas-api-xks-ca\") pod \"500072d0-7098-41b9-8498-1804ad8307a1\" (UID: \"500072d0-7098-41b9-8498-1804ad8307a1\") " Nov 01 16:32:07.143974 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:07.135817 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/500072d0-7098-41b9-8498-1804ad8307a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500072d0-7098-41b9-8498-1804ad8307a1-maas-api-openshift-ca\") pod \"500072d0-7098-41b9-8498-1804ad8307a1\" (UID: \"500072d0-7098-41b9-8498-1804ad8307a1\") " Nov 01 16:32:07.143974 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:07.135973 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/500072d0-7098-41b9-8498-1804ad8307a1-kube-api-access-8llzw\" (UniqueName: \"kubernetes.io/projected/500072d0-7098-41b9-8498-1804ad8307a1-kube-api-access-8llzw\") pod \"500072d0-7098-41b9-8498-1804ad8307a1\" (UID: \"500072d0-7098-41b9-8498-1804ad8307a1\") " Nov 01 16:32:07.156212 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:07.150470 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/500072d0-7098-41b9-8498-1804ad8307a1-maas-api-openshift-ca" pod "500072d0-7098-41b9-8498-1804ad8307a1" (UID: "500072d0-7098-41b9-8498-1804ad8307a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:32:07.164044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:07.162248 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/500072d0-7098-41b9-8498-1804ad8307a1-maas-api-xks-ca" pod "500072d0-7098-41b9-8498-1804ad8307a1" (UID: "500072d0-7098-41b9-8498-1804ad8307a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:32:07.164044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:07.163104 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/500072d0-7098-41b9-8498-1804ad8307a1-kube-api-access-8llzw" pod "500072d0-7098-41b9-8498-1804ad8307a1" (UID: "500072d0-7098-41b9-8498-1804ad8307a1"). InnerVolumeSpecName "kube-api-access-8llzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:07.237669 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:07.237631 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500072d0-7098-41b9-8498-1804ad8307a1-maas-api-xks-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:07.237669 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:07.237663 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500072d0-7098-41b9-8498-1804ad8307a1-maas-api-openshift-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:07.237669 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:07.237674 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8llzw\" (UniqueName: \"kubernetes.io/projected/500072d0-7098-41b9-8498-1804ad8307a1-kube-api-access-8llzw\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:07.849904 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:07.849872 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-028c46-rnvcs" Nov 01 16:32:07.850140 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:07.849909 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-028c46-rnvcs" event={"ID":"500072d0-7098-41b9-8498-1804ad8307a1","Type":"ContainerDied","Data":"16df0b7c97b8e1e173d981a7350fe4c0da90cff431f04678c8a5dc2a3ef9cdb3"} Nov 01 16:32:07.850140 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:07.849956 4363 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16df0b7c97b8e1e173d981a7350fe4c0da90cff431f04678c8a5dc2a3ef9cdb3" Nov 01 16:32:08.515894 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:08.515853 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" Nov 01 16:32:08.577126 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:08.577094 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69113cc0-84e0-4de6-8078-aa79ad5cad66-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69113cc0-84e0-4de6-8078-aa79ad5cad66-workload-certs\") pod \"69113cc0-84e0-4de6-8078-aa79ad5cad66\" (UID: \"69113cc0-84e0-4de6-8078-aa79ad5cad66\") " Nov 01 16:32:08.577325 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:08.577163 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69113cc0-84e0-4de6-8078-aa79ad5cad66-kube-api-access-7vr8n\" (UniqueName: \"kubernetes.io/projected/69113cc0-84e0-4de6-8078-aa79ad5cad66-kube-api-access-7vr8n\") pod \"69113cc0-84e0-4de6-8078-aa79ad5cad66\" (UID: \"69113cc0-84e0-4de6-8078-aa79ad5cad66\") " Nov 01 16:32:08.577325 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:08.577263 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69113cc0-84e0-4de6-8078-aa79ad5cad66-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69113cc0-84e0-4de6-8078-aa79ad5cad66-credential-socket\") pod \"69113cc0-84e0-4de6-8078-aa79ad5cad66\" (UID: \"69113cc0-84e0-4de6-8078-aa79ad5cad66\") " Nov 01 16:32:08.577463 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:08.577327 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69113cc0-84e0-4de6-8078-aa79ad5cad66-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69113cc0-84e0-4de6-8078-aa79ad5cad66-workload-socket\") pod \"69113cc0-84e0-4de6-8078-aa79ad5cad66\" (UID: \"69113cc0-84e0-4de6-8078-aa79ad5cad66\") " Nov 01 16:32:08.577463 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:08.577363 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69113cc0-84e0-4de6-8078-aa79ad5cad66-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69113cc0-84e0-4de6-8078-aa79ad5cad66-istio-podinfo\") pod \"69113cc0-84e0-4de6-8078-aa79ad5cad66\" (UID: \"69113cc0-84e0-4de6-8078-aa79ad5cad66\") " Nov 01 16:32:08.577463 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:08.577410 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69113cc0-84e0-4de6-8078-aa79ad5cad66-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69113cc0-84e0-4de6-8078-aa79ad5cad66-istiod-ca-cert\") pod \"69113cc0-84e0-4de6-8078-aa79ad5cad66\" (UID: \"69113cc0-84e0-4de6-8078-aa79ad5cad66\") " Nov 01 16:32:08.577463 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:08.577448 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69113cc0-84e0-4de6-8078-aa79ad5cad66-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69113cc0-84e0-4de6-8078-aa79ad5cad66-istio-data\") pod \"69113cc0-84e0-4de6-8078-aa79ad5cad66\" (UID: \"69113cc0-84e0-4de6-8078-aa79ad5cad66\") " Nov 01 16:32:08.577660 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:08.577460 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69113cc0-84e0-4de6-8078-aa79ad5cad66-workload-certs" pod "69113cc0-84e0-4de6-8078-aa79ad5cad66" (UID: "69113cc0-84e0-4de6-8078-aa79ad5cad66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:08.577660 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:08.577524 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69113cc0-84e0-4de6-8078-aa79ad5cad66-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69113cc0-84e0-4de6-8078-aa79ad5cad66-istio-envoy\") pod \"69113cc0-84e0-4de6-8078-aa79ad5cad66\" (UID: \"69113cc0-84e0-4de6-8078-aa79ad5cad66\") " Nov 01 16:32:08.577660 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:08.577557 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69113cc0-84e0-4de6-8078-aa79ad5cad66-credential-socket" pod "69113cc0-84e0-4de6-8078-aa79ad5cad66" (UID: "69113cc0-84e0-4de6-8078-aa79ad5cad66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:08.577828 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:08.577672 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69113cc0-84e0-4de6-8078-aa79ad5cad66-istio-token\" (UniqueName: \"kubernetes.io/projected/69113cc0-84e0-4de6-8078-aa79ad5cad66-istio-token\") pod \"69113cc0-84e0-4de6-8078-aa79ad5cad66\" (UID: \"69113cc0-84e0-4de6-8078-aa79ad5cad66\") " Nov 01 16:32:08.577828 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:08.577748 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69113cc0-84e0-4de6-8078-aa79ad5cad66-workload-socket" pod "69113cc0-84e0-4de6-8078-aa79ad5cad66" (UID: "69113cc0-84e0-4de6-8078-aa79ad5cad66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:08.577927 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:08.577909 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69113cc0-84e0-4de6-8078-aa79ad5cad66-istio-data" pod "69113cc0-84e0-4de6-8078-aa79ad5cad66" (UID: "69113cc0-84e0-4de6-8078-aa79ad5cad66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:08.578411 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:08.578385 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69113cc0-84e0-4de6-8078-aa79ad5cad66-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:08.578525 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:08.578414 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69113cc0-84e0-4de6-8078-aa79ad5cad66-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:08.578525 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:08.578430 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69113cc0-84e0-4de6-8078-aa79ad5cad66-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:08.578525 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:08.578443 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69113cc0-84e0-4de6-8078-aa79ad5cad66-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:08.578784 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:08.578764 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69113cc0-84e0-4de6-8078-aa79ad5cad66-istiod-ca-cert" pod "69113cc0-84e0-4de6-8078-aa79ad5cad66" (UID: "69113cc0-84e0-4de6-8078-aa79ad5cad66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:32:08.581268 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:08.581228 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69113cc0-84e0-4de6-8078-aa79ad5cad66-istio-podinfo" pod "69113cc0-84e0-4de6-8078-aa79ad5cad66" (UID: "69113cc0-84e0-4de6-8078-aa79ad5cad66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:32:08.581381 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:08.581357 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69113cc0-84e0-4de6-8078-aa79ad5cad66-kube-api-access-7vr8n" pod "69113cc0-84e0-4de6-8078-aa79ad5cad66" (UID: "69113cc0-84e0-4de6-8078-aa79ad5cad66"). InnerVolumeSpecName "kube-api-access-7vr8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:08.581541 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:08.581517 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69113cc0-84e0-4de6-8078-aa79ad5cad66-istio-envoy" pod "69113cc0-84e0-4de6-8078-aa79ad5cad66" (UID: "69113cc0-84e0-4de6-8078-aa79ad5cad66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:08.581692 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:08.581675 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69113cc0-84e0-4de6-8078-aa79ad5cad66-istio-token" pod "69113cc0-84e0-4de6-8078-aa79ad5cad66" (UID: "69113cc0-84e0-4de6-8078-aa79ad5cad66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:08.679980 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:08.679948 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vr8n\" (UniqueName: \"kubernetes.io/projected/69113cc0-84e0-4de6-8078-aa79ad5cad66-kube-api-access-7vr8n\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:08.679980 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:08.679979 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69113cc0-84e0-4de6-8078-aa79ad5cad66-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:08.680226 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:08.679989 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69113cc0-84e0-4de6-8078-aa79ad5cad66-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:08.680226 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:08.679998 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69113cc0-84e0-4de6-8078-aa79ad5cad66-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:08.680226 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:08.680007 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69113cc0-84e0-4de6-8078-aa79ad5cad66-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:08.859423 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:08.859386 4363 generic.go:356] "Generic (PLEG): container finished" podID="69113cc0-84e0-4de6-8078-aa79ad5cad66" containerID="2d98c1ea18ed165181d0193fc3ef20f590882d103df04962c57309157bc17ef7" exitCode=0 Nov 01 16:32:08.859594 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:08.859450 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" event={"ID":"69113cc0-84e0-4de6-8078-aa79ad5cad66","Type":"ContainerDied","Data":"2d98c1ea18ed165181d0193fc3ef20f590882d103df04962c57309157bc17ef7"} Nov 01 16:32:08.859594 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:08.859509 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" event={"ID":"69113cc0-84e0-4de6-8078-aa79ad5cad66","Type":"ContainerDied","Data":"0cb077b09086d13c7753b36c9464843d37947b08f8b8623cb177fd51f6ebeef5"} Nov 01 16:32:08.859594 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:08.859533 4363 scope.go:122] "RemoveContainer" containerID="2d98c1ea18ed165181d0193fc3ef20f590882d103df04962c57309157bc17ef7" Nov 01 16:32:08.859594 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:08.859461 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4" Nov 01 16:32:08.881231 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:08.881201 4363 scope.go:122] "RemoveContainer" containerID="23fa7d7a5213c3f856678ea3caab08f627ff8f5dd3bc2456e2f45811ed5a5ab0" Nov 01 16:32:08.887331 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:08.887306 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4"] Nov 01 16:32:08.901005 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:08.893590 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-6b6cfc-openshift-default-75cc4f89-d6dw4"] Nov 01 16:32:08.902863 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:08.902846 4363 scope.go:122] "RemoveContainer" containerID="2d98c1ea18ed165181d0193fc3ef20f590882d103df04962c57309157bc17ef7" Nov 01 16:32:08.903526 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:32:08.903502 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d98c1ea18ed165181d0193fc3ef20f590882d103df04962c57309157bc17ef7\": container with ID starting with 2d98c1ea18ed165181d0193fc3ef20f590882d103df04962c57309157bc17ef7 not found: ID does not exist" containerID="2d98c1ea18ed165181d0193fc3ef20f590882d103df04962c57309157bc17ef7" Nov 01 16:32:08.903637 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:08.903538 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d98c1ea18ed165181d0193fc3ef20f590882d103df04962c57309157bc17ef7"} err="failed to get container status \"2d98c1ea18ed165181d0193fc3ef20f590882d103df04962c57309157bc17ef7\": rpc error: code = NotFound desc = could not find container \"2d98c1ea18ed165181d0193fc3ef20f590882d103df04962c57309157bc17ef7\": container with ID starting with 2d98c1ea18ed165181d0193fc3ef20f590882d103df04962c57309157bc17ef7 not found: ID does not exist" Nov 01 16:32:08.903637 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:08.903563 4363 scope.go:122] "RemoveContainer" containerID="23fa7d7a5213c3f856678ea3caab08f627ff8f5dd3bc2456e2f45811ed5a5ab0" Nov 01 16:32:08.905146 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:32:08.905112 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23fa7d7a5213c3f856678ea3caab08f627ff8f5dd3bc2456e2f45811ed5a5ab0\": container with ID starting with 23fa7d7a5213c3f856678ea3caab08f627ff8f5dd3bc2456e2f45811ed5a5ab0 not found: ID does not exist" containerID="23fa7d7a5213c3f856678ea3caab08f627ff8f5dd3bc2456e2f45811ed5a5ab0" Nov 01 16:32:08.905244 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:08.905154 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23fa7d7a5213c3f856678ea3caab08f627ff8f5dd3bc2456e2f45811ed5a5ab0"} err="failed to get container status \"23fa7d7a5213c3f856678ea3caab08f627ff8f5dd3bc2456e2f45811ed5a5ab0\": rpc error: code = NotFound desc = could not find container \"23fa7d7a5213c3f856678ea3caab08f627ff8f5dd3bc2456e2f45811ed5a5ab0\": container with ID starting with 23fa7d7a5213c3f856678ea3caab08f627ff8f5dd3bc2456e2f45811ed5a5ab0 not found: ID does not exist" Nov 01 16:32:09.051696 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.051662 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr"] Nov 01 16:32:09.052417 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.052402 4363 cpu_manager.go:414] "containerMap: removing container" podUID="500072d0-7098-41b9-8498-1804ad8307a1" containerName="revoke-keys" Nov 01 16:32:09.052524 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.052420 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="500072d0-7098-41b9-8498-1804ad8307a1" containerName="revoke-keys" Nov 01 16:32:09.052524 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.052435 4363 cpu_manager.go:414] "containerMap: removing container" podUID="69113cc0-84e0-4de6-8078-aa79ad5cad66" containerName="istio-proxy" Nov 01 16:32:09.052524 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.052441 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69113cc0-84e0-4de6-8078-aa79ad5cad66" containerName="istio-proxy" Nov 01 16:32:09.052524 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.052466 4363 cpu_manager.go:414] "containerMap: removing container" podUID="3b5725f6-5b76-4300-9886-6a8c2556e6ae" containerName="istio-proxy" Nov 01 16:32:09.052524 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.052471 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b5725f6-5b76-4300-9886-6a8c2556e6ae" containerName="istio-proxy" Nov 01 16:32:09.052524 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.052490 4363 cpu_manager.go:414] "containerMap: removing container" podUID="3b5725f6-5b76-4300-9886-6a8c2556e6ae" containerName="istio-proxy" Nov 01 16:32:09.052524 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.052495 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b5725f6-5b76-4300-9886-6a8c2556e6ae" containerName="istio-proxy" Nov 01 16:32:09.053001 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.052593 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="500072d0-7098-41b9-8498-1804ad8307a1" containerName="revoke-keys" Nov 01 16:32:09.053001 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.052603 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="69113cc0-84e0-4de6-8078-aa79ad5cad66" containerName="istio-proxy" Nov 01 16:32:09.053001 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.052620 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b5725f6-5b76-4300-9886-6a8c2556e6ae" containerName="istio-proxy" Nov 01 16:32:09.053001 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.052629 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b5725f6-5b76-4300-9886-6a8c2556e6ae" containerName="istio-proxy" Nov 01 16:32:09.057556 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.057538 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" Nov 01 16:32:09.069574 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.069552 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr"] Nov 01 16:32:09.161663 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.161632 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-43fac8-84b47867b7-4gzrb"] Nov 01 16:32:09.194181 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.193395 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc3066c-d320-41fa-8a7b-933362174b48-istio-podinfo\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr\" (UID: \"0cc3066c-d320-41fa-8a7b-933362174b48\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" Nov 01 16:32:09.194181 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.193478 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc3066c-d320-41fa-8a7b-933362174b48-credential-socket\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr\" (UID: \"0cc3066c-d320-41fa-8a7b-933362174b48\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" Nov 01 16:32:09.194181 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.193534 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc3066c-d320-41fa-8a7b-933362174b48-istio-envoy\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr\" (UID: \"0cc3066c-d320-41fa-8a7b-933362174b48\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" Nov 01 16:32:09.194181 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.193557 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc3066c-d320-41fa-8a7b-933362174b48-workload-certs\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr\" (UID: \"0cc3066c-d320-41fa-8a7b-933362174b48\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" Nov 01 16:32:09.194181 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.193587 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc3066c-d320-41fa-8a7b-933362174b48-istio-token\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr\" (UID: \"0cc3066c-d320-41fa-8a7b-933362174b48\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" Nov 01 16:32:09.194181 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.193621 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc3066c-d320-41fa-8a7b-933362174b48-istio-data\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr\" (UID: \"0cc3066c-d320-41fa-8a7b-933362174b48\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" Nov 01 16:32:09.194181 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.193691 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t92vm\" (UniqueName: \"kubernetes.io/projected/0cc3066c-d320-41fa-8a7b-933362174b48-kube-api-access-t92vm\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr\" (UID: \"0cc3066c-d320-41fa-8a7b-933362174b48\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" Nov 01 16:32:09.194181 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.193715 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc3066c-d320-41fa-8a7b-933362174b48-workload-socket\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr\" (UID: \"0cc3066c-d320-41fa-8a7b-933362174b48\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" Nov 01 16:32:09.194181 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.193746 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc3066c-d320-41fa-8a7b-933362174b48-istiod-ca-cert\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr\" (UID: \"0cc3066c-d320-41fa-8a7b-933362174b48\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" Nov 01 16:32:09.294644 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.294607 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc3066c-d320-41fa-8a7b-933362174b48-istio-data\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr\" (UID: \"0cc3066c-d320-41fa-8a7b-933362174b48\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" Nov 01 16:32:09.294809 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.294706 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t92vm\" (UniqueName: \"kubernetes.io/projected/0cc3066c-d320-41fa-8a7b-933362174b48-kube-api-access-t92vm\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr\" (UID: \"0cc3066c-d320-41fa-8a7b-933362174b48\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" Nov 01 16:32:09.294809 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.294740 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc3066c-d320-41fa-8a7b-933362174b48-workload-socket\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr\" (UID: \"0cc3066c-d320-41fa-8a7b-933362174b48\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" Nov 01 16:32:09.294809 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.294772 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc3066c-d320-41fa-8a7b-933362174b48-istiod-ca-cert\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr\" (UID: \"0cc3066c-d320-41fa-8a7b-933362174b48\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" Nov 01 16:32:09.294934 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.294814 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc3066c-d320-41fa-8a7b-933362174b48-istio-podinfo\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr\" (UID: \"0cc3066c-d320-41fa-8a7b-933362174b48\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" Nov 01 16:32:09.294934 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.294899 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc3066c-d320-41fa-8a7b-933362174b48-credential-socket\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr\" (UID: \"0cc3066c-d320-41fa-8a7b-933362174b48\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" Nov 01 16:32:09.294998 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.294954 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc3066c-d320-41fa-8a7b-933362174b48-istio-data\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr\" (UID: \"0cc3066c-d320-41fa-8a7b-933362174b48\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" Nov 01 16:32:09.295085 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.294997 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc3066c-d320-41fa-8a7b-933362174b48-istio-envoy\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr\" (UID: \"0cc3066c-d320-41fa-8a7b-933362174b48\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" Nov 01 16:32:09.295085 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.295051 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc3066c-d320-41fa-8a7b-933362174b48-workload-certs\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr\" (UID: \"0cc3066c-d320-41fa-8a7b-933362174b48\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" Nov 01 16:32:09.295201 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.295098 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc3066c-d320-41fa-8a7b-933362174b48-istio-token\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr\" (UID: \"0cc3066c-d320-41fa-8a7b-933362174b48\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" Nov 01 16:32:09.295359 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.295308 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc3066c-d320-41fa-8a7b-933362174b48-credential-socket\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr\" (UID: \"0cc3066c-d320-41fa-8a7b-933362174b48\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" Nov 01 16:32:09.295607 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.295589 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc3066c-d320-41fa-8a7b-933362174b48-workload-certs\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr\" (UID: \"0cc3066c-d320-41fa-8a7b-933362174b48\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" Nov 01 16:32:09.295697 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.295674 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc3066c-d320-41fa-8a7b-933362174b48-workload-socket\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr\" (UID: \"0cc3066c-d320-41fa-8a7b-933362174b48\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" Nov 01 16:32:09.295784 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.295761 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc3066c-d320-41fa-8a7b-933362174b48-istiod-ca-cert\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr\" (UID: \"0cc3066c-d320-41fa-8a7b-933362174b48\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" Nov 01 16:32:09.297389 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.297360 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc3066c-d320-41fa-8a7b-933362174b48-istio-envoy\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr\" (UID: \"0cc3066c-d320-41fa-8a7b-933362174b48\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" Nov 01 16:32:09.297739 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.297720 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc3066c-d320-41fa-8a7b-933362174b48-istio-podinfo\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr\" (UID: \"0cc3066c-d320-41fa-8a7b-933362174b48\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" Nov 01 16:32:09.302428 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.302401 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc3066c-d320-41fa-8a7b-933362174b48-istio-token\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr\" (UID: \"0cc3066c-d320-41fa-8a7b-933362174b48\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" Nov 01 16:32:09.302428 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.302417 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t92vm\" (UniqueName: \"kubernetes.io/projected/0cc3066c-d320-41fa-8a7b-933362174b48-kube-api-access-t92vm\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr\" (UID: \"0cc3066c-d320-41fa-8a7b-933362174b48\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" Nov 01 16:32:09.374161 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.374135 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" Nov 01 16:32:09.865460 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:32:09.865423 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cc3066c_d320_41fa_8a7b_933362174b48.slice/crio-4265fdc03efacd8194a0b5275e811e0d6b31bc3f4a37b9272afc8496de844eb6 WatchSource:0}: Error finding container 4265fdc03efacd8194a0b5275e811e0d6b31bc3f4a37b9272afc8496de844eb6: Status 404 returned error can't find the container with id 4265fdc03efacd8194a0b5275e811e0d6b31bc3f4a37b9272afc8496de844eb6 Nov 01 16:32:09.865909 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.865488 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr"] Nov 01 16:32:09.891113 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.869777 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:32:09.891113 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.869939 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:32:09.891113 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:09.870046 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:32:10.820769 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:10.820726 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69113cc0-84e0-4de6-8078-aa79ad5cad66" path="/var/lib/kubelet/pods/69113cc0-84e0-4de6-8078-aa79ad5cad66/volumes" Nov 01 16:32:10.927301 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:10.927260 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" event={"ID":"0cc3066c-d320-41fa-8a7b-933362174b48","Type":"ContainerStarted","Data":"dd974f5e7dc1362a51e357707f821518234452bfe12c8535bdc36e95ea1c624d"} Nov 01 16:32:10.927823 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:10.927341 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" event={"ID":"0cc3066c-d320-41fa-8a7b-933362174b48","Type":"ContainerStarted","Data":"4265fdc03efacd8194a0b5275e811e0d6b31bc3f4a37b9272afc8496de844eb6"} Nov 01 16:32:10.971686 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:10.971633 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" podStartSLOduration=1.971603613 podStartE2EDuration="1.971603613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:32:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:32:10.969419594 +0000 UTC m=+2664.301157256" watchObservedRunningTime="2026-11-01 16:32:10.971603613 +0000 UTC m=+2664.303341262" Nov 01 16:32:11.374601 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:11.374571 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" Nov 01 16:32:11.375781 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:11.375755 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body= Nov 01 16:32:11.375930 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:11.375801 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" podUID="0cc3066c-d320-41fa-8a7b-933362174b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" Nov 01 16:32:12.374995 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:12.374961 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body= Nov 01 16:32:12.375430 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:12.375009 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" podUID="0cc3066c-d320-41fa-8a7b-933362174b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" Nov 01 16:32:13.375415 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:13.375379 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body= Nov 01 16:32:13.375901 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:13.375429 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" podUID="0cc3066c-d320-41fa-8a7b-933362174b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" Nov 01 16:32:13.802640 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:13.802554 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f47890-84fdfb9c59-fs5nn"] Nov 01 16:32:14.180052 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:14.174714 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-43fac8-84b47867b7-4gzrb" podUID="bb1a7676-ecc1-451e-ba91-83f91fdf1bed" containerName="payload-pre-processing" containerID="cri-o://2e2108f9def8588cd4c86249d485464e54f2df8836971eb7c1a8b872c24ab362" gracePeriod=25 Nov 01 16:32:14.375377 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:14.375344 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body= Nov 01 16:32:14.375546 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:14.375395 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" podUID="0cc3066c-d320-41fa-8a7b-933362174b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" Nov 01 16:32:15.374790 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:15.374736 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body= Nov 01 16:32:15.375057 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:15.374812 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" podUID="0cc3066c-d320-41fa-8a7b-933362174b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" Nov 01 16:32:16.328065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:16.328005 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8b7b001-lkbmv"] Nov 01 16:32:16.329589 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:16.329548 4363 cpu_manager.go:414] "containerMap: removing container" podUID="69113cc0-84e0-4de6-8078-aa79ad5cad66" containerName="istio-proxy" Nov 01 16:32:16.329589 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:16.329577 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69113cc0-84e0-4de6-8078-aa79ad5cad66" containerName="istio-proxy" Nov 01 16:32:16.329789 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:16.329768 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="69113cc0-84e0-4de6-8078-aa79ad5cad66" containerName="istio-proxy" Nov 01 16:32:16.352080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:16.336606 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b7b001-lkbmv" Nov 01 16:32:16.361419 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:16.361371 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8b7b001-lkbmv"] Nov 01 16:32:16.375376 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:16.375343 4363 patch_prober.go:29] interesting pod/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body= Nov 01 16:32:16.375596 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:16.375562 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" podUID="0cc3066c-d320-41fa-8a7b-933362174b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" Nov 01 16:32:16.420074 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:16.420025 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpf6q\" (UniqueName: \"kubernetes.io/projected/bf475632-fcc1-4d2d-a98b-65464c14f9b7-kube-api-access-jpf6q\") pod \"maas-api-revoke-sub-f8b7b001-lkbmv\" (UID: \"bf475632-fcc1-4d2d-a98b-65464c14f9b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b7b001-lkbmv" Nov 01 16:32:16.420542 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:16.420515 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf475632-fcc1-4d2d-a98b-65464c14f9b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8b7b001-lkbmv\" (UID: \"bf475632-fcc1-4d2d-a98b-65464c14f9b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b7b001-lkbmv" Nov 01 16:32:16.420720 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:16.420700 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf475632-fcc1-4d2d-a98b-65464c14f9b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8b7b001-lkbmv\" (UID: \"bf475632-fcc1-4d2d-a98b-65464c14f9b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b7b001-lkbmv" Nov 01 16:32:16.422441 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:16.422421 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-028c46-9f59dd876-9bxwr"] Nov 01 16:32:16.422729 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:16.422705 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-028c46-9f59dd876-9bxwr" podUID="c033e4f6-d864-4579-9793-037ec810172d" containerName="payload-processing" containerID="cri-o://10fa56441b78b5bb8865f2e69663e7c72b1a27d3fb2b4ac37af8a6d59c3125e9" gracePeriod=30 Nov 01 16:32:16.427222 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:16.427042 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:32:16.428142 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:16.428128 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:32:16.526139 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:16.526095 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf475632-fcc1-4d2d-a98b-65464c14f9b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8b7b001-lkbmv\" (UID: \"bf475632-fcc1-4d2d-a98b-65464c14f9b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b7b001-lkbmv" Nov 01 16:32:16.526319 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:16.526182 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf475632-fcc1-4d2d-a98b-65464c14f9b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8b7b001-lkbmv\" (UID: \"bf475632-fcc1-4d2d-a98b-65464c14f9b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b7b001-lkbmv" Nov 01 16:32:16.526319 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:16.526258 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpf6q\" (UniqueName: \"kubernetes.io/projected/bf475632-fcc1-4d2d-a98b-65464c14f9b7-kube-api-access-jpf6q\") pod \"maas-api-revoke-sub-f8b7b001-lkbmv\" (UID: \"bf475632-fcc1-4d2d-a98b-65464c14f9b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b7b001-lkbmv" Nov 01 16:32:16.536253 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:16.533945 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf475632-fcc1-4d2d-a98b-65464c14f9b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8b7b001-lkbmv\" (UID: \"bf475632-fcc1-4d2d-a98b-65464c14f9b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b7b001-lkbmv" Nov 01 16:32:16.560269 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:16.559334 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpf6q\" (UniqueName: \"kubernetes.io/projected/bf475632-fcc1-4d2d-a98b-65464c14f9b7-kube-api-access-jpf6q\") pod \"maas-api-revoke-sub-f8b7b001-lkbmv\" (UID: \"bf475632-fcc1-4d2d-a98b-65464c14f9b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b7b001-lkbmv" Nov 01 16:32:16.567906 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:16.565767 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf475632-fcc1-4d2d-a98b-65464c14f9b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8b7b001-lkbmv\" (UID: \"bf475632-fcc1-4d2d-a98b-65464c14f9b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b7b001-lkbmv" Nov 01 16:32:16.675258 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:16.674603 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b7b001-lkbmv" Nov 01 16:32:16.677653 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:16.677427 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-028c46-rnvcs"] Nov 01 16:32:16.681672 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:16.681618 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-028c46-rnvcs"] Nov 01 16:32:16.961583 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:16.961546 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="500072d0-7098-41b9-8498-1804ad8307a1" path="/var/lib/kubelet/pods/500072d0-7098-41b9-8498-1804ad8307a1/volumes" Nov 01 16:32:16.985120 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:16.968657 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-028c46-9f59dd876-9bxwr" Nov 01 16:32:17.025708 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:17.024847 4363 generic.go:356] "Generic (PLEG): container finished" podID="c033e4f6-d864-4579-9793-037ec810172d" containerID="10fa56441b78b5bb8865f2e69663e7c72b1a27d3fb2b4ac37af8a6d59c3125e9" exitCode=0 Nov 01 16:32:17.025708 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:17.025032 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-028c46-9f59dd876-9bxwr" event={"ID":"c033e4f6-d864-4579-9793-037ec810172d","Type":"ContainerDied","Data":"10fa56441b78b5bb8865f2e69663e7c72b1a27d3fb2b4ac37af8a6d59c3125e9"} Nov 01 16:32:17.025708 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:17.025087 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-028c46-9f59dd876-9bxwr" event={"ID":"c033e4f6-d864-4579-9793-037ec810172d","Type":"ContainerDied","Data":"de4ff1353bb7ee3e89567f6e75f7e66b54f3df6d5eaf22d92a79605470bbbf1d"} Nov 01 16:32:17.025708 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:17.025119 4363 scope.go:122] "RemoveContainer" containerID="10fa56441b78b5bb8865f2e69663e7c72b1a27d3fb2b4ac37af8a6d59c3125e9" Nov 01 16:32:17.025708 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:17.025319 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-028c46-9f59dd876-9bxwr" Nov 01 16:32:17.120138 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:17.119768 4363 scope.go:122] "RemoveContainer" containerID="10fa56441b78b5bb8865f2e69663e7c72b1a27d3fb2b4ac37af8a6d59c3125e9" Nov 01 16:32:17.120488 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:32:17.120386 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10fa56441b78b5bb8865f2e69663e7c72b1a27d3fb2b4ac37af8a6d59c3125e9\": container with ID starting with 10fa56441b78b5bb8865f2e69663e7c72b1a27d3fb2b4ac37af8a6d59c3125e9 not found: ID does not exist" containerID="10fa56441b78b5bb8865f2e69663e7c72b1a27d3fb2b4ac37af8a6d59c3125e9" Nov 01 16:32:17.120488 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:17.120423 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10fa56441b78b5bb8865f2e69663e7c72b1a27d3fb2b4ac37af8a6d59c3125e9"} err="failed to get container status \"10fa56441b78b5bb8865f2e69663e7c72b1a27d3fb2b4ac37af8a6d59c3125e9\": rpc error: code = NotFound desc = could not find container \"10fa56441b78b5bb8865f2e69663e7c72b1a27d3fb2b4ac37af8a6d59c3125e9\": container with ID starting with 10fa56441b78b5bb8865f2e69663e7c72b1a27d3fb2b4ac37af8a6d59c3125e9 not found: ID does not exist" Nov 01 16:32:17.153044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:17.151318 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c033e4f6-d864-4579-9793-037ec810172d-kube-api-access-px5pr\" (UniqueName: \"kubernetes.io/projected/c033e4f6-d864-4579-9793-037ec810172d-kube-api-access-px5pr\") pod \"c033e4f6-d864-4579-9793-037ec810172d\" (UID: \"c033e4f6-d864-4579-9793-037ec810172d\") " Nov 01 16:32:17.153044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:17.151498 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c033e4f6-d864-4579-9793-037ec810172d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c033e4f6-d864-4579-9793-037ec810172d-plugins-config-volume\") pod \"c033e4f6-d864-4579-9793-037ec810172d\" (UID: \"c033e4f6-d864-4579-9793-037ec810172d\") " Nov 01 16:32:17.153044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:17.152152 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c033e4f6-d864-4579-9793-037ec810172d-plugins-config-volume" pod "c033e4f6-d864-4579-9793-037ec810172d" (UID: "c033e4f6-d864-4579-9793-037ec810172d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:32:17.154328 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:17.154301 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c033e4f6-d864-4579-9793-037ec810172d-kube-api-access-px5pr" pod "c033e4f6-d864-4579-9793-037ec810172d" (UID: "c033e4f6-d864-4579-9793-037ec810172d"). InnerVolumeSpecName "kube-api-access-px5pr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:17.254098 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:17.254062 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c033e4f6-d864-4579-9793-037ec810172d-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:17.254098 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:17.254097 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px5pr\" (UniqueName: \"kubernetes.io/projected/c033e4f6-d864-4579-9793-037ec810172d-kube-api-access-px5pr\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:17.346678 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:17.346645 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8b7b001-lkbmv"] Nov 01 16:32:17.347098 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:32:17.347073 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf475632_fcc1_4d2d_a98b_65464c14f9b7.slice/crio-acba650f0b10e9e01f91351811763bddd49182d834f4159174d153347f8007e3 WatchSource:0}: Error finding container acba650f0b10e9e01f91351811763bddd49182d834f4159174d153347f8007e3: Status 404 returned error can't find the container with id acba650f0b10e9e01f91351811763bddd49182d834f4159174d153347f8007e3 Nov 01 16:32:17.361953 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:17.361929 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-028c46-9f59dd876-9bxwr"] Nov 01 16:32:17.368677 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:17.367953 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-028c46-9f59dd876-9bxwr"] Nov 01 16:32:17.380431 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:17.380408 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" Nov 01 16:32:17.380734 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:17.380709 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" Nov 01 16:32:17.381596 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:17.381578 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" Nov 01 16:32:18.034405 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:18.034316 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b7b001-lkbmv" event={"ID":"bf475632-fcc1-4d2d-a98b-65464c14f9b7","Type":"ContainerStarted","Data":"3f32c601b7820c9f7ce4f2745fcb8f519a0c7c0e0c8874ea66d25a0147f0c7df"} Nov 01 16:32:18.034405 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:18.034374 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b7b001-lkbmv" event={"ID":"bf475632-fcc1-4d2d-a98b-65464c14f9b7","Type":"ContainerStarted","Data":"acba650f0b10e9e01f91351811763bddd49182d834f4159174d153347f8007e3"} Nov 01 16:32:18.076777 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:18.076729 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b7b001-lkbmv" podStartSLOduration=2.076716342 podStartE2EDuration="2.076716342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:32:18.07384227 +0000 UTC m=+2671.405579921" watchObservedRunningTime="2026-11-01 16:32:18.076716342 +0000 UTC m=+2671.408454020" Nov 01 16:32:18.313501 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:32:18.313467 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:32:18.800510 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:18.800475 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c033e4f6-d864-4579-9793-037ec810172d" path="/var/lib/kubelet/pods/c033e4f6-d864-4579-9793-037ec810172d/volumes" Nov 01 16:32:18.803555 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:18.803526 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f47890-84fdfb9c59-fs5nn" podUID="18faa082-6418-4665-b862-09efa76dfcc0" containerName="payload-pre-processing" containerID="cri-o://29c3809b77e4154932cca1a2bde9b52af7ad2e0b5f4d661ed64bb44b9d9c7e85" gracePeriod=25 Nov 01 16:32:18.829900 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:18.829865 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" containerID="cri-o://f2adbe3cf2e7a60c142f65b1279658679fa4e5efabad423f928331184d7ee7d0" gracePeriod=45 Nov 01 16:32:18.895466 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:18.895405 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": dial tcp 10.134.0.65:8000: connect: connection refused" Nov 01 16:32:19.052327 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:19.052238 4363 generic.go:356] "Generic (PLEG): container finished" podID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerID="f2adbe3cf2e7a60c142f65b1279658679fa4e5efabad423f928331184d7ee7d0" exitCode=0 Nov 01 16:32:19.052497 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:19.052316 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" event={"ID":"e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4","Type":"ContainerDied","Data":"f2adbe3cf2e7a60c142f65b1279658679fa4e5efabad423f928331184d7ee7d0"} Nov 01 16:32:19.054513 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:19.054487 4363 generic.go:356] "Generic (PLEG): container finished" podID="bf475632-fcc1-4d2d-a98b-65464c14f9b7" containerID="3f32c601b7820c9f7ce4f2745fcb8f519a0c7c0e0c8874ea66d25a0147f0c7df" exitCode=0 Nov 01 16:32:19.054648 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:19.054569 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b7b001-lkbmv" event={"ID":"bf475632-fcc1-4d2d-a98b-65464c14f9b7","Type":"ContainerDied","Data":"3f32c601b7820c9f7ce4f2745fcb8f519a0c7c0e0c8874ea66d25a0147f0c7df"} Nov 01 16:32:19.121319 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:19.121291 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:32:19.199053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:19.198999 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4-kube-api-access-kmdkv\" (UniqueName: \"kubernetes.io/projected/e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4-kube-api-access-kmdkv\") pod \"e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4\" (UID: \"e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4\") " Nov 01 16:32:19.199244 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:19.199121 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4-model-cache\") pod \"e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4\" (UID: \"e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4\") " Nov 01 16:32:19.199314 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:19.199285 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4-tmp-dir\") pod \"e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4\" (UID: \"e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4\") " Nov 01 16:32:19.199381 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:19.199340 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4-dshm\") pod \"e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4\" (UID: \"e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4\") " Nov 01 16:32:19.199434 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:19.199374 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4-tls-certs\" (UniqueName: \"kubernetes.io/secret/e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4-tls-certs\") pod \"e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4\" (UID: \"e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4\") " Nov 01 16:32:19.199434 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:19.199381 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4-model-cache" pod "e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" (UID: "e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:19.199546 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:19.199447 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4-home\" (UniqueName: \"kubernetes.io/empty-dir/e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4-home\") pod \"e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4\" (UID: \"e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4\") " Nov 01 16:32:19.199612 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:19.199564 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4-tmp-dir" pod "e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" (UID: "e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:19.199853 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:19.199826 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4-home" pod "e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" (UID: "e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:19.200365 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:19.200341 4363 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4-model-cache\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:19.200491 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:19.200370 4363 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4-tmp-dir\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:19.200491 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:19.200386 4363 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4-home\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:19.201824 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:19.201782 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4-tls-certs" pod "e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" (UID: "e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:32:19.201925 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:19.201867 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4-dshm" pod "e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" (UID: "e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:19.201990 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:19.201974 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4-kube-api-access-kmdkv" pod "e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" (UID: "e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4"). InnerVolumeSpecName "kube-api-access-kmdkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:19.302229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:19.302184 4363 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4-dshm\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:19.302229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:19.302224 4363 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4-tls-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:19.302229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:19.302236 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmdkv\" (UniqueName: \"kubernetes.io/projected/e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4-kube-api-access-kmdkv\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:20.100042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:20.099210 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" Nov 01 16:32:20.101211 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:20.101179 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql" event={"ID":"e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4","Type":"ContainerDied","Data":"81f597a986648e1ca4e300dc865b67e136b368e5b132f186e30a441ac41bb957"} Nov 01 16:32:20.101414 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:20.101398 4363 scope.go:122] "RemoveContainer" containerID="f2adbe3cf2e7a60c142f65b1279658679fa4e5efabad423f928331184d7ee7d0" Nov 01 16:32:20.289189 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:20.289130 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql"] Nov 01 16:32:20.357160 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:20.357071 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-43fac8/distinct-2-w6-43fac8-kserve-7994bb8d4c-bhwql"] Nov 01 16:32:20.545995 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:20.545926 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-43fac8-84b47867b7-4gzrb" podUID="bb1a7676-ecc1-451e-ba91-83f91fdf1bed" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:32:20.863278 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:20.863239 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" path="/var/lib/kubelet/pods/e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4/volumes" Nov 01 16:32:21.047671 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:21.047647 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b7b001-lkbmv" Nov 01 16:32:21.085625 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:21.085580 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf475632-fcc1-4d2d-a98b-65464c14f9b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf475632-fcc1-4d2d-a98b-65464c14f9b7-maas-api-openshift-ca\") pod \"bf475632-fcc1-4d2d-a98b-65464c14f9b7\" (UID: \"bf475632-fcc1-4d2d-a98b-65464c14f9b7\") " Nov 01 16:32:21.085625 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:21.085618 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf475632-fcc1-4d2d-a98b-65464c14f9b7-kube-api-access-jpf6q\" (UniqueName: \"kubernetes.io/projected/bf475632-fcc1-4d2d-a98b-65464c14f9b7-kube-api-access-jpf6q\") pod \"bf475632-fcc1-4d2d-a98b-65464c14f9b7\" (UID: \"bf475632-fcc1-4d2d-a98b-65464c14f9b7\") " Nov 01 16:32:21.085921 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:21.085851 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf475632-fcc1-4d2d-a98b-65464c14f9b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf475632-fcc1-4d2d-a98b-65464c14f9b7-maas-api-xks-ca\") pod \"bf475632-fcc1-4d2d-a98b-65464c14f9b7\" (UID: \"bf475632-fcc1-4d2d-a98b-65464c14f9b7\") " Nov 01 16:32:21.086130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:21.086093 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf475632-fcc1-4d2d-a98b-65464c14f9b7-maas-api-openshift-ca" pod "bf475632-fcc1-4d2d-a98b-65464c14f9b7" (UID: "bf475632-fcc1-4d2d-a98b-65464c14f9b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:32:21.086572 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:21.086544 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf475632-fcc1-4d2d-a98b-65464c14f9b7-maas-api-openshift-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:21.088249 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:21.088222 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf475632-fcc1-4d2d-a98b-65464c14f9b7-kube-api-access-jpf6q" pod "bf475632-fcc1-4d2d-a98b-65464c14f9b7" (UID: "bf475632-fcc1-4d2d-a98b-65464c14f9b7"). InnerVolumeSpecName "kube-api-access-jpf6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:21.088648 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:21.088610 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf475632-fcc1-4d2d-a98b-65464c14f9b7-maas-api-xks-ca" pod "bf475632-fcc1-4d2d-a98b-65464c14f9b7" (UID: "bf475632-fcc1-4d2d-a98b-65464c14f9b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:32:21.111703 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:21.111675 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b7b001-lkbmv" Nov 01 16:32:21.112150 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:21.111715 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b7b001-lkbmv" event={"ID":"bf475632-fcc1-4d2d-a98b-65464c14f9b7","Type":"ContainerDied","Data":"acba650f0b10e9e01f91351811763bddd49182d834f4159174d153347f8007e3"} Nov 01 16:32:21.112150 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:21.111760 4363 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acba650f0b10e9e01f91351811763bddd49182d834f4159174d153347f8007e3" Nov 01 16:32:21.187646 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:21.187611 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf475632-fcc1-4d2d-a98b-65464c14f9b7-maas-api-xks-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:21.187646 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:21.187638 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpf6q\" (UniqueName: \"kubernetes.io/projected/bf475632-fcc1-4d2d-a98b-65464c14f9b7-kube-api-access-jpf6q\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:23.618942 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:23.618737 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl"] Nov 01 16:32:23.619477 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:23.619003 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" podUID="f3655831-2619-45f1-9da3-5bbe7dca87f4" containerName="istio-proxy" containerID="cri-o://82772b8f1c16ec6807099169d4a16b294ecc89b48490df7a20150da84c0d3beb" gracePeriod=30 Nov 01 16:32:23.629531 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:23.629499 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r"] Nov 01 16:32:23.629983 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:23.629742 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" podUID="4995f2f9-c830-41b4-b501-d8aacb690731" containerName="istio-proxy" containerID="cri-o://375710f4d1c2055c11f357a67a5833fa64df78c2deea66dac61991b8484b52ed" gracePeriod=30 Nov 01 16:32:26.338292 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:26.338260 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f47890-84fdfb9c59-fs5nn" podUID="18faa082-6418-4665-b862-09efa76dfcc0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:32:27.900312 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:27.900277 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-43fac8-ff59b5dfb-r8zjf"] Nov 01 16:32:27.900894 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:27.900564 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-43fac8-ff59b5dfb-r8zjf" podUID="6be52ed0-203a-4a91-abee-9420e0dff7dd" containerName="maas-api" containerID="cri-o://f61adfd659bef00b44fed69eb071f6d14300a9b8e907db26172a4c7b964273db" gracePeriod=30 Nov 01 16:32:28.185104 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:28.179640 4363 patch_prober.go:29] interesting pod/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:32:28.185104 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:28.179745 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" podUID="f3655831-2619-45f1-9da3-5bbe7dca87f4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:32:28.188180 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:28.187906 4363 generic.go:356] "Generic (PLEG): container finished" podID="6be52ed0-203a-4a91-abee-9420e0dff7dd" containerID="f61adfd659bef00b44fed69eb071f6d14300a9b8e907db26172a4c7b964273db" exitCode=0 Nov 01 16:32:28.188180 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:28.188075 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-43fac8-ff59b5dfb-r8zjf" event={"ID":"6be52ed0-203a-4a91-abee-9420e0dff7dd","Type":"ContainerDied","Data":"f61adfd659bef00b44fed69eb071f6d14300a9b8e907db26172a4c7b964273db"} Nov 01 16:32:28.333373 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:28.333323 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-43fac8-mr4wd"] Nov 01 16:32:28.340717 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:28.339879 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-43fac8-mr4wd"] Nov 01 16:32:28.405498 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:28.405284 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-43fac8-ff59b5dfb-r8zjf" Nov 01 16:32:28.493087 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:28.492993 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6be52ed0-203a-4a91-abee-9420e0dff7dd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6be52ed0-203a-4a91-abee-9420e0dff7dd-maas-api-tls\") pod \"6be52ed0-203a-4a91-abee-9420e0dff7dd\" (UID: \"6be52ed0-203a-4a91-abee-9420e0dff7dd\") " Nov 01 16:32:28.493261 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:28.493126 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6be52ed0-203a-4a91-abee-9420e0dff7dd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6be52ed0-203a-4a91-abee-9420e0dff7dd-metrics-certs\") pod \"6be52ed0-203a-4a91-abee-9420e0dff7dd\" (UID: \"6be52ed0-203a-4a91-abee-9420e0dff7dd\") " Nov 01 16:32:28.493261 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:28.493204 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6be52ed0-203a-4a91-abee-9420e0dff7dd-kube-api-access-2rzq2\" (UniqueName: \"kubernetes.io/projected/6be52ed0-203a-4a91-abee-9420e0dff7dd-kube-api-access-2rzq2\") pod \"6be52ed0-203a-4a91-abee-9420e0dff7dd\" (UID: \"6be52ed0-203a-4a91-abee-9420e0dff7dd\") " Nov 01 16:32:28.493403 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:28.493264 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6be52ed0-203a-4a91-abee-9420e0dff7dd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6be52ed0-203a-4a91-abee-9420e0dff7dd-openshift-service-ca\") pod \"6be52ed0-203a-4a91-abee-9420e0dff7dd\" (UID: \"6be52ed0-203a-4a91-abee-9420e0dff7dd\") " Nov 01 16:32:28.493628 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:28.493601 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6be52ed0-203a-4a91-abee-9420e0dff7dd-openshift-service-ca" pod "6be52ed0-203a-4a91-abee-9420e0dff7dd" (UID: "6be52ed0-203a-4a91-abee-9420e0dff7dd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:32:28.494093 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:28.494071 4363 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6be52ed0-203a-4a91-abee-9420e0dff7dd-openshift-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:28.495347 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:28.495324 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6be52ed0-203a-4a91-abee-9420e0dff7dd-metrics-certs" pod "6be52ed0-203a-4a91-abee-9420e0dff7dd" (UID: "6be52ed0-203a-4a91-abee-9420e0dff7dd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:32:28.495836 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:28.495816 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6be52ed0-203a-4a91-abee-9420e0dff7dd-maas-api-tls" pod "6be52ed0-203a-4a91-abee-9420e0dff7dd" (UID: "6be52ed0-203a-4a91-abee-9420e0dff7dd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:32:28.496440 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:28.496418 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be52ed0-203a-4a91-abee-9420e0dff7dd-kube-api-access-2rzq2" pod "6be52ed0-203a-4a91-abee-9420e0dff7dd" (UID: "6be52ed0-203a-4a91-abee-9420e0dff7dd"). InnerVolumeSpecName "kube-api-access-2rzq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:28.597178 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:28.597139 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6be52ed0-203a-4a91-abee-9420e0dff7dd-maas-api-tls\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:28.597178 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:28.597177 4363 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6be52ed0-203a-4a91-abee-9420e0dff7dd-metrics-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:28.597408 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:28.597193 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rzq2\" (UniqueName: \"kubernetes.io/projected/6be52ed0-203a-4a91-abee-9420e0dff7dd-kube-api-access-2rzq2\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:28.809365 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:28.809329 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="857faaf7-ea2e-4d29-8b08-32e0d5b083e8" path="/var/lib/kubelet/pods/857faaf7-ea2e-4d29-8b08-32e0d5b083e8/volumes" Nov 01 16:32:29.001166 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.001122 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" Nov 01 16:32:29.062771 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.062597 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" Nov 01 16:32:29.119763 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.119730 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4995f2f9-c830-41b4-b501-d8aacb690731-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4995f2f9-c830-41b4-b501-d8aacb690731-workload-certs\") pod \"4995f2f9-c830-41b4-b501-d8aacb690731\" (UID: \"4995f2f9-c830-41b4-b501-d8aacb690731\") " Nov 01 16:32:29.119948 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.119810 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4995f2f9-c830-41b4-b501-d8aacb690731-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4995f2f9-c830-41b4-b501-d8aacb690731-istio-data\") pod \"4995f2f9-c830-41b4-b501-d8aacb690731\" (UID: \"4995f2f9-c830-41b4-b501-d8aacb690731\") " Nov 01 16:32:29.119948 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.119847 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3655831-2619-45f1-9da3-5bbe7dca87f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3655831-2619-45f1-9da3-5bbe7dca87f4-istiod-ca-cert\") pod \"f3655831-2619-45f1-9da3-5bbe7dca87f4\" (UID: \"f3655831-2619-45f1-9da3-5bbe7dca87f4\") " Nov 01 16:32:29.119948 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.119864 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4995f2f9-c830-41b4-b501-d8aacb690731-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4995f2f9-c830-41b4-b501-d8aacb690731-istio-podinfo\") pod \"4995f2f9-c830-41b4-b501-d8aacb690731\" (UID: \"4995f2f9-c830-41b4-b501-d8aacb690731\") " Nov 01 16:32:29.119948 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.119889 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3655831-2619-45f1-9da3-5bbe7dca87f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3655831-2619-45f1-9da3-5bbe7dca87f4-workload-socket\") pod \"f3655831-2619-45f1-9da3-5bbe7dca87f4\" (UID: \"f3655831-2619-45f1-9da3-5bbe7dca87f4\") " Nov 01 16:32:29.119948 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.119934 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3655831-2619-45f1-9da3-5bbe7dca87f4-kube-api-access-64dcf\" (UniqueName: \"kubernetes.io/projected/f3655831-2619-45f1-9da3-5bbe7dca87f4-kube-api-access-64dcf\") pod \"f3655831-2619-45f1-9da3-5bbe7dca87f4\" (UID: \"f3655831-2619-45f1-9da3-5bbe7dca87f4\") " Nov 01 16:32:29.120287 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.119971 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4995f2f9-c830-41b4-b501-d8aacb690731-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4995f2f9-c830-41b4-b501-d8aacb690731-workload-socket\") pod \"4995f2f9-c830-41b4-b501-d8aacb690731\" (UID: \"4995f2f9-c830-41b4-b501-d8aacb690731\") " Nov 01 16:32:29.120287 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.120003 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4995f2f9-c830-41b4-b501-d8aacb690731-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4995f2f9-c830-41b4-b501-d8aacb690731-credential-socket\") pod \"4995f2f9-c830-41b4-b501-d8aacb690731\" (UID: \"4995f2f9-c830-41b4-b501-d8aacb690731\") " Nov 01 16:32:29.120287 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.120084 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4995f2f9-c830-41b4-b501-d8aacb690731-istio-token\" (UniqueName: \"kubernetes.io/projected/4995f2f9-c830-41b4-b501-d8aacb690731-istio-token\") pod \"4995f2f9-c830-41b4-b501-d8aacb690731\" (UID: \"4995f2f9-c830-41b4-b501-d8aacb690731\") " Nov 01 16:32:29.120287 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.120104 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4995f2f9-c830-41b4-b501-d8aacb690731-istio-data" pod "4995f2f9-c830-41b4-b501-d8aacb690731" (UID: "4995f2f9-c830-41b4-b501-d8aacb690731"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:29.120287 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.120122 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4995f2f9-c830-41b4-b501-d8aacb690731-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4995f2f9-c830-41b4-b501-d8aacb690731-istiod-ca-cert\") pod \"4995f2f9-c830-41b4-b501-d8aacb690731\" (UID: \"4995f2f9-c830-41b4-b501-d8aacb690731\") " Nov 01 16:32:29.120287 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.120193 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3655831-2619-45f1-9da3-5bbe7dca87f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3655831-2619-45f1-9da3-5bbe7dca87f4-credential-socket\") pod \"f3655831-2619-45f1-9da3-5bbe7dca87f4\" (UID: \"f3655831-2619-45f1-9da3-5bbe7dca87f4\") " Nov 01 16:32:29.120287 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.120245 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3655831-2619-45f1-9da3-5bbe7dca87f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3655831-2619-45f1-9da3-5bbe7dca87f4-istio-podinfo\") pod \"f3655831-2619-45f1-9da3-5bbe7dca87f4\" (UID: \"f3655831-2619-45f1-9da3-5bbe7dca87f4\") " Nov 01 16:32:29.120737 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.120319 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4995f2f9-c830-41b4-b501-d8aacb690731-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4995f2f9-c830-41b4-b501-d8aacb690731-istio-envoy\") pod \"4995f2f9-c830-41b4-b501-d8aacb690731\" (UID: \"4995f2f9-c830-41b4-b501-d8aacb690731\") " Nov 01 16:32:29.120737 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.120387 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3655831-2619-45f1-9da3-5bbe7dca87f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3655831-2619-45f1-9da3-5bbe7dca87f4-workload-certs\") pod \"f3655831-2619-45f1-9da3-5bbe7dca87f4\" (UID: \"f3655831-2619-45f1-9da3-5bbe7dca87f4\") " Nov 01 16:32:29.120737 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.120460 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4995f2f9-c830-41b4-b501-d8aacb690731-istiod-ca-cert" pod "4995f2f9-c830-41b4-b501-d8aacb690731" (UID: "4995f2f9-c830-41b4-b501-d8aacb690731"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:32:29.120737 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.120479 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3655831-2619-45f1-9da3-5bbe7dca87f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3655831-2619-45f1-9da3-5bbe7dca87f4-istio-data\") pod \"f3655831-2619-45f1-9da3-5bbe7dca87f4\" (UID: \"f3655831-2619-45f1-9da3-5bbe7dca87f4\") " Nov 01 16:32:29.120737 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.120537 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3655831-2619-45f1-9da3-5bbe7dca87f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3655831-2619-45f1-9da3-5bbe7dca87f4-istio-envoy\") pod \"f3655831-2619-45f1-9da3-5bbe7dca87f4\" (UID: \"f3655831-2619-45f1-9da3-5bbe7dca87f4\") " Nov 01 16:32:29.120737 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.120578 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4995f2f9-c830-41b4-b501-d8aacb690731-kube-api-access-xt5z7\" (UniqueName: \"kubernetes.io/projected/4995f2f9-c830-41b4-b501-d8aacb690731-kube-api-access-xt5z7\") pod \"4995f2f9-c830-41b4-b501-d8aacb690731\" (UID: \"4995f2f9-c830-41b4-b501-d8aacb690731\") " Nov 01 16:32:29.120737 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.120613 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3655831-2619-45f1-9da3-5bbe7dca87f4-istio-token\" (UniqueName: \"kubernetes.io/projected/f3655831-2619-45f1-9da3-5bbe7dca87f4-istio-token\") pod \"f3655831-2619-45f1-9da3-5bbe7dca87f4\" (UID: \"f3655831-2619-45f1-9da3-5bbe7dca87f4\") " Nov 01 16:32:29.120737 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.120721 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4995f2f9-c830-41b4-b501-d8aacb690731-workload-socket" pod "4995f2f9-c830-41b4-b501-d8aacb690731" (UID: "4995f2f9-c830-41b4-b501-d8aacb690731"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:29.121220 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.120902 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4995f2f9-c830-41b4-b501-d8aacb690731-credential-socket" pod "4995f2f9-c830-41b4-b501-d8aacb690731" (UID: "4995f2f9-c830-41b4-b501-d8aacb690731"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:29.121285 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.121262 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3655831-2619-45f1-9da3-5bbe7dca87f4-istiod-ca-cert" pod "f3655831-2619-45f1-9da3-5bbe7dca87f4" (UID: "f3655831-2619-45f1-9da3-5bbe7dca87f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:32:29.121411 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.121385 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4995f2f9-c830-41b4-b501-d8aacb690731-workload-certs" pod "4995f2f9-c830-41b4-b501-d8aacb690731" (UID: "4995f2f9-c830-41b4-b501-d8aacb690731"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:29.121625 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.121604 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4995f2f9-c830-41b4-b501-d8aacb690731-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:29.121694 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.121632 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3655831-2619-45f1-9da3-5bbe7dca87f4-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:29.121694 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.121645 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4995f2f9-c830-41b4-b501-d8aacb690731-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:29.121694 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.121658 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4995f2f9-c830-41b4-b501-d8aacb690731-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:29.121694 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.121671 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4995f2f9-c830-41b4-b501-d8aacb690731-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:29.122401 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.122162 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3655831-2619-45f1-9da3-5bbe7dca87f4-credential-socket" pod "f3655831-2619-45f1-9da3-5bbe7dca87f4" (UID: "f3655831-2619-45f1-9da3-5bbe7dca87f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:29.122698 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.122680 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3655831-2619-45f1-9da3-5bbe7dca87f4-workload-socket" pod "f3655831-2619-45f1-9da3-5bbe7dca87f4" (UID: "f3655831-2619-45f1-9da3-5bbe7dca87f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:29.123191 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.123170 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3655831-2619-45f1-9da3-5bbe7dca87f4-istio-data" pod "f3655831-2619-45f1-9da3-5bbe7dca87f4" (UID: "f3655831-2619-45f1-9da3-5bbe7dca87f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:29.127650 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.127409 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4995f2f9-c830-41b4-b501-d8aacb690731-kube-api-access-xt5z7" pod "4995f2f9-c830-41b4-b501-d8aacb690731" (UID: "4995f2f9-c830-41b4-b501-d8aacb690731"). InnerVolumeSpecName "kube-api-access-xt5z7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:29.127650 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.127497 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4995f2f9-c830-41b4-b501-d8aacb690731-istio-token" pod "4995f2f9-c830-41b4-b501-d8aacb690731" (UID: "4995f2f9-c830-41b4-b501-d8aacb690731"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:29.127835 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.127708 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3655831-2619-45f1-9da3-5bbe7dca87f4-workload-certs" pod "f3655831-2619-45f1-9da3-5bbe7dca87f4" (UID: "f3655831-2619-45f1-9da3-5bbe7dca87f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:29.128992 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.128969 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3655831-2619-45f1-9da3-5bbe7dca87f4-istio-token" pod "f3655831-2619-45f1-9da3-5bbe7dca87f4" (UID: "f3655831-2619-45f1-9da3-5bbe7dca87f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:29.129126 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.129098 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3655831-2619-45f1-9da3-5bbe7dca87f4-istio-podinfo" pod "f3655831-2619-45f1-9da3-5bbe7dca87f4" (UID: "f3655831-2619-45f1-9da3-5bbe7dca87f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:32:29.129198 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.129186 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4995f2f9-c830-41b4-b501-d8aacb690731-istio-podinfo" pod "4995f2f9-c830-41b4-b501-d8aacb690731" (UID: "4995f2f9-c830-41b4-b501-d8aacb690731"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:32:29.129450 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.129423 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4995f2f9-c830-41b4-b501-d8aacb690731-istio-envoy" pod "4995f2f9-c830-41b4-b501-d8aacb690731" (UID: "4995f2f9-c830-41b4-b501-d8aacb690731"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:29.129698 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.129672 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3655831-2619-45f1-9da3-5bbe7dca87f4-istio-envoy" pod "f3655831-2619-45f1-9da3-5bbe7dca87f4" (UID: "f3655831-2619-45f1-9da3-5bbe7dca87f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:29.131600 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.131564 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3655831-2619-45f1-9da3-5bbe7dca87f4-kube-api-access-64dcf" pod "f3655831-2619-45f1-9da3-5bbe7dca87f4" (UID: "f3655831-2619-45f1-9da3-5bbe7dca87f4"). InnerVolumeSpecName "kube-api-access-64dcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:29.200881 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.200832 4363 generic.go:356] "Generic (PLEG): container finished" podID="f3655831-2619-45f1-9da3-5bbe7dca87f4" containerID="82772b8f1c16ec6807099169d4a16b294ecc89b48490df7a20150da84c0d3beb" exitCode=0 Nov 01 16:32:29.201096 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.200889 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" Nov 01 16:32:29.201096 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.200922 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" event={"ID":"f3655831-2619-45f1-9da3-5bbe7dca87f4","Type":"ContainerDied","Data":"82772b8f1c16ec6807099169d4a16b294ecc89b48490df7a20150da84c0d3beb"} Nov 01 16:32:29.201096 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.200990 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl" event={"ID":"f3655831-2619-45f1-9da3-5bbe7dca87f4","Type":"ContainerDied","Data":"42bbde4f019d77e774ed377fb56efa8906f57dfb498a4ef45c5ae967729ab5e3"} Nov 01 16:32:29.201096 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.201005 4363 scope.go:122] "RemoveContainer" containerID="82772b8f1c16ec6807099169d4a16b294ecc89b48490df7a20150da84c0d3beb" Nov 01 16:32:29.203069 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.202707 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-43fac8-ff59b5dfb-r8zjf" Nov 01 16:32:29.203069 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.202728 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-43fac8-ff59b5dfb-r8zjf" event={"ID":"6be52ed0-203a-4a91-abee-9420e0dff7dd","Type":"ContainerDied","Data":"f99d66d888075aa206496af3a504e2d73c1c3be25fff2a348a259e2dfec3b17a"} Nov 01 16:32:29.205129 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.204520 4363 generic.go:356] "Generic (PLEG): container finished" podID="4995f2f9-c830-41b4-b501-d8aacb690731" containerID="375710f4d1c2055c11f357a67a5833fa64df78c2deea66dac61991b8484b52ed" exitCode=0 Nov 01 16:32:29.205129 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.204682 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" event={"ID":"4995f2f9-c830-41b4-b501-d8aacb690731","Type":"ContainerDied","Data":"375710f4d1c2055c11f357a67a5833fa64df78c2deea66dac61991b8484b52ed"} Nov 01 16:32:29.205129 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.204722 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" event={"ID":"4995f2f9-c830-41b4-b501-d8aacb690731","Type":"ContainerDied","Data":"d49e4640d2068b20cab3099b6b8357858f47ca66ccce90fed993fa73944c21ba"} Nov 01 16:32:29.205129 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.204792 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r" Nov 01 16:32:29.215187 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.215168 4363 scope.go:122] "RemoveContainer" containerID="1c053fc029571cf2587d21ea99f73ad2fdc6c84e9249f49578275898ea8d521e" Nov 01 16:32:29.223948 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.223604 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3655831-2619-45f1-9da3-5bbe7dca87f4-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:29.224154 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.223961 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt5z7\" (UniqueName: \"kubernetes.io/projected/4995f2f9-c830-41b4-b501-d8aacb690731-kube-api-access-xt5z7\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:29.224154 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.223979 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3655831-2619-45f1-9da3-5bbe7dca87f4-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:29.224154 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.223995 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4995f2f9-c830-41b4-b501-d8aacb690731-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:29.224342 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.224009 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4995f2f9-c830-41b4-b501-d8aacb690731-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:29.224342 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.224249 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3655831-2619-45f1-9da3-5bbe7dca87f4-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:29.224342 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.224264 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64dcf\" (UniqueName: \"kubernetes.io/projected/f3655831-2619-45f1-9da3-5bbe7dca87f4-kube-api-access-64dcf\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:29.224342 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.224280 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4995f2f9-c830-41b4-b501-d8aacb690731-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:29.224342 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.224294 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3655831-2619-45f1-9da3-5bbe7dca87f4-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:29.224342 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.224307 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3655831-2619-45f1-9da3-5bbe7dca87f4-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:29.224342 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.224320 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4995f2f9-c830-41b4-b501-d8aacb690731-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:29.224342 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.224333 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3655831-2619-45f1-9da3-5bbe7dca87f4-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:29.224765 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.224346 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3655831-2619-45f1-9da3-5bbe7dca87f4-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:29.227774 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.227757 4363 scope.go:122] "RemoveContainer" containerID="82772b8f1c16ec6807099169d4a16b294ecc89b48490df7a20150da84c0d3beb" Nov 01 16:32:29.227944 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.227920 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-43fac8-ff59b5dfb-r8zjf"] Nov 01 16:32:29.228149 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:32:29.228131 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82772b8f1c16ec6807099169d4a16b294ecc89b48490df7a20150da84c0d3beb\": container with ID starting with 82772b8f1c16ec6807099169d4a16b294ecc89b48490df7a20150da84c0d3beb not found: ID does not exist" containerID="82772b8f1c16ec6807099169d4a16b294ecc89b48490df7a20150da84c0d3beb" Nov 01 16:32:29.228242 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.228160 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82772b8f1c16ec6807099169d4a16b294ecc89b48490df7a20150da84c0d3beb"} err="failed to get container status \"82772b8f1c16ec6807099169d4a16b294ecc89b48490df7a20150da84c0d3beb\": rpc error: code = NotFound desc = could not find container \"82772b8f1c16ec6807099169d4a16b294ecc89b48490df7a20150da84c0d3beb\": container with ID starting with 82772b8f1c16ec6807099169d4a16b294ecc89b48490df7a20150da84c0d3beb not found: ID does not exist" Nov 01 16:32:29.228242 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.228181 4363 scope.go:122] "RemoveContainer" containerID="1c053fc029571cf2587d21ea99f73ad2fdc6c84e9249f49578275898ea8d521e" Nov 01 16:32:29.228470 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:32:29.228454 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c053fc029571cf2587d21ea99f73ad2fdc6c84e9249f49578275898ea8d521e\": container with ID starting with 1c053fc029571cf2587d21ea99f73ad2fdc6c84e9249f49578275898ea8d521e not found: ID does not exist" containerID="1c053fc029571cf2587d21ea99f73ad2fdc6c84e9249f49578275898ea8d521e" Nov 01 16:32:29.228524 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.228478 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c053fc029571cf2587d21ea99f73ad2fdc6c84e9249f49578275898ea8d521e"} err="failed to get container status \"1c053fc029571cf2587d21ea99f73ad2fdc6c84e9249f49578275898ea8d521e\": rpc error: code = NotFound desc = could not find container \"1c053fc029571cf2587d21ea99f73ad2fdc6c84e9249f49578275898ea8d521e\": container with ID starting with 1c053fc029571cf2587d21ea99f73ad2fdc6c84e9249f49578275898ea8d521e not found: ID does not exist" Nov 01 16:32:29.228524 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.228494 4363 scope.go:122] "RemoveContainer" containerID="f61adfd659bef00b44fed69eb071f6d14300a9b8e907db26172a4c7b964273db" Nov 01 16:32:29.234833 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.234812 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-43fac8-ff59b5dfb-r8zjf"] Nov 01 16:32:29.238091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.238073 4363 scope.go:122] "RemoveContainer" containerID="375710f4d1c2055c11f357a67a5833fa64df78c2deea66dac61991b8484b52ed" Nov 01 16:32:29.252987 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.252962 4363 scope.go:122] "RemoveContainer" containerID="375710f4d1c2055c11f357a67a5833fa64df78c2deea66dac61991b8484b52ed" Nov 01 16:32:29.253125 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.253078 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl"] Nov 01 16:32:29.253681 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:32:29.253649 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"375710f4d1c2055c11f357a67a5833fa64df78c2deea66dac61991b8484b52ed\": container with ID starting with 375710f4d1c2055c11f357a67a5833fa64df78c2deea66dac61991b8484b52ed not found: ID does not exist" containerID="375710f4d1c2055c11f357a67a5833fa64df78c2deea66dac61991b8484b52ed" Nov 01 16:32:29.253768 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.253691 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"375710f4d1c2055c11f357a67a5833fa64df78c2deea66dac61991b8484b52ed"} err="failed to get container status \"375710f4d1c2055c11f357a67a5833fa64df78c2deea66dac61991b8484b52ed\": rpc error: code = NotFound desc = could not find container \"375710f4d1c2055c11f357a67a5833fa64df78c2deea66dac61991b8484b52ed\": container with ID starting with 375710f4d1c2055c11f357a67a5833fa64df78c2deea66dac61991b8484b52ed not found: ID does not exist" Nov 01 16:32:29.253829 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.253770 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-ckkbl"] Nov 01 16:32:29.270228 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.270169 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r"] Nov 01 16:32:29.273433 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:29.273413 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-p8v9r"] Nov 01 16:32:30.512786 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:30.512747 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-43fac8-84b47867b7-4gzrb" podUID="bb1a7676-ecc1-451e-ba91-83f91fdf1bed" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:32:30.806088 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:30.805547 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4995f2f9-c830-41b4-b501-d8aacb690731" path="/var/lib/kubelet/pods/4995f2f9-c830-41b4-b501-d8aacb690731/volumes" Nov 01 16:32:30.811310 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:30.807456 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6be52ed0-203a-4a91-abee-9420e0dff7dd" path="/var/lib/kubelet/pods/6be52ed0-203a-4a91-abee-9420e0dff7dd/volumes" Nov 01 16:32:30.811310 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:30.808096 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3655831-2619-45f1-9da3-5bbe7dca87f4" path="/var/lib/kubelet/pods/f3655831-2619-45f1-9da3-5bbe7dca87f4/volumes" Nov 01 16:32:32.838417 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:32.838378 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd"] Nov 01 16:32:32.838987 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:32.838655 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" podUID="cf9b2de9-f52e-4580-af9e-d77e06ee6e64" containerName="istio-proxy" containerID="cri-o://9012a4038cbd6b621e94b09b6f15ba48bac9d607395e4dd1937895ee00ecd6e2" gracePeriod=30 Nov 01 16:32:32.851006 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:32.850976 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr"] Nov 01 16:32:32.851255 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:32.851224 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" podUID="635f1ba7-95b4-4c9c-9dc4-106eca4a54ad" containerName="istio-proxy" containerID="cri-o://ccd7813e01b568a03a63370557d80a54b0ed2c782f593853761de6b9f65d0df8" gracePeriod=30 Nov 01 16:32:33.286844 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:33.286810 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-734859-7lhdq"] Nov 01 16:32:33.287513 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:33.287497 4363 cpu_manager.go:414] "containerMap: removing container" podUID="f3655831-2619-45f1-9da3-5bbe7dca87f4" containerName="istio-proxy" Nov 01 16:32:33.287574 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:33.287516 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3655831-2619-45f1-9da3-5bbe7dca87f4" containerName="istio-proxy" Nov 01 16:32:33.287574 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:33.287527 4363 cpu_manager.go:414] "containerMap: removing container" podUID="c033e4f6-d864-4579-9793-037ec810172d" containerName="payload-processing" Nov 01 16:32:33.287574 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:33.287533 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c033e4f6-d864-4579-9793-037ec810172d" containerName="payload-processing" Nov 01 16:32:33.287574 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:33.287549 4363 cpu_manager.go:414] "containerMap: removing container" podUID="f3655831-2619-45f1-9da3-5bbe7dca87f4" containerName="istio-proxy" Nov 01 16:32:33.287574 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:33.287554 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3655831-2619-45f1-9da3-5bbe7dca87f4" containerName="istio-proxy" Nov 01 16:32:33.287574 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:33.287564 4363 cpu_manager.go:414] "containerMap: removing container" podUID="bf475632-fcc1-4d2d-a98b-65464c14f9b7" containerName="revoke-keys" Nov 01 16:32:33.287574 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:33.287569 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf475632-fcc1-4d2d-a98b-65464c14f9b7" containerName="revoke-keys" Nov 01 16:32:33.287790 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:33.287580 4363 cpu_manager.go:414] "containerMap: removing container" podUID="4995f2f9-c830-41b4-b501-d8aacb690731" containerName="istio-proxy" Nov 01 16:32:33.287790 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:33.287585 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4995f2f9-c830-41b4-b501-d8aacb690731" containerName="istio-proxy" Nov 01 16:32:33.287790 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:33.287613 4363 cpu_manager.go:414] "containerMap: removing container" podUID="6be52ed0-203a-4a91-abee-9420e0dff7dd" containerName="maas-api" Nov 01 16:32:33.287790 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:33.287617 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6be52ed0-203a-4a91-abee-9420e0dff7dd" containerName="maas-api" Nov 01 16:32:33.287790 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:33.287624 4363 cpu_manager.go:414] "containerMap: removing container" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" Nov 01 16:32:33.287790 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:33.287628 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" Nov 01 16:32:33.287790 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:33.287722 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="6be52ed0-203a-4a91-abee-9420e0dff7dd" containerName="maas-api" Nov 01 16:32:33.287790 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:33.287731 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="e24b88bf-ca80-4cff-ad74-5e3a69a5b7d4" containerName="main" Nov 01 16:32:33.287790 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:33.287739 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3655831-2619-45f1-9da3-5bbe7dca87f4" containerName="istio-proxy" Nov 01 16:32:33.287790 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:33.287746 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="c033e4f6-d864-4579-9793-037ec810172d" containerName="payload-processing" Nov 01 16:32:33.287790 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:33.287755 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf475632-fcc1-4d2d-a98b-65464c14f9b7" containerName="revoke-keys" Nov 01 16:32:33.287790 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:33.287762 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="4995f2f9-c830-41b4-b501-d8aacb690731" containerName="istio-proxy" Nov 01 16:32:33.292904 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:33.292887 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-734859-7lhdq" Nov 01 16:32:33.300150 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:33.299791 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-734859-7lhdq"] Nov 01 16:32:33.365042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:33.364230 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:32:33.367454 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:33.365424 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:32:33.384489 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:33.383939 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrfvc\" (UniqueName: \"kubernetes.io/projected/2cc4aea6-d369-49de-89f5-73ab72ae0d5b-kube-api-access-zrfvc\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-734859-7lhdq\" (UID: \"2cc4aea6-d369-49de-89f5-73ab72ae0d5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-734859-7lhdq" Nov 01 16:32:33.384489 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:33.383997 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cc4aea6-d369-49de-89f5-73ab72ae0d5b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-734859-7lhdq\" (UID: \"2cc4aea6-d369-49de-89f5-73ab72ae0d5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-734859-7lhdq" Nov 01 16:32:33.384489 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:33.384224 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cc4aea6-d369-49de-89f5-73ab72ae0d5b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-734859-7lhdq\" (UID: \"2cc4aea6-d369-49de-89f5-73ab72ae0d5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-734859-7lhdq" Nov 01 16:32:33.485900 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:33.485866 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrfvc\" (UniqueName: \"kubernetes.io/projected/2cc4aea6-d369-49de-89f5-73ab72ae0d5b-kube-api-access-zrfvc\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-734859-7lhdq\" (UID: \"2cc4aea6-d369-49de-89f5-73ab72ae0d5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-734859-7lhdq" Nov 01 16:32:33.485900 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:33.485904 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cc4aea6-d369-49de-89f5-73ab72ae0d5b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-734859-7lhdq\" (UID: \"2cc4aea6-d369-49de-89f5-73ab72ae0d5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-734859-7lhdq" Nov 01 16:32:33.486150 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:33.486026 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cc4aea6-d369-49de-89f5-73ab72ae0d5b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-734859-7lhdq\" (UID: \"2cc4aea6-d369-49de-89f5-73ab72ae0d5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-734859-7lhdq" Nov 01 16:32:33.486729 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:33.486704 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cc4aea6-d369-49de-89f5-73ab72ae0d5b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-734859-7lhdq\" (UID: \"2cc4aea6-d369-49de-89f5-73ab72ae0d5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-734859-7lhdq" Nov 01 16:32:33.488495 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:33.488472 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cc4aea6-d369-49de-89f5-73ab72ae0d5b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-734859-7lhdq\" (UID: \"2cc4aea6-d369-49de-89f5-73ab72ae0d5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-734859-7lhdq" Nov 01 16:32:33.493316 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:33.493293 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrfvc\" (UniqueName: \"kubernetes.io/projected/2cc4aea6-d369-49de-89f5-73ab72ae0d5b-kube-api-access-zrfvc\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-734859-7lhdq\" (UID: \"2cc4aea6-d369-49de-89f5-73ab72ae0d5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-734859-7lhdq" Nov 01 16:32:33.614927 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:33.614893 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-734859-7lhdq" Nov 01 16:32:33.909675 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:33.909596 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-734859-7lhdq"] Nov 01 16:32:34.266583 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:34.266548 4363 generic.go:356] "Generic (PLEG): container finished" podID="bb1a7676-ecc1-451e-ba91-83f91fdf1bed" containerID="2e2108f9def8588cd4c86249d485464e54f2df8836971eb7c1a8b872c24ab362" exitCode=0 Nov 01 16:32:34.266740 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:34.266627 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-43fac8-84b47867b7-4gzrb" event={"ID":"bb1a7676-ecc1-451e-ba91-83f91fdf1bed","Type":"ContainerDied","Data":"2e2108f9def8588cd4c86249d485464e54f2df8836971eb7c1a8b872c24ab362"} Nov 01 16:32:34.272121 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:34.272071 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-734859-7lhdq" event={"ID":"2cc4aea6-d369-49de-89f5-73ab72ae0d5b","Type":"ContainerStarted","Data":"c83d66f9e08f683a80df6a991d434b0e18ae5dc6f7d1fd6370ad1ae2976024d0"} Nov 01 16:32:34.272278 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:34.272137 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-734859-7lhdq" event={"ID":"2cc4aea6-d369-49de-89f5-73ab72ae0d5b","Type":"ContainerStarted","Data":"c1f1056e1573fdeb47c44787034855970e9b97fdbc08635828496227f52ac4f3"} Nov 01 16:32:34.294521 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:34.294467 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-734859-7lhdq" podStartSLOduration=1.2944488760000001 podStartE2EDuration="1.294448876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:32:34.289263625 +0000 UTC m=+2687.621001298" watchObservedRunningTime="2026-11-01 16:32:34.294448876 +0000 UTC m=+2687.626186526" Nov 01 16:32:34.403135 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:34.403106 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-43fac8-84b47867b7-4gzrb" Nov 01 16:32:34.505719 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:34.505693 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb1a7676-ecc1-451e-ba91-83f91fdf1bed-kube-api-access-55hbk\" (UniqueName: \"kubernetes.io/projected/bb1a7676-ecc1-451e-ba91-83f91fdf1bed-kube-api-access-55hbk\") pod \"bb1a7676-ecc1-451e-ba91-83f91fdf1bed\" (UID: \"bb1a7676-ecc1-451e-ba91-83f91fdf1bed\") " Nov 01 16:32:34.505865 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:34.505810 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb1a7676-ecc1-451e-ba91-83f91fdf1bed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb1a7676-ecc1-451e-ba91-83f91fdf1bed-plugins-config-volume\") pod \"bb1a7676-ecc1-451e-ba91-83f91fdf1bed\" (UID: \"bb1a7676-ecc1-451e-ba91-83f91fdf1bed\") " Nov 01 16:32:34.506178 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:34.506154 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb1a7676-ecc1-451e-ba91-83f91fdf1bed-plugins-config-volume" pod "bb1a7676-ecc1-451e-ba91-83f91fdf1bed" (UID: "bb1a7676-ecc1-451e-ba91-83f91fdf1bed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:32:34.506379 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:34.506365 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb1a7676-ecc1-451e-ba91-83f91fdf1bed-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:34.507883 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:34.507862 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb1a7676-ecc1-451e-ba91-83f91fdf1bed-kube-api-access-55hbk" pod "bb1a7676-ecc1-451e-ba91-83f91fdf1bed" (UID: "bb1a7676-ecc1-451e-ba91-83f91fdf1bed"). InnerVolumeSpecName "kube-api-access-55hbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:34.611874 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:34.611838 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55hbk\" (UniqueName: \"kubernetes.io/projected/bb1a7676-ecc1-451e-ba91-83f91fdf1bed-kube-api-access-55hbk\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:34.897487 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:34.897423 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b"] Nov 01 16:32:34.898477 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:34.898455 4363 cpu_manager.go:414] "containerMap: removing container" podUID="bb1a7676-ecc1-451e-ba91-83f91fdf1bed" containerName="payload-pre-processing" Nov 01 16:32:34.898588 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:34.898479 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb1a7676-ecc1-451e-ba91-83f91fdf1bed" containerName="payload-pre-processing" Nov 01 16:32:34.898695 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:34.898681 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb1a7676-ecc1-451e-ba91-83f91fdf1bed" containerName="payload-pre-processing" Nov 01 16:32:34.898775 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:34.898707 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3655831-2619-45f1-9da3-5bbe7dca87f4" containerName="istio-proxy" Nov 01 16:32:34.903624 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:34.903582 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" Nov 01 16:32:34.925057 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:34.922090 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-workload-socket\") pod \"e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b\" (UID: \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\") " pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" Nov 01 16:32:34.926391 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:34.925803 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-workload-certs\") pod \"e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b\" (UID: \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\") " pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" Nov 01 16:32:34.926391 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:34.925863 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-credential-socket\") pod \"e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b\" (UID: \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\") " pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" Nov 01 16:32:34.926391 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:34.925903 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-istiod-ca-cert\") pod \"e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b\" (UID: \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\") " pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" Nov 01 16:32:34.926391 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:34.925940 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-istio-data\") pod \"e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b\" (UID: \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\") " pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" Nov 01 16:32:34.926391 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:34.925966 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-istio-envoy\") pod \"e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b\" (UID: \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\") " pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" Nov 01 16:32:34.926391 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:34.926070 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-istio-podinfo\") pod \"e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b\" (UID: \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\") " pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" Nov 01 16:32:34.926391 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:34.926101 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp7zc\" (UniqueName: \"kubernetes.io/projected/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-kube-api-access-pp7zc\") pod \"e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b\" (UID: \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\") " pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" Nov 01 16:32:34.926391 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:34.926159 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-istio-token\") pod \"e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b\" (UID: \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\") " pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" Nov 01 16:32:34.935450 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:34.935428 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-501d63-openshift-default-dockercfg-4nzlf\"" Nov 01 16:32:34.935989 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:34.935963 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b"] Nov 01 16:32:35.027548 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:35.027514 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-workload-certs\") pod \"e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b\" (UID: \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\") " pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" Nov 01 16:32:35.027718 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:35.027587 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-credential-socket\") pod \"e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b\" (UID: \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\") " pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" Nov 01 16:32:35.027718 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:35.027635 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-istiod-ca-cert\") pod \"e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b\" (UID: \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\") " pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" Nov 01 16:32:35.027718 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:35.027671 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-istio-data\") pod \"e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b\" (UID: \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\") " pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" Nov 01 16:32:35.027718 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:35.027697 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-istio-envoy\") pod \"e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b\" (UID: \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\") " pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" Nov 01 16:32:35.027982 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:35.027783 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-istio-podinfo\") pod \"e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b\" (UID: \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\") " pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" Nov 01 16:32:35.027982 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:35.027813 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp7zc\" (UniqueName: \"kubernetes.io/projected/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-kube-api-access-pp7zc\") pod \"e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b\" (UID: \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\") " pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" Nov 01 16:32:35.027982 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:35.027873 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-istio-token\") pod \"e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b\" (UID: \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\") " pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" Nov 01 16:32:35.027982 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:35.027942 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-workload-certs\") pod \"e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b\" (UID: \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\") " pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" Nov 01 16:32:35.028253 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:35.027986 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-workload-socket\") pod \"e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b\" (UID: \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\") " pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" Nov 01 16:32:35.028253 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:35.027989 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-credential-socket\") pod \"e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b\" (UID: \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\") " pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" Nov 01 16:32:35.028253 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:35.028080 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-istio-data\") pod \"e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b\" (UID: \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\") " pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" Nov 01 16:32:35.028375 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:35.028266 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-workload-socket\") pod \"e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b\" (UID: \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\") " pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" Nov 01 16:32:35.028412 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:35.028397 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-istiod-ca-cert\") pod \"e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b\" (UID: \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\") " pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" Nov 01 16:32:35.030333 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:35.030303 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-istio-envoy\") pod \"e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b\" (UID: \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\") " pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" Nov 01 16:32:35.030751 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:35.030732 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-istio-podinfo\") pod \"e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b\" (UID: \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\") " pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" Nov 01 16:32:35.037002 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:35.036977 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-istio-token\") pod \"e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b\" (UID: \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\") " pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" Nov 01 16:32:35.037201 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:35.037183 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp7zc\" (UniqueName: \"kubernetes.io/projected/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-kube-api-access-pp7zc\") pod \"e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b\" (UID: \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\") " pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" Nov 01 16:32:35.268040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:35.266905 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" Nov 01 16:32:35.299238 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:35.299202 4363 generic.go:356] "Generic (PLEG): container finished" podID="2cc4aea6-d369-49de-89f5-73ab72ae0d5b" containerID="c83d66f9e08f683a80df6a991d434b0e18ae5dc6f7d1fd6370ad1ae2976024d0" exitCode=0 Nov 01 16:32:35.299438 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:35.299408 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-734859-7lhdq" event={"ID":"2cc4aea6-d369-49de-89f5-73ab72ae0d5b","Type":"ContainerDied","Data":"c83d66f9e08f683a80df6a991d434b0e18ae5dc6f7d1fd6370ad1ae2976024d0"} Nov 01 16:32:35.302567 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:35.302534 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-43fac8-84b47867b7-4gzrb" event={"ID":"bb1a7676-ecc1-451e-ba91-83f91fdf1bed","Type":"ContainerDied","Data":"a967126e78bb6239e89bc977ef99c3fb11cb0c57052311b4ceeddcea4c5e751f"} Nov 01 16:32:35.302816 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:35.302791 4363 scope.go:122] "RemoveContainer" containerID="2e2108f9def8588cd4c86249d485464e54f2df8836971eb7c1a8b872c24ab362" Nov 01 16:32:35.303039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:35.302999 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-43fac8-84b47867b7-4gzrb" Nov 01 16:32:35.335463 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:35.335434 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-43fac8-84b47867b7-4gzrb"] Nov 01 16:32:35.343107 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:35.343074 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-43fac8-84b47867b7-4gzrb"] Nov 01 16:32:35.595386 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:35.595355 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b"] Nov 01 16:32:35.597421 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:32:35.597390 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2b69d5a_3eef_4aaf_a3ad_454ac3dd6697.slice/crio-b49f14168080113075111ee267d67394c7e52446585f8f045194cec6d042011d WatchSource:0}: Error finding container b49f14168080113075111ee267d67394c7e52446585f8f045194cec6d042011d: Status 404 returned error can't find the container with id b49f14168080113075111ee267d67394c7e52446585f8f045194cec6d042011d Nov 01 16:32:35.600571 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:35.600525 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:32:35.600686 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:35.600648 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:32:35.600728 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:35.600696 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:32:35.819918 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:35.818906 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs"] Nov 01 16:32:35.819918 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:35.819159 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" podUID="ff3d8641-74e7-496e-8b67-fd5db4a94e8e" containerName="istio-proxy" containerID="cri-o://047032a99321f97e085c9eaf28c06b2be9413fcee7a10ac837a0d72e404b5659" gracePeriod=30 Nov 01 16:32:36.043561 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:36.043520 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a101f7e1-7r2hq"] Nov 01 16:32:36.052378 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:36.051909 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a101f7e1-7r2hq"] Nov 01 16:32:36.318192 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:36.318093 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" event={"ID":"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697","Type":"ContainerStarted","Data":"11d85b9dc8a77601e7240682f3127f7464c911c041076576559958705ce20c57"} Nov 01 16:32:36.318192 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:36.318147 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" event={"ID":"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697","Type":"ContainerStarted","Data":"b49f14168080113075111ee267d67394c7e52446585f8f045194cec6d042011d"} Nov 01 16:32:36.338039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:36.337661 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" podStartSLOduration=2.337642332 podStartE2EDuration="2.337642332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:32:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:32:36.337109435 +0000 UTC m=+2689.668847085" watchObservedRunningTime="2026-11-01 16:32:36.337642332 +0000 UTC m=+2689.669379983" Nov 01 16:32:36.348202 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:36.348165 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f47890-84fdfb9c59-fs5nn" podUID="18faa082-6418-4665-b862-09efa76dfcc0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:32:36.570239 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:36.570180 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-734859-7lhdq" Nov 01 16:32:36.656348 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:36.656311 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cc4aea6-d369-49de-89f5-73ab72ae0d5b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cc4aea6-d369-49de-89f5-73ab72ae0d5b-maas-api-openshift-ca\") pod \"2cc4aea6-d369-49de-89f5-73ab72ae0d5b\" (UID: \"2cc4aea6-d369-49de-89f5-73ab72ae0d5b\") " Nov 01 16:32:36.656523 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:36.656440 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cc4aea6-d369-49de-89f5-73ab72ae0d5b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cc4aea6-d369-49de-89f5-73ab72ae0d5b-maas-api-xks-ca\") pod \"2cc4aea6-d369-49de-89f5-73ab72ae0d5b\" (UID: \"2cc4aea6-d369-49de-89f5-73ab72ae0d5b\") " Nov 01 16:32:36.656523 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:36.656510 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cc4aea6-d369-49de-89f5-73ab72ae0d5b-kube-api-access-zrfvc\" (UniqueName: \"kubernetes.io/projected/2cc4aea6-d369-49de-89f5-73ab72ae0d5b-kube-api-access-zrfvc\") pod \"2cc4aea6-d369-49de-89f5-73ab72ae0d5b\" (UID: \"2cc4aea6-d369-49de-89f5-73ab72ae0d5b\") " Nov 01 16:32:36.656712 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:36.656676 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cc4aea6-d369-49de-89f5-73ab72ae0d5b-maas-api-openshift-ca" pod "2cc4aea6-d369-49de-89f5-73ab72ae0d5b" (UID: "2cc4aea6-d369-49de-89f5-73ab72ae0d5b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:32:36.657078 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:36.657057 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cc4aea6-d369-49de-89f5-73ab72ae0d5b-maas-api-openshift-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:36.658816 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:36.658789 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cc4aea6-d369-49de-89f5-73ab72ae0d5b-kube-api-access-zrfvc" pod "2cc4aea6-d369-49de-89f5-73ab72ae0d5b" (UID: "2cc4aea6-d369-49de-89f5-73ab72ae0d5b"). InnerVolumeSpecName "kube-api-access-zrfvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:36.658897 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:36.658825 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cc4aea6-d369-49de-89f5-73ab72ae0d5b-maas-api-xks-ca" pod "2cc4aea6-d369-49de-89f5-73ab72ae0d5b" (UID: "2cc4aea6-d369-49de-89f5-73ab72ae0d5b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:32:36.758768 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:36.758740 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cc4aea6-d369-49de-89f5-73ab72ae0d5b-maas-api-xks-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:36.758768 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:36.758767 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrfvc\" (UniqueName: \"kubernetes.io/projected/2cc4aea6-d369-49de-89f5-73ab72ae0d5b-kube-api-access-zrfvc\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:36.797032 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:36.796988 4363 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:32:36.797207 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:36.797063 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" podUID="ff3d8641-74e7-496e-8b67-fd5db4a94e8e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:32:36.800549 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:36.800524 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ace069d-b1a6-4ca9-b630-2828fdd25be2" path="/var/lib/kubelet/pods/1ace069d-b1a6-4ca9-b630-2828fdd25be2/volumes" Nov 01 16:32:36.801061 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:36.801042 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb1a7676-ecc1-451e-ba91-83f91fdf1bed" path="/var/lib/kubelet/pods/bb1a7676-ecc1-451e-ba91-83f91fdf1bed/volumes" Nov 01 16:32:37.267495 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:37.267454 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" Nov 01 16:32:37.275222 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:37.275194 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" Nov 01 16:32:37.326970 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:37.326940 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-734859-7lhdq" Nov 01 16:32:37.327174 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:37.326962 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-734859-7lhdq" event={"ID":"2cc4aea6-d369-49de-89f5-73ab72ae0d5b","Type":"ContainerDied","Data":"c1f1056e1573fdeb47c44787034855970e9b97fdbc08635828496227f52ac4f3"} Nov 01 16:32:37.327174 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:37.327035 4363 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1f1056e1573fdeb47c44787034855970e9b97fdbc08635828496227f52ac4f3" Nov 01 16:32:37.331144 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:37.327660 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" Nov 01 16:32:37.342094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:37.332047 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" Nov 01 16:32:38.150862 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.150836 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" Nov 01 16:32:38.181520 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.181488 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-workload-socket\") pod \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\" (UID: \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\") " Nov 01 16:32:38.181731 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.181545 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-kube-api-access-fgqsk\" (UniqueName: \"kubernetes.io/projected/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-kube-api-access-fgqsk\") pod \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\" (UID: \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\") " Nov 01 16:32:38.181731 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.181588 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-istio-envoy\") pod \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\" (UID: \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\") " Nov 01 16:32:38.181731 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.181610 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-istio-data\") pod \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\" (UID: \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\") " Nov 01 16:32:38.181731 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.181652 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-istiod-ca-cert\") pod \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\" (UID: \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\") " Nov 01 16:32:38.181731 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.181701 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-workload-certs\") pod \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\" (UID: \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\") " Nov 01 16:32:38.182052 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.181997 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-istio-podinfo\") pod \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\" (UID: \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\") " Nov 01 16:32:38.182120 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.182105 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-credential-socket\") pod \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\" (UID: \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\") " Nov 01 16:32:38.182213 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.182197 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-istio-token\" (UniqueName: \"kubernetes.io/projected/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-istio-token\") pod \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\" (UID: \"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad\") " Nov 01 16:32:38.183763 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.183576 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-istio-data" pod "635f1ba7-95b4-4c9c-9dc4-106eca4a54ad" (UID: "635f1ba7-95b4-4c9c-9dc4-106eca4a54ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:38.183763 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.183688 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-istiod-ca-cert" pod "635f1ba7-95b4-4c9c-9dc4-106eca4a54ad" (UID: "635f1ba7-95b4-4c9c-9dc4-106eca4a54ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:32:38.184033 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.183987 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-credential-socket" pod "635f1ba7-95b4-4c9c-9dc4-106eca4a54ad" (UID: "635f1ba7-95b4-4c9c-9dc4-106eca4a54ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:38.189405 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.184239 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-workload-certs" pod "635f1ba7-95b4-4c9c-9dc4-106eca4a54ad" (UID: "635f1ba7-95b4-4c9c-9dc4-106eca4a54ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:38.192331 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.192287 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-istio-podinfo" pod "635f1ba7-95b4-4c9c-9dc4-106eca4a54ad" (UID: "635f1ba7-95b4-4c9c-9dc4-106eca4a54ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:32:38.198198 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.198105 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-istio-envoy" pod "635f1ba7-95b4-4c9c-9dc4-106eca4a54ad" (UID: "635f1ba7-95b4-4c9c-9dc4-106eca4a54ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:38.208361 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.208329 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-workload-socket" pod "635f1ba7-95b4-4c9c-9dc4-106eca4a54ad" (UID: "635f1ba7-95b4-4c9c-9dc4-106eca4a54ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:38.214003 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.213974 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-istio-token" pod "635f1ba7-95b4-4c9c-9dc4-106eca4a54ad" (UID: "635f1ba7-95b4-4c9c-9dc4-106eca4a54ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:38.217512 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.217481 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-kube-api-access-fgqsk" pod "635f1ba7-95b4-4c9c-9dc4-106eca4a54ad" (UID: "635f1ba7-95b4-4c9c-9dc4-106eca4a54ad"). InnerVolumeSpecName "kube-api-access-fgqsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:38.294214 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.293823 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:38.294214 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.293891 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgqsk\" (UniqueName: \"kubernetes.io/projected/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-kube-api-access-fgqsk\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:38.294214 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.293912 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:38.294214 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.293925 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:38.294214 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.293942 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:38.294214 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.293957 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:38.294214 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.293972 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:38.294214 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.293985 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:38.294214 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.293997 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:38.339364 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.339329 4363 generic.go:356] "Generic (PLEG): container finished" podID="635f1ba7-95b4-4c9c-9dc4-106eca4a54ad" containerID="ccd7813e01b568a03a63370557d80a54b0ed2c782f593853761de6b9f65d0df8" exitCode=0 Nov 01 16:32:38.339547 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.339394 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" Nov 01 16:32:38.339547 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.339422 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" event={"ID":"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad","Type":"ContainerDied","Data":"ccd7813e01b568a03a63370557d80a54b0ed2c782f593853761de6b9f65d0df8"} Nov 01 16:32:38.339547 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.339517 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr" event={"ID":"635f1ba7-95b4-4c9c-9dc4-106eca4a54ad","Type":"ContainerDied","Data":"371e74b5cecac32fbfeaad1ffc9017f4971abd095d9e0a82a18f26ad6a09d779"} Nov 01 16:32:38.339547 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.339546 4363 scope.go:122] "RemoveContainer" containerID="ccd7813e01b568a03a63370557d80a54b0ed2c782f593853761de6b9f65d0df8" Nov 01 16:32:38.341488 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.341466 4363 generic.go:356] "Generic (PLEG): container finished" podID="cf9b2de9-f52e-4580-af9e-d77e06ee6e64" containerID="9012a4038cbd6b621e94b09b6f15ba48bac9d607395e4dd1937895ee00ecd6e2" exitCode=0 Nov 01 16:32:38.341619 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.341538 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" event={"ID":"cf9b2de9-f52e-4580-af9e-d77e06ee6e64","Type":"ContainerDied","Data":"9012a4038cbd6b621e94b09b6f15ba48bac9d607395e4dd1937895ee00ecd6e2"} Nov 01 16:32:38.351270 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.351252 4363 scope.go:122] "RemoveContainer" containerID="ccd7813e01b568a03a63370557d80a54b0ed2c782f593853761de6b9f65d0df8" Nov 01 16:32:38.351563 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:32:38.351541 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccd7813e01b568a03a63370557d80a54b0ed2c782f593853761de6b9f65d0df8\": container with ID starting with ccd7813e01b568a03a63370557d80a54b0ed2c782f593853761de6b9f65d0df8 not found: ID does not exist" containerID="ccd7813e01b568a03a63370557d80a54b0ed2c782f593853761de6b9f65d0df8" Nov 01 16:32:38.351636 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.351578 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccd7813e01b568a03a63370557d80a54b0ed2c782f593853761de6b9f65d0df8"} err="failed to get container status \"ccd7813e01b568a03a63370557d80a54b0ed2c782f593853761de6b9f65d0df8\": rpc error: code = NotFound desc = could not find container \"ccd7813e01b568a03a63370557d80a54b0ed2c782f593853761de6b9f65d0df8\": container with ID starting with ccd7813e01b568a03a63370557d80a54b0ed2c782f593853761de6b9f65d0df8 not found: ID does not exist" Nov 01 16:32:38.362986 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.362944 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr"] Nov 01 16:32:38.367084 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.367061 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-bbrlr"] Nov 01 16:32:38.536237 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.536210 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" Nov 01 16:32:38.603736 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.603703 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-istio-token\" (UniqueName: \"kubernetes.io/projected/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-istio-token\") pod \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\" (UID: \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\") " Nov 01 16:32:38.603923 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.603753 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-istiod-ca-cert\") pod \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\" (UID: \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\") " Nov 01 16:32:38.603923 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.603825 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-istio-envoy\") pod \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\" (UID: \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\") " Nov 01 16:32:38.603923 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.603889 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-istio-podinfo\") pod \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\" (UID: \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\") " Nov 01 16:32:38.604152 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.604035 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-kube-api-access-84q4q\" (UniqueName: \"kubernetes.io/projected/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-kube-api-access-84q4q\") pod \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\" (UID: \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\") " Nov 01 16:32:38.604152 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.604081 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-workload-certs\") pod \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\" (UID: \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\") " Nov 01 16:32:38.604489 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.604464 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-workload-socket\") pod \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\" (UID: \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\") " Nov 01 16:32:38.604592 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.604527 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-istio-data\") pod \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\" (UID: \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\") " Nov 01 16:32:38.604592 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.604574 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-credential-socket\") pod \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\" (UID: \"cf9b2de9-f52e-4580-af9e-d77e06ee6e64\") " Nov 01 16:32:38.604715 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.604466 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-workload-certs" pod "cf9b2de9-f52e-4580-af9e-d77e06ee6e64" (UID: "cf9b2de9-f52e-4580-af9e-d77e06ee6e64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:38.604805 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.604781 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-istio-data" pod "cf9b2de9-f52e-4580-af9e-d77e06ee6e64" (UID: "cf9b2de9-f52e-4580-af9e-d77e06ee6e64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:38.604877 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.604856 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-istiod-ca-cert" pod "cf9b2de9-f52e-4580-af9e-d77e06ee6e64" (UID: "cf9b2de9-f52e-4580-af9e-d77e06ee6e64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:32:38.604991 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.604967 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-credential-socket" pod "cf9b2de9-f52e-4580-af9e-d77e06ee6e64" (UID: "cf9b2de9-f52e-4580-af9e-d77e06ee6e64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:38.605404 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.605380 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:38.605516 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.605408 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:38.605516 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.605456 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:38.605516 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.605469 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:38.605690 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.605529 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-workload-socket" pod "cf9b2de9-f52e-4580-af9e-d77e06ee6e64" (UID: "cf9b2de9-f52e-4580-af9e-d77e06ee6e64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:38.607613 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.607539 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-istio-podinfo" pod "cf9b2de9-f52e-4580-af9e-d77e06ee6e64" (UID: "cf9b2de9-f52e-4580-af9e-d77e06ee6e64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:32:38.608333 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.608309 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-istio-token" pod "cf9b2de9-f52e-4580-af9e-d77e06ee6e64" (UID: "cf9b2de9-f52e-4580-af9e-d77e06ee6e64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:38.608530 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.608510 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-kube-api-access-84q4q" pod "cf9b2de9-f52e-4580-af9e-d77e06ee6e64" (UID: "cf9b2de9-f52e-4580-af9e-d77e06ee6e64"). InnerVolumeSpecName "kube-api-access-84q4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:38.608737 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.608719 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-istio-envoy" pod "cf9b2de9-f52e-4580-af9e-d77e06ee6e64" (UID: "cf9b2de9-f52e-4580-af9e-d77e06ee6e64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:38.707196 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.707099 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:38.707196 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.707135 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:38.707196 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.707146 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84q4q\" (UniqueName: \"kubernetes.io/projected/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-kube-api-access-84q4q\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:38.707196 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.707156 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:38.707196 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.707165 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf9b2de9-f52e-4580-af9e-d77e06ee6e64-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:38.801026 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:38.800990 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="635f1ba7-95b4-4c9c-9dc4-106eca4a54ad" path="/var/lib/kubelet/pods/635f1ba7-95b4-4c9c-9dc4-106eca4a54ad/volumes" Nov 01 16:32:39.098751 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.098721 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f47890-84fdfb9c59-fs5nn" Nov 01 16:32:39.214978 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.214936 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18faa082-6418-4665-b862-09efa76dfcc0-kube-api-access-mbvbt\" (UniqueName: \"kubernetes.io/projected/18faa082-6418-4665-b862-09efa76dfcc0-kube-api-access-mbvbt\") pod \"18faa082-6418-4665-b862-09efa76dfcc0\" (UID: \"18faa082-6418-4665-b862-09efa76dfcc0\") " Nov 01 16:32:39.215193 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.215134 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18faa082-6418-4665-b862-09efa76dfcc0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18faa082-6418-4665-b862-09efa76dfcc0-plugins-config-volume\") pod \"18faa082-6418-4665-b862-09efa76dfcc0\" (UID: \"18faa082-6418-4665-b862-09efa76dfcc0\") " Nov 01 16:32:39.215806 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.215775 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18faa082-6418-4665-b862-09efa76dfcc0-plugins-config-volume" pod "18faa082-6418-4665-b862-09efa76dfcc0" (UID: "18faa082-6418-4665-b862-09efa76dfcc0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:32:39.217697 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.217672 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18faa082-6418-4665-b862-09efa76dfcc0-kube-api-access-mbvbt" pod "18faa082-6418-4665-b862-09efa76dfcc0" (UID: "18faa082-6418-4665-b862-09efa76dfcc0"). InnerVolumeSpecName "kube-api-access-mbvbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:39.316755 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.316718 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18faa082-6418-4665-b862-09efa76dfcc0-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:39.316755 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.316748 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbvbt\" (UniqueName: \"kubernetes.io/projected/18faa082-6418-4665-b862-09efa76dfcc0-kube-api-access-mbvbt\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:39.350771 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.350743 4363 generic.go:356] "Generic (PLEG): container finished" podID="18faa082-6418-4665-b862-09efa76dfcc0" containerID="29c3809b77e4154932cca1a2bde9b52af7ad2e0b5f4d661ed64bb44b9d9c7e85" exitCode=0 Nov 01 16:32:39.350974 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.350786 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f47890-84fdfb9c59-fs5nn" Nov 01 16:32:39.350974 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.350831 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f47890-84fdfb9c59-fs5nn" event={"ID":"18faa082-6418-4665-b862-09efa76dfcc0","Type":"ContainerDied","Data":"29c3809b77e4154932cca1a2bde9b52af7ad2e0b5f4d661ed64bb44b9d9c7e85"} Nov 01 16:32:39.350974 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.350897 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f47890-84fdfb9c59-fs5nn" event={"ID":"18faa082-6418-4665-b862-09efa76dfcc0","Type":"ContainerDied","Data":"3f6db0e6f8d11490c6aa2b6e355e1b714069c1304719171183cc69e3324b4d5f"} Nov 01 16:32:39.350974 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.350916 4363 scope.go:122] "RemoveContainer" containerID="29c3809b77e4154932cca1a2bde9b52af7ad2e0b5f4d661ed64bb44b9d9c7e85" Nov 01 16:32:39.352517 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.352497 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" Nov 01 16:32:39.352652 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.352537 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd" event={"ID":"cf9b2de9-f52e-4580-af9e-d77e06ee6e64","Type":"ContainerDied","Data":"be586793e31a557aa9ae900a9c4d1267086d5b4ba93db804210484bab9831ec4"} Nov 01 16:32:39.362544 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.362528 4363 scope.go:122] "RemoveContainer" containerID="29c3809b77e4154932cca1a2bde9b52af7ad2e0b5f4d661ed64bb44b9d9c7e85" Nov 01 16:32:39.362794 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:32:39.362775 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29c3809b77e4154932cca1a2bde9b52af7ad2e0b5f4d661ed64bb44b9d9c7e85\": container with ID starting with 29c3809b77e4154932cca1a2bde9b52af7ad2e0b5f4d661ed64bb44b9d9c7e85 not found: ID does not exist" containerID="29c3809b77e4154932cca1a2bde9b52af7ad2e0b5f4d661ed64bb44b9d9c7e85" Nov 01 16:32:39.362876 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.362801 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29c3809b77e4154932cca1a2bde9b52af7ad2e0b5f4d661ed64bb44b9d9c7e85"} err="failed to get container status \"29c3809b77e4154932cca1a2bde9b52af7ad2e0b5f4d661ed64bb44b9d9c7e85\": rpc error: code = NotFound desc = could not find container \"29c3809b77e4154932cca1a2bde9b52af7ad2e0b5f4d661ed64bb44b9d9c7e85\": container with ID starting with 29c3809b77e4154932cca1a2bde9b52af7ad2e0b5f4d661ed64bb44b9d9c7e85 not found: ID does not exist" Nov 01 16:32:39.362876 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.362815 4363 scope.go:122] "RemoveContainer" containerID="9012a4038cbd6b621e94b09b6f15ba48bac9d607395e4dd1937895ee00ecd6e2" Nov 01 16:32:39.372941 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.372616 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd"] Nov 01 16:32:39.377654 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.377632 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cfhrd"] Nov 01 16:32:39.387782 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.387749 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f47890-84fdfb9c59-fs5nn"] Nov 01 16:32:39.393451 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.393421 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f47890-84fdfb9c59-fs5nn"] Nov 01 16:32:39.752226 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.752187 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-501d63-d548b6d79-w9jxd"] Nov 01 16:32:39.753401 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.753375 4363 cpu_manager.go:414] "containerMap: removing container" podUID="18faa082-6418-4665-b862-09efa76dfcc0" containerName="payload-pre-processing" Nov 01 16:32:39.753401 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.753400 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18faa082-6418-4665-b862-09efa76dfcc0" containerName="payload-pre-processing" Nov 01 16:32:39.753610 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.753424 4363 cpu_manager.go:414] "containerMap: removing container" podUID="635f1ba7-95b4-4c9c-9dc4-106eca4a54ad" containerName="istio-proxy" Nov 01 16:32:39.753610 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.753432 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="635f1ba7-95b4-4c9c-9dc4-106eca4a54ad" containerName="istio-proxy" Nov 01 16:32:39.753610 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.753461 4363 cpu_manager.go:414] "containerMap: removing container" podUID="2cc4aea6-d369-49de-89f5-73ab72ae0d5b" containerName="revoke-keys" Nov 01 16:32:39.753610 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.753469 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cc4aea6-d369-49de-89f5-73ab72ae0d5b" containerName="revoke-keys" Nov 01 16:32:39.753610 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.753493 4363 cpu_manager.go:414] "containerMap: removing container" podUID="cf9b2de9-f52e-4580-af9e-d77e06ee6e64" containerName="istio-proxy" Nov 01 16:32:39.753610 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.753502 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf9b2de9-f52e-4580-af9e-d77e06ee6e64" containerName="istio-proxy" Nov 01 16:32:39.753838 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.753641 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="18faa082-6418-4665-b862-09efa76dfcc0" containerName="payload-pre-processing" Nov 01 16:32:39.753838 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.753659 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="635f1ba7-95b4-4c9c-9dc4-106eca4a54ad" containerName="istio-proxy" Nov 01 16:32:39.753838 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.753670 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cc4aea6-d369-49de-89f5-73ab72ae0d5b" containerName="revoke-keys" Nov 01 16:32:39.753838 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.753683 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf9b2de9-f52e-4580-af9e-d77e06ee6e64" containerName="istio-proxy" Nov 01 16:32:39.759487 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.759466 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-501d63-d548b6d79-w9jxd" Nov 01 16:32:39.765793 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.765252 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-501d63\"" Nov 01 16:32:39.769978 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.769944 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-501d63-d548b6d79-w9jxd"] Nov 01 16:32:39.773720 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.773698 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-6689474455-mhl6q"] Nov 01 16:32:39.778539 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.778518 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-6689474455-mhl6q" Nov 01 16:32:39.786864 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.786836 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-501d63-dockercfg-4lmw9\"" Nov 01 16:32:39.805033 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.802718 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-6689474455-mhl6q"] Nov 01 16:32:39.824343 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.824303 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hqrq\" (UniqueName: \"kubernetes.io/projected/f70ac519-b330-4122-b106-2269b0bfd261-kube-api-access-2hqrq\") pod \"payload-processing-e2e-ipp-a-501d63-d548b6d79-w9jxd\" (UID: \"f70ac519-b330-4122-b106-2269b0bfd261\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-501d63-d548b6d79-w9jxd" Nov 01 16:32:39.824571 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.824550 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f70ac519-b330-4122-b106-2269b0bfd261-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-501d63-d548b6d79-w9jxd\" (UID: \"f70ac519-b330-4122-b106-2269b0bfd261\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-501d63-d548b6d79-w9jxd" Nov 01 16:32:39.930892 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.930850 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hqrq\" (UniqueName: \"kubernetes.io/projected/f70ac519-b330-4122-b106-2269b0bfd261-kube-api-access-2hqrq\") pod \"payload-processing-e2e-ipp-a-501d63-d548b6d79-w9jxd\" (UID: \"f70ac519-b330-4122-b106-2269b0bfd261\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-501d63-d548b6d79-w9jxd" Nov 01 16:32:39.944207 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.944171 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4301c73d-6b01-40b7-a4b7-c8d4384520ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-501d63-6689474455-mhl6q\" (UID: \"4301c73d-6b01-40b7-a4b7-c8d4384520ca\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-6689474455-mhl6q" Nov 01 16:32:39.944612 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.944588 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f70ac519-b330-4122-b106-2269b0bfd261-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-501d63-d548b6d79-w9jxd\" (UID: \"f70ac519-b330-4122-b106-2269b0bfd261\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-501d63-d548b6d79-w9jxd" Nov 01 16:32:39.945744 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.945720 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhgp5\" (UniqueName: \"kubernetes.io/projected/4301c73d-6b01-40b7-a4b7-c8d4384520ca-kube-api-access-bhgp5\") pod \"payload-pre-processing-e2e-ipp-a-501d63-6689474455-mhl6q\" (UID: \"4301c73d-6b01-40b7-a4b7-c8d4384520ca\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-6689474455-mhl6q" Nov 01 16:32:39.959753 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.945650 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f70ac519-b330-4122-b106-2269b0bfd261-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-501d63-d548b6d79-w9jxd\" (UID: \"f70ac519-b330-4122-b106-2269b0bfd261\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-501d63-d548b6d79-w9jxd" Nov 01 16:32:39.971145 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:39.962201 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hqrq\" (UniqueName: \"kubernetes.io/projected/f70ac519-b330-4122-b106-2269b0bfd261-kube-api-access-2hqrq\") pod \"payload-processing-e2e-ipp-a-501d63-d548b6d79-w9jxd\" (UID: \"f70ac519-b330-4122-b106-2269b0bfd261\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-501d63-d548b6d79-w9jxd" Nov 01 16:32:40.054814 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:40.052905 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4301c73d-6b01-40b7-a4b7-c8d4384520ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-501d63-6689474455-mhl6q\" (UID: \"4301c73d-6b01-40b7-a4b7-c8d4384520ca\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-6689474455-mhl6q" Nov 01 16:32:40.054814 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:40.053120 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhgp5\" (UniqueName: \"kubernetes.io/projected/4301c73d-6b01-40b7-a4b7-c8d4384520ca-kube-api-access-bhgp5\") pod \"payload-pre-processing-e2e-ipp-a-501d63-6689474455-mhl6q\" (UID: \"4301c73d-6b01-40b7-a4b7-c8d4384520ca\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-6689474455-mhl6q" Nov 01 16:32:40.054814 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:40.054418 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4301c73d-6b01-40b7-a4b7-c8d4384520ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-501d63-6689474455-mhl6q\" (UID: \"4301c73d-6b01-40b7-a4b7-c8d4384520ca\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-6689474455-mhl6q" Nov 01 16:32:40.076117 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:40.076090 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-501d63-d548b6d79-w9jxd" Nov 01 16:32:40.087934 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:40.087906 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhgp5\" (UniqueName: \"kubernetes.io/projected/4301c73d-6b01-40b7-a4b7-c8d4384520ca-kube-api-access-bhgp5\") pod \"payload-pre-processing-e2e-ipp-a-501d63-6689474455-mhl6q\" (UID: \"4301c73d-6b01-40b7-a4b7-c8d4384520ca\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-6689474455-mhl6q" Nov 01 16:32:40.114125 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:40.114072 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-6689474455-mhl6q" Nov 01 16:32:40.387144 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:40.385167 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-501d63-d548b6d79-w9jxd"] Nov 01 16:32:40.392484 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:32:40.392454 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf70ac519_b330_4122_b106_2269b0bfd261.slice/crio-37bb80a8104b366620eb28b469231dff006aecd7e9b79761b9da28760b79edc3 WatchSource:0}: Error finding container 37bb80a8104b366620eb28b469231dff006aecd7e9b79761b9da28760b79edc3: Status 404 returned error can't find the container with id 37bb80a8104b366620eb28b469231dff006aecd7e9b79761b9da28760b79edc3 Nov 01 16:32:40.475595 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:40.475564 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-6689474455-mhl6q"] Nov 01 16:32:40.476083 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:32:40.476055 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4301c73d_6b01_40b7_a4b7_c8d4384520ca.slice/crio-acedfbea0333b30a157957a0a3b43e58eadef2b40a7650c2a65d238013f13696 WatchSource:0}: Error finding container acedfbea0333b30a157957a0a3b43e58eadef2b40a7650c2a65d238013f13696: Status 404 returned error can't find the container with id acedfbea0333b30a157957a0a3b43e58eadef2b40a7650c2a65d238013f13696 Nov 01 16:32:40.801875 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:40.801784 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18faa082-6418-4665-b862-09efa76dfcc0" path="/var/lib/kubelet/pods/18faa082-6418-4665-b862-09efa76dfcc0/volumes" Nov 01 16:32:40.802474 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:40.802452 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf9b2de9-f52e-4580-af9e-d77e06ee6e64" path="/var/lib/kubelet/pods/cf9b2de9-f52e-4580-af9e-d77e06ee6e64/volumes" Nov 01 16:32:41.183458 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.183431 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" Nov 01 16:32:41.279253 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.279216 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-istio-podinfo\") pod \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\" (UID: \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\") " Nov 01 16:32:41.279444 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.279370 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-workload-certs\") pod \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\" (UID: \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\") " Nov 01 16:32:41.279444 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.279430 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-istio-data\") pod \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\" (UID: \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\") " Nov 01 16:32:41.279519 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.279480 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-kube-api-access-pmglj\" (UniqueName: \"kubernetes.io/projected/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-kube-api-access-pmglj\") pod \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\" (UID: \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\") " Nov 01 16:32:41.279553 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.279527 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-credential-socket\") pod \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\" (UID: \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\") " Nov 01 16:32:41.279586 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.279561 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-istio-token\" (UniqueName: \"kubernetes.io/projected/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-istio-token\") pod \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\" (UID: \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\") " Nov 01 16:32:41.279679 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.279665 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-istiod-ca-cert\") pod \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\" (UID: \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\") " Nov 01 16:32:41.279767 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.279754 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-istio-envoy\") pod \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\" (UID: \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\") " Nov 01 16:32:41.279836 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.279823 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-workload-socket\") pod \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\" (UID: \"ff3d8641-74e7-496e-8b67-fd5db4a94e8e\") " Nov 01 16:32:41.280677 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.280651 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-workload-socket" pod "ff3d8641-74e7-496e-8b67-fd5db4a94e8e" (UID: "ff3d8641-74e7-496e-8b67-fd5db4a94e8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:41.281309 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.281282 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-istio-data" pod "ff3d8641-74e7-496e-8b67-fd5db4a94e8e" (UID: "ff3d8641-74e7-496e-8b67-fd5db4a94e8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:41.281522 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.281289 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-credential-socket" pod "ff3d8641-74e7-496e-8b67-fd5db4a94e8e" (UID: "ff3d8641-74e7-496e-8b67-fd5db4a94e8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:41.281676 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.281306 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-workload-certs" pod "ff3d8641-74e7-496e-8b67-fd5db4a94e8e" (UID: "ff3d8641-74e7-496e-8b67-fd5db4a94e8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:41.281994 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.281972 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-istiod-ca-cert" pod "ff3d8641-74e7-496e-8b67-fd5db4a94e8e" (UID: "ff3d8641-74e7-496e-8b67-fd5db4a94e8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:32:41.283322 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.283281 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-istio-podinfo" pod "ff3d8641-74e7-496e-8b67-fd5db4a94e8e" (UID: "ff3d8641-74e7-496e-8b67-fd5db4a94e8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:32:41.283736 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.283612 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-kube-api-access-pmglj" pod "ff3d8641-74e7-496e-8b67-fd5db4a94e8e" (UID: "ff3d8641-74e7-496e-8b67-fd5db4a94e8e"). InnerVolumeSpecName "kube-api-access-pmglj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:41.283736 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.283695 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-istio-token" pod "ff3d8641-74e7-496e-8b67-fd5db4a94e8e" (UID: "ff3d8641-74e7-496e-8b67-fd5db4a94e8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:41.283891 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.283780 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-istio-envoy" pod "ff3d8641-74e7-496e-8b67-fd5db4a94e8e" (UID: "ff3d8641-74e7-496e-8b67-fd5db4a94e8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:41.373170 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.373138 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-6689474455-mhl6q" event={"ID":"4301c73d-6b01-40b7-a4b7-c8d4384520ca","Type":"ContainerStarted","Data":"6161c68e95cc62d641d6b9794f70d556f5a51a888e2755b4bba58cb66e080f3c"} Nov 01 16:32:41.373329 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.373196 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-6689474455-mhl6q" event={"ID":"4301c73d-6b01-40b7-a4b7-c8d4384520ca","Type":"ContainerStarted","Data":"acedfbea0333b30a157957a0a3b43e58eadef2b40a7650c2a65d238013f13696"} Nov 01 16:32:41.373329 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.373240 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-6689474455-mhl6q" Nov 01 16:32:41.374599 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.374575 4363 generic.go:356] "Generic (PLEG): container finished" podID="ff3d8641-74e7-496e-8b67-fd5db4a94e8e" containerID="047032a99321f97e085c9eaf28c06b2be9413fcee7a10ac837a0d72e404b5659" exitCode=0 Nov 01 16:32:41.374687 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.374616 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" Nov 01 16:32:41.374687 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.374652 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" event={"ID":"ff3d8641-74e7-496e-8b67-fd5db4a94e8e","Type":"ContainerDied","Data":"047032a99321f97e085c9eaf28c06b2be9413fcee7a10ac837a0d72e404b5659"} Nov 01 16:32:41.374782 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.374698 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs" event={"ID":"ff3d8641-74e7-496e-8b67-fd5db4a94e8e","Type":"ContainerDied","Data":"719c331401c8aa6489decbea4f9839c08a44409641d2d0d92e528347693ab868"} Nov 01 16:32:41.374782 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.374722 4363 scope.go:122] "RemoveContainer" containerID="047032a99321f97e085c9eaf28c06b2be9413fcee7a10ac837a0d72e404b5659" Nov 01 16:32:41.376726 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.376697 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-501d63-d548b6d79-w9jxd" event={"ID":"f70ac519-b330-4122-b106-2269b0bfd261","Type":"ContainerStarted","Data":"7e1b1fc826f95e248d4902c7a7d46f804e7ce8be8a1005c0d54073f12e3786e0"} Nov 01 16:32:41.376833 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.376751 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-501d63-d548b6d79-w9jxd" event={"ID":"f70ac519-b330-4122-b106-2269b0bfd261","Type":"ContainerStarted","Data":"37bb80a8104b366620eb28b469231dff006aecd7e9b79761b9da28760b79edc3"} Nov 01 16:32:41.376833 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.376779 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-501d63-d548b6d79-w9jxd" Nov 01 16:32:41.406046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.402123 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:41.406046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.402155 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:41.406046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.402173 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:41.406046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.402189 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:41.406046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.402202 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:41.406046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.402215 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmglj\" (UniqueName: \"kubernetes.io/projected/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-kube-api-access-pmglj\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:41.406046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.402229 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:41.406046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.402242 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:41.406046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.402256 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff3d8641-74e7-496e-8b67-fd5db4a94e8e-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:41.419675 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.414685 4363 scope.go:122] "RemoveContainer" containerID="047032a99321f97e085c9eaf28c06b2be9413fcee7a10ac837a0d72e404b5659" Nov 01 16:32:41.425192 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:32:41.419932 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"047032a99321f97e085c9eaf28c06b2be9413fcee7a10ac837a0d72e404b5659\": container with ID starting with 047032a99321f97e085c9eaf28c06b2be9413fcee7a10ac837a0d72e404b5659 not found: ID does not exist" containerID="047032a99321f97e085c9eaf28c06b2be9413fcee7a10ac837a0d72e404b5659" Nov 01 16:32:41.425192 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.419977 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"047032a99321f97e085c9eaf28c06b2be9413fcee7a10ac837a0d72e404b5659"} err="failed to get container status \"047032a99321f97e085c9eaf28c06b2be9413fcee7a10ac837a0d72e404b5659\": rpc error: code = NotFound desc = could not find container \"047032a99321f97e085c9eaf28c06b2be9413fcee7a10ac837a0d72e404b5659\": container with ID starting with 047032a99321f97e085c9eaf28c06b2be9413fcee7a10ac837a0d72e404b5659 not found: ID does not exist" Nov 01 16:32:41.436085 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.435046 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-6689474455-mhl6q" podStartSLOduration=2.435005964 podStartE2EDuration="2.435005964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:32:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:32:41.396766553 +0000 UTC m=+2694.728504225" watchObservedRunningTime="2026-11-01 16:32:41.435005964 +0000 UTC m=+2694.766743617" Nov 01 16:32:41.464437 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.464369 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-501d63-d548b6d79-w9jxd" podStartSLOduration=2.464351235 podStartE2EDuration="2.464351235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:32:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:32:41.421240806 +0000 UTC m=+2694.752978458" watchObservedRunningTime="2026-11-01 16:32:41.464351235 +0000 UTC m=+2694.796088887" Nov 01 16:32:41.476704 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.476497 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs"] Nov 01 16:32:41.481059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:41.478197 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-p5ffs"] Nov 01 16:32:42.801693 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:42.801663 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff3d8641-74e7-496e-8b67-fd5db4a94e8e" path="/var/lib/kubelet/pods/ff3d8641-74e7-496e-8b67-fd5db4a94e8e/volumes" Nov 01 16:32:43.381924 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:43.381886 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-734859-5fc5fb544f-n2j8r"] Nov 01 16:32:43.401101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:43.401069 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-734859-589cbc787-xqhjc"] Nov 01 16:32:45.003660 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:45.003628 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa6303-5bcf66496-8dckq"] Nov 01 16:32:45.004078 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:45.003859 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa6303-5bcf66496-8dckq" podUID="bc9aad52-e26b-4e45-9241-ad27218724d7" containerName="payload-processing" containerID="cri-o://0522aa7f097281170b331bd8ca4019e7a50fc9a3f04307f17e6757c943dd4a30" gracePeriod=30 Nov 01 16:32:45.015579 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:45.015134 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-cd5bc765f-9f5tw"] Nov 01 16:32:45.015579 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:45.015441 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-cd5bc765f-9f5tw" podUID="0b8aa932-6702-4c56-acaf-2bebb3d5074f" containerName="payload-pre-processing" containerID="cri-o://e76892e8178156c4582f94d49a3f13ce62b305b082826857f36c6da96a2ba152" gracePeriod=30 Nov 01 16:32:45.363732 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:45.363701 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa6303-5bcf66496-8dckq" Nov 01 16:32:45.495102 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:45.476609 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc9aad52-e26b-4e45-9241-ad27218724d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc9aad52-e26b-4e45-9241-ad27218724d7-plugins-config-volume\") pod \"bc9aad52-e26b-4e45-9241-ad27218724d7\" (UID: \"bc9aad52-e26b-4e45-9241-ad27218724d7\") " Nov 01 16:32:45.495102 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:45.476785 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc9aad52-e26b-4e45-9241-ad27218724d7-kube-api-access-7n4kg\" (UniqueName: \"kubernetes.io/projected/bc9aad52-e26b-4e45-9241-ad27218724d7-kube-api-access-7n4kg\") pod \"bc9aad52-e26b-4e45-9241-ad27218724d7\" (UID: \"bc9aad52-e26b-4e45-9241-ad27218724d7\") " Nov 01 16:32:45.495102 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:45.481261 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc9aad52-e26b-4e45-9241-ad27218724d7-kube-api-access-7n4kg" pod "bc9aad52-e26b-4e45-9241-ad27218724d7" (UID: "bc9aad52-e26b-4e45-9241-ad27218724d7"). InnerVolumeSpecName "kube-api-access-7n4kg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:45.495102 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:45.482941 4363 generic.go:356] "Generic (PLEG): container finished" podID="bc9aad52-e26b-4e45-9241-ad27218724d7" containerID="0522aa7f097281170b331bd8ca4019e7a50fc9a3f04307f17e6757c943dd4a30" exitCode=0 Nov 01 16:32:45.495102 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:45.483205 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa6303-5bcf66496-8dckq" event={"ID":"bc9aad52-e26b-4e45-9241-ad27218724d7","Type":"ContainerDied","Data":"0522aa7f097281170b331bd8ca4019e7a50fc9a3f04307f17e6757c943dd4a30"} Nov 01 16:32:45.495102 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:45.483261 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa6303-5bcf66496-8dckq" event={"ID":"bc9aad52-e26b-4e45-9241-ad27218724d7","Type":"ContainerDied","Data":"c8fdc4b2fe7e9e20ee972fa719f5496b6e118163dc68bf042fd04466b3f1ffbb"} Nov 01 16:32:45.495102 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:45.483291 4363 scope.go:122] "RemoveContainer" containerID="0522aa7f097281170b331bd8ca4019e7a50fc9a3f04307f17e6757c943dd4a30" Nov 01 16:32:45.495102 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:45.483433 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa6303-5bcf66496-8dckq" Nov 01 16:32:45.495102 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:45.484527 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc9aad52-e26b-4e45-9241-ad27218724d7-plugins-config-volume" pod "bc9aad52-e26b-4e45-9241-ad27218724d7" (UID: "bc9aad52-e26b-4e45-9241-ad27218724d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:32:45.507161 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:45.500652 4363 generic.go:356] "Generic (PLEG): container finished" podID="0b8aa932-6702-4c56-acaf-2bebb3d5074f" containerID="e76892e8178156c4582f94d49a3f13ce62b305b082826857f36c6da96a2ba152" exitCode=0 Nov 01 16:32:45.507161 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:45.500703 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-cd5bc765f-9f5tw" event={"ID":"0b8aa932-6702-4c56-acaf-2bebb3d5074f","Type":"ContainerDied","Data":"e76892e8178156c4582f94d49a3f13ce62b305b082826857f36c6da96a2ba152"} Nov 01 16:32:45.530360 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:45.528605 4363 scope.go:122] "RemoveContainer" containerID="0522aa7f097281170b331bd8ca4019e7a50fc9a3f04307f17e6757c943dd4a30" Nov 01 16:32:45.530360 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:32:45.529136 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0522aa7f097281170b331bd8ca4019e7a50fc9a3f04307f17e6757c943dd4a30\": container with ID starting with 0522aa7f097281170b331bd8ca4019e7a50fc9a3f04307f17e6757c943dd4a30 not found: ID does not exist" containerID="0522aa7f097281170b331bd8ca4019e7a50fc9a3f04307f17e6757c943dd4a30" Nov 01 16:32:45.530360 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:45.529177 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0522aa7f097281170b331bd8ca4019e7a50fc9a3f04307f17e6757c943dd4a30"} err="failed to get container status \"0522aa7f097281170b331bd8ca4019e7a50fc9a3f04307f17e6757c943dd4a30\": rpc error: code = NotFound desc = could not find container \"0522aa7f097281170b331bd8ca4019e7a50fc9a3f04307f17e6757c943dd4a30\": container with ID starting with 0522aa7f097281170b331bd8ca4019e7a50fc9a3f04307f17e6757c943dd4a30 not found: ID does not exist" Nov 01 16:32:45.544529 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:45.544214 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-cd5bc765f-9f5tw" Nov 01 16:32:45.579891 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:45.579778 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc9aad52-e26b-4e45-9241-ad27218724d7-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:45.579891 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:45.579811 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n4kg\" (UniqueName: \"kubernetes.io/projected/bc9aad52-e26b-4e45-9241-ad27218724d7-kube-api-access-7n4kg\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:45.681040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:45.680941 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b8aa932-6702-4c56-acaf-2bebb3d5074f-kube-api-access-mbw68\" (UniqueName: \"kubernetes.io/projected/0b8aa932-6702-4c56-acaf-2bebb3d5074f-kube-api-access-mbw68\") pod \"0b8aa932-6702-4c56-acaf-2bebb3d5074f\" (UID: \"0b8aa932-6702-4c56-acaf-2bebb3d5074f\") " Nov 01 16:32:45.681262 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:45.681098 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b8aa932-6702-4c56-acaf-2bebb3d5074f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b8aa932-6702-4c56-acaf-2bebb3d5074f-plugins-config-volume\") pod \"0b8aa932-6702-4c56-acaf-2bebb3d5074f\" (UID: \"0b8aa932-6702-4c56-acaf-2bebb3d5074f\") " Nov 01 16:32:45.681931 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:45.681900 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b8aa932-6702-4c56-acaf-2bebb3d5074f-plugins-config-volume" pod "0b8aa932-6702-4c56-acaf-2bebb3d5074f" (UID: "0b8aa932-6702-4c56-acaf-2bebb3d5074f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:32:45.684371 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:45.684329 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b8aa932-6702-4c56-acaf-2bebb3d5074f-kube-api-access-mbw68" pod "0b8aa932-6702-4c56-acaf-2bebb3d5074f" (UID: "0b8aa932-6702-4c56-acaf-2bebb3d5074f"). InnerVolumeSpecName "kube-api-access-mbw68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:45.783157 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:45.783107 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbw68\" (UniqueName: \"kubernetes.io/projected/0b8aa932-6702-4c56-acaf-2bebb3d5074f-kube-api-access-mbw68\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:45.783157 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:45.783151 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b8aa932-6702-4c56-acaf-2bebb3d5074f-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:45.811428 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:45.811402 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa6303-5bcf66496-8dckq"] Nov 01 16:32:45.817282 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:45.817258 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa6303-5bcf66496-8dckq"] Nov 01 16:32:46.024133 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:46.024096 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d864b18-rdvnh"] Nov 01 16:32:46.026560 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:46.026539 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d864b18-rdvnh"] Nov 01 16:32:46.539745 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:46.539699 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-cd5bc765f-9f5tw" event={"ID":"0b8aa932-6702-4c56-acaf-2bebb3d5074f","Type":"ContainerDied","Data":"0260126030b8ab1baf44bcf85db597a46f2ecb1e741d7cfebadcb5d80a134c06"} Nov 01 16:32:46.539745 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:46.539735 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-cd5bc765f-9f5tw" Nov 01 16:32:46.540056 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:46.539771 4363 scope.go:122] "RemoveContainer" containerID="e76892e8178156c4582f94d49a3f13ce62b305b082826857f36c6da96a2ba152" Nov 01 16:32:46.564785 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:46.564758 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-cd5bc765f-9f5tw"] Nov 01 16:32:46.568060 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:46.568034 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-cd5bc765f-9f5tw"] Nov 01 16:32:46.800562 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:46.800486 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b8aa932-6702-4c56-acaf-2bebb3d5074f" path="/var/lib/kubelet/pods/0b8aa932-6702-4c56-acaf-2bebb3d5074f/volumes" Nov 01 16:32:46.800943 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:46.800929 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b11d52ce-758e-42f5-bf7b-88adcb1cf938" path="/var/lib/kubelet/pods/b11d52ce-758e-42f5-bf7b-88adcb1cf938/volumes" Nov 01 16:32:46.801317 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:46.801303 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc9aad52-e26b-4e45-9241-ad27218724d7" path="/var/lib/kubelet/pods/bc9aad52-e26b-4e45-9241-ad27218724d7/volumes" Nov 01 16:32:47.953667 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:47.953633 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtwhw_8254aafa-dd5b-4150-87e4-d165dab6bcb5/ovn-acl-logging/0.log" Nov 01 16:32:47.968311 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:47.968291 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtwhw_8254aafa-dd5b-4150-87e4-d165dab6bcb5/ovn-acl-logging/0.log" Nov 01 16:32:48.383154 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:48.383093 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-734859-5fc5fb544f-n2j8r" podUID="2970538a-8b72-47f6-a76b-455f828c8d83" containerName="payload-processing" containerID="cri-o://9a2cfd0a9e4a231492630b9e50963f0372fd1dd18284b3f34d3af9e267f00136" gracePeriod=25 Nov 01 16:32:48.404309 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:48.404248 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-734859-589cbc787-xqhjc" podUID="c56804c0-0810-4106-b393-d292cb816780" containerName="payload-pre-processing" containerID="cri-o://26a7ab71a9f9ca0a81874a278ac144ca1c17b3cb9a8250783630073d54b2e1cd" gracePeriod=25 Nov 01 16:32:48.484701 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:32:48.484661 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:32:49.250636 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:49.250591 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-501d63-56568c796c-8lg27"] Nov 01 16:32:49.251593 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:49.251567 4363 cpu_manager.go:414] "containerMap: removing container" podUID="0b8aa932-6702-4c56-acaf-2bebb3d5074f" containerName="payload-pre-processing" Nov 01 16:32:49.251593 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:49.251593 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b8aa932-6702-4c56-acaf-2bebb3d5074f" containerName="payload-pre-processing" Nov 01 16:32:49.251888 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:49.251617 4363 cpu_manager.go:414] "containerMap: removing container" podUID="ff3d8641-74e7-496e-8b67-fd5db4a94e8e" containerName="istio-proxy" Nov 01 16:32:49.251888 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:49.251626 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff3d8641-74e7-496e-8b67-fd5db4a94e8e" containerName="istio-proxy" Nov 01 16:32:49.251888 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:49.251693 4363 cpu_manager.go:414] "containerMap: removing container" podUID="bc9aad52-e26b-4e45-9241-ad27218724d7" containerName="payload-processing" Nov 01 16:32:49.251888 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:49.251702 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc9aad52-e26b-4e45-9241-ad27218724d7" containerName="payload-processing" Nov 01 16:32:49.251888 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:49.251850 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc9aad52-e26b-4e45-9241-ad27218724d7" containerName="payload-processing" Nov 01 16:32:49.251888 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:49.251871 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b8aa932-6702-4c56-acaf-2bebb3d5074f" containerName="payload-pre-processing" Nov 01 16:32:49.251888 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:49.251884 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff3d8641-74e7-496e-8b67-fd5db4a94e8e" containerName="istio-proxy" Nov 01 16:32:49.262088 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:49.262064 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-501d63-56568c796c-8lg27" Nov 01 16:32:49.271776 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:49.265371 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-501d63-56568c796c-8lg27"] Nov 01 16:32:49.271776 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:49.271273 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-501d63-serving-cert\"" Nov 01 16:32:49.338261 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:49.338222 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29qvs\" (UniqueName: \"kubernetes.io/projected/159aed21-0910-4554-a841-b7c096765924-kube-api-access-29qvs\") pod \"maas-api-e2e-ipp-a-501d63-56568c796c-8lg27\" (UID: \"159aed21-0910-4554-a841-b7c096765924\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-501d63-56568c796c-8lg27" Nov 01 16:32:49.338475 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:49.338425 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/159aed21-0910-4554-a841-b7c096765924-maas-api-tls\") pod \"maas-api-e2e-ipp-a-501d63-56568c796c-8lg27\" (UID: \"159aed21-0910-4554-a841-b7c096765924\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-501d63-56568c796c-8lg27" Nov 01 16:32:49.338475 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:49.338463 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/159aed21-0910-4554-a841-b7c096765924-metrics-certs\") pod \"maas-api-e2e-ipp-a-501d63-56568c796c-8lg27\" (UID: \"159aed21-0910-4554-a841-b7c096765924\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-501d63-56568c796c-8lg27" Nov 01 16:32:49.338641 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:49.338612 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/159aed21-0910-4554-a841-b7c096765924-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-501d63-56568c796c-8lg27\" (UID: \"159aed21-0910-4554-a841-b7c096765924\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-501d63-56568c796c-8lg27" Nov 01 16:32:49.439800 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:49.439756 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/159aed21-0910-4554-a841-b7c096765924-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-501d63-56568c796c-8lg27\" (UID: \"159aed21-0910-4554-a841-b7c096765924\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-501d63-56568c796c-8lg27" Nov 01 16:32:49.440134 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:49.440105 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29qvs\" (UniqueName: \"kubernetes.io/projected/159aed21-0910-4554-a841-b7c096765924-kube-api-access-29qvs\") pod \"maas-api-e2e-ipp-a-501d63-56568c796c-8lg27\" (UID: \"159aed21-0910-4554-a841-b7c096765924\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-501d63-56568c796c-8lg27" Nov 01 16:32:49.440364 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:49.440345 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/159aed21-0910-4554-a841-b7c096765924-maas-api-tls\") pod \"maas-api-e2e-ipp-a-501d63-56568c796c-8lg27\" (UID: \"159aed21-0910-4554-a841-b7c096765924\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-501d63-56568c796c-8lg27" Nov 01 16:32:49.440440 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:49.440392 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/159aed21-0910-4554-a841-b7c096765924-metrics-certs\") pod \"maas-api-e2e-ipp-a-501d63-56568c796c-8lg27\" (UID: \"159aed21-0910-4554-a841-b7c096765924\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-501d63-56568c796c-8lg27" Nov 01 16:32:49.440501 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:49.440435 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/159aed21-0910-4554-a841-b7c096765924-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-501d63-56568c796c-8lg27\" (UID: \"159aed21-0910-4554-a841-b7c096765924\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-501d63-56568c796c-8lg27" Nov 01 16:32:49.459137 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:49.443770 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/159aed21-0910-4554-a841-b7c096765924-metrics-certs\") pod \"maas-api-e2e-ipp-a-501d63-56568c796c-8lg27\" (UID: \"159aed21-0910-4554-a841-b7c096765924\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-501d63-56568c796c-8lg27" Nov 01 16:32:49.459137 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:49.458707 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/159aed21-0910-4554-a841-b7c096765924-maas-api-tls\") pod \"maas-api-e2e-ipp-a-501d63-56568c796c-8lg27\" (UID: \"159aed21-0910-4554-a841-b7c096765924\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-501d63-56568c796c-8lg27" Nov 01 16:32:49.463375 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:49.461063 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29qvs\" (UniqueName: \"kubernetes.io/projected/159aed21-0910-4554-a841-b7c096765924-kube-api-access-29qvs\") pod \"maas-api-e2e-ipp-a-501d63-56568c796c-8lg27\" (UID: \"159aed21-0910-4554-a841-b7c096765924\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-501d63-56568c796c-8lg27" Nov 01 16:32:49.592849 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:49.592823 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-501d63-56568c796c-8lg27" Nov 01 16:32:49.597865 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:49.597833 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9a482b/rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl"] Nov 01 16:32:49.933426 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:49.931326 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-501d63-56568c796c-8lg27"] Nov 01 16:32:49.936457 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:32:49.935273 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod159aed21_0910_4554_a841_b7c096765924.slice/crio-acb171ad1c32b47eced4829ed00e1b9fa250f435d77b810ec7dba4ab0c5998e1 WatchSource:0}: Error finding container acb171ad1c32b47eced4829ed00e1b9fa250f435d77b810ec7dba4ab0c5998e1: Status 404 returned error can't find the container with id acb171ad1c32b47eced4829ed00e1b9fa250f435d77b810ec7dba4ab0c5998e1 Nov 01 16:32:49.937541 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:49.937519 4363 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 16:32:50.610839 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:50.610805 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-501d63-56568c796c-8lg27" event={"ID":"159aed21-0910-4554-a841-b7c096765924","Type":"ContainerStarted","Data":"4799dce9509cf8342931fe38b89a5500d33bfe13610b4b8eba2a72dc08d1e3d1"} Nov 01 16:32:50.611289 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:50.610858 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-501d63-56568c796c-8lg27" event={"ID":"159aed21-0910-4554-a841-b7c096765924","Type":"ContainerStarted","Data":"acb171ad1c32b47eced4829ed00e1b9fa250f435d77b810ec7dba4ab0c5998e1"} Nov 01 16:32:50.611289 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:50.610958 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-501d63-56568c796c-8lg27" Nov 01 16:32:50.630096 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:50.630044 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-501d63-56568c796c-8lg27" podStartSLOduration=1.159219118 podStartE2EDuration="1.630027789s" totalImagesPullingTime="470.808671ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:32:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:32:50.625695777 +0000 UTC m=+2703.957433441" watchObservedRunningTime="2026-11-01 16:32:50.630027789 +0000 UTC m=+2703.961765436" Nov 01 16:32:52.134960 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:52.134924 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h"] Nov 01 16:32:52.135375 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:52.135235 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" podUID="7c97faba-9728-4c13-b99d-19fa20b7e85b" containerName="istio-proxy" containerID="cri-o://5a9a235ac5ca4fc6a6d4ce8b7b3d1a1a240b519dded5c4dd5de5e264c334ee9e" gracePeriod=30 Nov 01 16:32:52.405100 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:52.404417 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-501d63-d548b6d79-w9jxd" Nov 01 16:32:52.422054 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:52.406950 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-6689474455-mhl6q" Nov 01 16:32:52.589669 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:52.589637 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-734859-7lhdq"] Nov 01 16:32:52.592819 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:52.592794 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-734859-7lhdq"] Nov 01 16:32:52.801166 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:52.801074 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cc4aea6-d369-49de-89f5-73ab72ae0d5b" path="/var/lib/kubelet/pods/2cc4aea6-d369-49de-89f5-73ab72ae0d5b/volumes" Nov 01 16:32:54.739042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:54.738146 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-734859-5fc5fb544f-n2j8r" podUID="2970538a-8b72-47f6-a76b-455f828c8d83" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:32:55.722327 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:55.722285 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-734859-589cbc787-xqhjc" podUID="c56804c0-0810-4106-b393-d292cb816780" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:32:56.658660 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:56.656744 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-501d63-56568c796c-8lg27" Nov 01 16:32:57.481350 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:57.481326 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" Nov 01 16:32:57.591259 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:57.591226 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c97faba-9728-4c13-b99d-19fa20b7e85b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c97faba-9728-4c13-b99d-19fa20b7e85b-istiod-ca-cert\") pod \"7c97faba-9728-4c13-b99d-19fa20b7e85b\" (UID: \"7c97faba-9728-4c13-b99d-19fa20b7e85b\") " Nov 01 16:32:57.591455 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:57.591360 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c97faba-9728-4c13-b99d-19fa20b7e85b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c97faba-9728-4c13-b99d-19fa20b7e85b-istio-podinfo\") pod \"7c97faba-9728-4c13-b99d-19fa20b7e85b\" (UID: \"7c97faba-9728-4c13-b99d-19fa20b7e85b\") " Nov 01 16:32:57.591455 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:57.591382 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c97faba-9728-4c13-b99d-19fa20b7e85b-istio-token\" (UniqueName: \"kubernetes.io/projected/7c97faba-9728-4c13-b99d-19fa20b7e85b-istio-token\") pod \"7c97faba-9728-4c13-b99d-19fa20b7e85b\" (UID: \"7c97faba-9728-4c13-b99d-19fa20b7e85b\") " Nov 01 16:32:57.591455 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:57.591400 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c97faba-9728-4c13-b99d-19fa20b7e85b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c97faba-9728-4c13-b99d-19fa20b7e85b-workload-certs\") pod \"7c97faba-9728-4c13-b99d-19fa20b7e85b\" (UID: \"7c97faba-9728-4c13-b99d-19fa20b7e85b\") " Nov 01 16:32:57.591639 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:57.591474 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c97faba-9728-4c13-b99d-19fa20b7e85b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c97faba-9728-4c13-b99d-19fa20b7e85b-istio-envoy\") pod \"7c97faba-9728-4c13-b99d-19fa20b7e85b\" (UID: \"7c97faba-9728-4c13-b99d-19fa20b7e85b\") " Nov 01 16:32:57.591639 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:57.591517 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c97faba-9728-4c13-b99d-19fa20b7e85b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c97faba-9728-4c13-b99d-19fa20b7e85b-credential-socket\") pod \"7c97faba-9728-4c13-b99d-19fa20b7e85b\" (UID: \"7c97faba-9728-4c13-b99d-19fa20b7e85b\") " Nov 01 16:32:57.591639 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:57.591567 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c97faba-9728-4c13-b99d-19fa20b7e85b-kube-api-access-fszcr\" (UniqueName: \"kubernetes.io/projected/7c97faba-9728-4c13-b99d-19fa20b7e85b-kube-api-access-fszcr\") pod \"7c97faba-9728-4c13-b99d-19fa20b7e85b\" (UID: \"7c97faba-9728-4c13-b99d-19fa20b7e85b\") " Nov 01 16:32:57.591639 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:57.591610 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c97faba-9728-4c13-b99d-19fa20b7e85b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c97faba-9728-4c13-b99d-19fa20b7e85b-istio-data\") pod \"7c97faba-9728-4c13-b99d-19fa20b7e85b\" (UID: \"7c97faba-9728-4c13-b99d-19fa20b7e85b\") " Nov 01 16:32:57.591639 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:57.591605 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c97faba-9728-4c13-b99d-19fa20b7e85b-istiod-ca-cert" pod "7c97faba-9728-4c13-b99d-19fa20b7e85b" (UID: "7c97faba-9728-4c13-b99d-19fa20b7e85b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:32:57.591902 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:57.591716 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c97faba-9728-4c13-b99d-19fa20b7e85b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c97faba-9728-4c13-b99d-19fa20b7e85b-workload-socket\") pod \"7c97faba-9728-4c13-b99d-19fa20b7e85b\" (UID: \"7c97faba-9728-4c13-b99d-19fa20b7e85b\") " Nov 01 16:32:57.591902 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:57.591791 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c97faba-9728-4c13-b99d-19fa20b7e85b-workload-certs" pod "7c97faba-9728-4c13-b99d-19fa20b7e85b" (UID: "7c97faba-9728-4c13-b99d-19fa20b7e85b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:57.591902 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:57.591890 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c97faba-9728-4c13-b99d-19fa20b7e85b-credential-socket" pod "7c97faba-9728-4c13-b99d-19fa20b7e85b" (UID: "7c97faba-9728-4c13-b99d-19fa20b7e85b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:57.592167 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:57.592141 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c97faba-9728-4c13-b99d-19fa20b7e85b-istio-data" pod "7c97faba-9728-4c13-b99d-19fa20b7e85b" (UID: "7c97faba-9728-4c13-b99d-19fa20b7e85b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:57.592365 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:57.592344 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c97faba-9728-4c13-b99d-19fa20b7e85b-workload-socket" pod "7c97faba-9728-4c13-b99d-19fa20b7e85b" (UID: "7c97faba-9728-4c13-b99d-19fa20b7e85b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:57.592454 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:57.592410 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c97faba-9728-4c13-b99d-19fa20b7e85b-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:57.592454 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:57.592433 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c97faba-9728-4c13-b99d-19fa20b7e85b-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:57.592454 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:57.592448 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c97faba-9728-4c13-b99d-19fa20b7e85b-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:57.592621 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:57.592462 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c97faba-9728-4c13-b99d-19fa20b7e85b-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:57.593915 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:57.593886 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c97faba-9728-4c13-b99d-19fa20b7e85b-istio-podinfo" pod "7c97faba-9728-4c13-b99d-19fa20b7e85b" (UID: "7c97faba-9728-4c13-b99d-19fa20b7e85b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:32:57.594164 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:57.594143 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c97faba-9728-4c13-b99d-19fa20b7e85b-kube-api-access-fszcr" pod "7c97faba-9728-4c13-b99d-19fa20b7e85b" (UID: "7c97faba-9728-4c13-b99d-19fa20b7e85b"). InnerVolumeSpecName "kube-api-access-fszcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:57.594266 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:57.594241 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c97faba-9728-4c13-b99d-19fa20b7e85b-istio-envoy" pod "7c97faba-9728-4c13-b99d-19fa20b7e85b" (UID: "7c97faba-9728-4c13-b99d-19fa20b7e85b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:57.594514 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:57.594492 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c97faba-9728-4c13-b99d-19fa20b7e85b-istio-token" pod "7c97faba-9728-4c13-b99d-19fa20b7e85b" (UID: "7c97faba-9728-4c13-b99d-19fa20b7e85b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:57.717103 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:57.703130 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c97faba-9728-4c13-b99d-19fa20b7e85b-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:57.717103 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:57.703172 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c97faba-9728-4c13-b99d-19fa20b7e85b-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:57.717103 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:57.703187 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c97faba-9728-4c13-b99d-19fa20b7e85b-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:57.717103 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:57.703201 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fszcr\" (UniqueName: \"kubernetes.io/projected/7c97faba-9728-4c13-b99d-19fa20b7e85b-kube-api-access-fszcr\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:57.717103 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:57.703218 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c97faba-9728-4c13-b99d-19fa20b7e85b-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:57.783356 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:57.783319 4363 generic.go:356] "Generic (PLEG): container finished" podID="7c97faba-9728-4c13-b99d-19fa20b7e85b" containerID="5a9a235ac5ca4fc6a6d4ce8b7b3d1a1a240b519dded5c4dd5de5e264c334ee9e" exitCode=0 Nov 01 16:32:57.783552 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:57.783422 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" Nov 01 16:32:57.783552 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:57.783414 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" event={"ID":"7c97faba-9728-4c13-b99d-19fa20b7e85b","Type":"ContainerDied","Data":"5a9a235ac5ca4fc6a6d4ce8b7b3d1a1a240b519dded5c4dd5de5e264c334ee9e"} Nov 01 16:32:57.783661 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:57.783569 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h" event={"ID":"7c97faba-9728-4c13-b99d-19fa20b7e85b","Type":"ContainerDied","Data":"fdf0b4bdd4f6944f0868bdc636ba984adb0d494fe79c181b74370d8d23ac46ac"} Nov 01 16:32:57.783661 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:57.783612 4363 scope.go:122] "RemoveContainer" containerID="5a9a235ac5ca4fc6a6d4ce8b7b3d1a1a240b519dded5c4dd5de5e264c334ee9e" Nov 01 16:32:57.797747 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:57.797645 4363 scope.go:122] "RemoveContainer" containerID="be5a939df28fce2552f2b99c55ede881a6d3f0bdbfd0604f45cfd72a5ee4eaa3" Nov 01 16:32:57.820432 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:57.820403 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h"] Nov 01 16:32:57.827242 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:57.827220 4363 scope.go:122] "RemoveContainer" containerID="5a9a235ac5ca4fc6a6d4ce8b7b3d1a1a240b519dded5c4dd5de5e264c334ee9e" Nov 01 16:32:57.827675 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:32:57.827650 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a9a235ac5ca4fc6a6d4ce8b7b3d1a1a240b519dded5c4dd5de5e264c334ee9e\": container with ID starting with 5a9a235ac5ca4fc6a6d4ce8b7b3d1a1a240b519dded5c4dd5de5e264c334ee9e not found: ID does not exist" containerID="5a9a235ac5ca4fc6a6d4ce8b7b3d1a1a240b519dded5c4dd5de5e264c334ee9e" Nov 01 16:32:57.827803 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:57.827682 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a9a235ac5ca4fc6a6d4ce8b7b3d1a1a240b519dded5c4dd5de5e264c334ee9e"} err="failed to get container status \"5a9a235ac5ca4fc6a6d4ce8b7b3d1a1a240b519dded5c4dd5de5e264c334ee9e\": rpc error: code = NotFound desc = could not find container \"5a9a235ac5ca4fc6a6d4ce8b7b3d1a1a240b519dded5c4dd5de5e264c334ee9e\": container with ID starting with 5a9a235ac5ca4fc6a6d4ce8b7b3d1a1a240b519dded5c4dd5de5e264c334ee9e not found: ID does not exist" Nov 01 16:32:57.827803 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:57.827701 4363 scope.go:122] "RemoveContainer" containerID="be5a939df28fce2552f2b99c55ede881a6d3f0bdbfd0604f45cfd72a5ee4eaa3" Nov 01 16:32:57.828063 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:32:57.828041 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be5a939df28fce2552f2b99c55ede881a6d3f0bdbfd0604f45cfd72a5ee4eaa3\": container with ID starting with be5a939df28fce2552f2b99c55ede881a6d3f0bdbfd0604f45cfd72a5ee4eaa3 not found: ID does not exist" containerID="be5a939df28fce2552f2b99c55ede881a6d3f0bdbfd0604f45cfd72a5ee4eaa3" Nov 01 16:32:57.828150 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:57.828070 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be5a939df28fce2552f2b99c55ede881a6d3f0bdbfd0604f45cfd72a5ee4eaa3"} err="failed to get container status \"be5a939df28fce2552f2b99c55ede881a6d3f0bdbfd0604f45cfd72a5ee4eaa3\": rpc error: code = NotFound desc = could not find container \"be5a939df28fce2552f2b99c55ede881a6d3f0bdbfd0604f45cfd72a5ee4eaa3\": container with ID starting with be5a939df28fce2552f2b99c55ede881a6d3f0bdbfd0604f45cfd72a5ee4eaa3 not found: ID does not exist" Nov 01 16:32:57.829880 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:57.829856 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-aa6303-openshift-default-6cc7c47b85-ccx5h"] Nov 01 16:32:58.798647 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:58.798615 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c97faba-9728-4c13-b99d-19fa20b7e85b" path="/var/lib/kubelet/pods/7c97faba-9728-4c13-b99d-19fa20b7e85b/volumes" Nov 01 16:32:59.777043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:59.776988 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv"] Nov 01 16:32:59.777227 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:32:59.777205 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" podUID="fcd61588-08cf-49c1-9dab-febc5c612ca8" containerName="istio-proxy" containerID="cri-o://1b6fd1f6379c2ca60d8ee0af7b8bd2dd0870bec9c1f4cc2e89721e16010d4d1b" gracePeriod=30 Nov 01 16:33:00.493426 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:00.493385 4363 patch_prober.go:29] interesting pod/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:33:00.493820 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:00.493455 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" podUID="fcd61588-08cf-49c1-9dab-febc5c612ca8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:33:03.601376 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:03.601338 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-501d63-d548b6d79-w9jxd"] Nov 01 16:33:03.616926 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:03.616852 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-6689474455-mhl6q"] Nov 01 16:33:04.627951 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:04.627914 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-9a482b/rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl" podUID="190bf9fe-551d-4f76-8709-b637ff81d4a1" containerName="main" containerID="cri-o://6144fdc0eede1c2cb6f369996f14e9c4ac1a5e2cd85690d87a71143871fc8347" gracePeriod=45 Nov 01 16:33:04.705241 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:04.705200 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-734859-5fc5fb544f-n2j8r" podUID="2970538a-8b72-47f6-a76b-455f828c8d83" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:33:04.917751 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:04.917532 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-9a482b-86b559b58f-frnp7"] Nov 01 16:33:04.935644 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:04.935115 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9a482b/rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl" Nov 01 16:33:04.938660 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:04.938626 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a482b-79b5ff7995-5zhdr"] Nov 01 16:33:04.944450 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:04.943869 4363 generic.go:356] "Generic (PLEG): container finished" podID="190bf9fe-551d-4f76-8709-b637ff81d4a1" containerID="6144fdc0eede1c2cb6f369996f14e9c4ac1a5e2cd85690d87a71143871fc8347" exitCode=0 Nov 01 16:33:04.944450 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:04.944036 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9a482b/rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl" event={"ID":"190bf9fe-551d-4f76-8709-b637ff81d4a1","Type":"ContainerDied","Data":"6144fdc0eede1c2cb6f369996f14e9c4ac1a5e2cd85690d87a71143871fc8347"} Nov 01 16:33:04.944450 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:04.944076 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9a482b/rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl" event={"ID":"190bf9fe-551d-4f76-8709-b637ff81d4a1","Type":"ContainerDied","Data":"ca4b8a90cc6b5381ab224b2b4608cde76d9e96feccbd1c7a29796f06fa3a7b76"} Nov 01 16:33:04.944450 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:04.944110 4363 scope.go:122] "RemoveContainer" containerID="6144fdc0eede1c2cb6f369996f14e9c4ac1a5e2cd85690d87a71143871fc8347" Nov 01 16:33:04.944450 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:04.944284 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9a482b/rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl" Nov 01 16:33:04.955355 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:04.955229 4363 generic.go:356] "Generic (PLEG): container finished" podID="fcd61588-08cf-49c1-9dab-febc5c612ca8" containerID="1b6fd1f6379c2ca60d8ee0af7b8bd2dd0870bec9c1f4cc2e89721e16010d4d1b" exitCode=0 Nov 01 16:33:04.956191 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:04.955606 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" event={"ID":"fcd61588-08cf-49c1-9dab-febc5c612ca8","Type":"ContainerDied","Data":"1b6fd1f6379c2ca60d8ee0af7b8bd2dd0870bec9c1f4cc2e89721e16010d4d1b"} Nov 01 16:33:04.980559 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:04.980534 4363 scope.go:122] "RemoveContainer" containerID="6144fdc0eede1c2cb6f369996f14e9c4ac1a5e2cd85690d87a71143871fc8347" Nov 01 16:33:04.980947 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:33:04.980922 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6144fdc0eede1c2cb6f369996f14e9c4ac1a5e2cd85690d87a71143871fc8347\": container with ID starting with 6144fdc0eede1c2cb6f369996f14e9c4ac1a5e2cd85690d87a71143871fc8347 not found: ID does not exist" containerID="6144fdc0eede1c2cb6f369996f14e9c4ac1a5e2cd85690d87a71143871fc8347" Nov 01 16:33:04.981083 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:04.980960 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6144fdc0eede1c2cb6f369996f14e9c4ac1a5e2cd85690d87a71143871fc8347"} err="failed to get container status \"6144fdc0eede1c2cb6f369996f14e9c4ac1a5e2cd85690d87a71143871fc8347\": rpc error: code = NotFound desc = could not find container \"6144fdc0eede1c2cb6f369996f14e9c4ac1a5e2cd85690d87a71143871fc8347\": container with ID starting with 6144fdc0eede1c2cb6f369996f14e9c4ac1a5e2cd85690d87a71143871fc8347 not found: ID does not exist" Nov 01 16:33:05.007948 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.007912 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/190bf9fe-551d-4f76-8709-b637ff81d4a1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/190bf9fe-551d-4f76-8709-b637ff81d4a1-tmp-dir\") pod \"190bf9fe-551d-4f76-8709-b637ff81d4a1\" (UID: \"190bf9fe-551d-4f76-8709-b637ff81d4a1\") " Nov 01 16:33:05.008590 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.008561 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/190bf9fe-551d-4f76-8709-b637ff81d4a1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/190bf9fe-551d-4f76-8709-b637ff81d4a1-model-cache\") pod \"190bf9fe-551d-4f76-8709-b637ff81d4a1\" (UID: \"190bf9fe-551d-4f76-8709-b637ff81d4a1\") " Nov 01 16:33:05.008903 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.008879 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/190bf9fe-551d-4f76-8709-b637ff81d4a1-kube-api-access-jz8x9\" (UniqueName: \"kubernetes.io/projected/190bf9fe-551d-4f76-8709-b637ff81d4a1-kube-api-access-jz8x9\") pod \"190bf9fe-551d-4f76-8709-b637ff81d4a1\" (UID: \"190bf9fe-551d-4f76-8709-b637ff81d4a1\") " Nov 01 16:33:05.009423 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.009402 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/190bf9fe-551d-4f76-8709-b637ff81d4a1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/190bf9fe-551d-4f76-8709-b637ff81d4a1-dshm\") pod \"190bf9fe-551d-4f76-8709-b637ff81d4a1\" (UID: \"190bf9fe-551d-4f76-8709-b637ff81d4a1\") " Nov 01 16:33:05.009766 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.009747 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/190bf9fe-551d-4f76-8709-b637ff81d4a1-home\" (UniqueName: \"kubernetes.io/empty-dir/190bf9fe-551d-4f76-8709-b637ff81d4a1-home\") pod \"190bf9fe-551d-4f76-8709-b637ff81d4a1\" (UID: \"190bf9fe-551d-4f76-8709-b637ff81d4a1\") " Nov 01 16:33:05.009911 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.009896 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/190bf9fe-551d-4f76-8709-b637ff81d4a1-tls-certs\" (UniqueName: \"kubernetes.io/secret/190bf9fe-551d-4f76-8709-b637ff81d4a1-tls-certs\") pod \"190bf9fe-551d-4f76-8709-b637ff81d4a1\" (UID: \"190bf9fe-551d-4f76-8709-b637ff81d4a1\") " Nov 01 16:33:05.010969 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.008436 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/190bf9fe-551d-4f76-8709-b637ff81d4a1-tmp-dir" pod "190bf9fe-551d-4f76-8709-b637ff81d4a1" (UID: "190bf9fe-551d-4f76-8709-b637ff81d4a1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:05.011157 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.008798 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/190bf9fe-551d-4f76-8709-b637ff81d4a1-model-cache" pod "190bf9fe-551d-4f76-8709-b637ff81d4a1" (UID: "190bf9fe-551d-4f76-8709-b637ff81d4a1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:05.011367 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.011341 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/190bf9fe-551d-4f76-8709-b637ff81d4a1-home" pod "190bf9fe-551d-4f76-8709-b637ff81d4a1" (UID: "190bf9fe-551d-4f76-8709-b637ff81d4a1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:05.012884 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.012849 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/190bf9fe-551d-4f76-8709-b637ff81d4a1-kube-api-access-jz8x9" pod "190bf9fe-551d-4f76-8709-b637ff81d4a1" (UID: "190bf9fe-551d-4f76-8709-b637ff81d4a1"). InnerVolumeSpecName "kube-api-access-jz8x9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:05.013144 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.013108 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/190bf9fe-551d-4f76-8709-b637ff81d4a1-tls-certs" pod "190bf9fe-551d-4f76-8709-b637ff81d4a1" (UID: "190bf9fe-551d-4f76-8709-b637ff81d4a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:33:05.014682 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.014654 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/190bf9fe-551d-4f76-8709-b637ff81d4a1-dshm" pod "190bf9fe-551d-4f76-8709-b637ff81d4a1" (UID: "190bf9fe-551d-4f76-8709-b637ff81d4a1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:05.131058 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.112140 4363 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/190bf9fe-551d-4f76-8709-b637ff81d4a1-model-cache\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:05.131058 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.112184 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz8x9\" (UniqueName: \"kubernetes.io/projected/190bf9fe-551d-4f76-8709-b637ff81d4a1-kube-api-access-jz8x9\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:05.131058 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.112200 4363 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/190bf9fe-551d-4f76-8709-b637ff81d4a1-dshm\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:05.131058 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.112214 4363 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/190bf9fe-551d-4f76-8709-b637ff81d4a1-home\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:05.131058 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.112227 4363 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/190bf9fe-551d-4f76-8709-b637ff81d4a1-tls-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:05.131058 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.112240 4363 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/190bf9fe-551d-4f76-8709-b637ff81d4a1-tmp-dir\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:05.135946 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.135886 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm"] Nov 01 16:33:05.298269 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.298233 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" Nov 01 16:33:05.314037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.313985 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9a482b/rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl"] Nov 01 16:33:05.325880 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.325841 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9a482b/rate-test-model-9a482b-kserve-78bf7fc8b9-tfbpl"] Nov 01 16:33:05.430045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.430005 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fcd61588-08cf-49c1-9dab-febc5c612ca8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcd61588-08cf-49c1-9dab-febc5c612ca8-istio-podinfo\") pod \"fcd61588-08cf-49c1-9dab-febc5c612ca8\" (UID: \"fcd61588-08cf-49c1-9dab-febc5c612ca8\") " Nov 01 16:33:05.430227 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.430168 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcd61588-08cf-49c1-9dab-febc5c612ca8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcd61588-08cf-49c1-9dab-febc5c612ca8-istiod-ca-cert\") pod \"fcd61588-08cf-49c1-9dab-febc5c612ca8\" (UID: \"fcd61588-08cf-49c1-9dab-febc5c612ca8\") " Nov 01 16:33:05.430293 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.430232 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcd61588-08cf-49c1-9dab-febc5c612ca8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd61588-08cf-49c1-9dab-febc5c612ca8-workload-socket\") pod \"fcd61588-08cf-49c1-9dab-febc5c612ca8\" (UID: \"fcd61588-08cf-49c1-9dab-febc5c612ca8\") " Nov 01 16:33:05.431369 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.430407 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcd61588-08cf-49c1-9dab-febc5c612ca8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcd61588-08cf-49c1-9dab-febc5c612ca8-istio-envoy\") pod \"fcd61588-08cf-49c1-9dab-febc5c612ca8\" (UID: \"fcd61588-08cf-49c1-9dab-febc5c612ca8\") " Nov 01 16:33:05.431369 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.430465 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcd61588-08cf-49c1-9dab-febc5c612ca8-kube-api-access-662ph\" (UniqueName: \"kubernetes.io/projected/fcd61588-08cf-49c1-9dab-febc5c612ca8-kube-api-access-662ph\") pod \"fcd61588-08cf-49c1-9dab-febc5c612ca8\" (UID: \"fcd61588-08cf-49c1-9dab-febc5c612ca8\") " Nov 01 16:33:05.431369 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.430485 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcd61588-08cf-49c1-9dab-febc5c612ca8-workload-socket" pod "fcd61588-08cf-49c1-9dab-febc5c612ca8" (UID: "fcd61588-08cf-49c1-9dab-febc5c612ca8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:05.431369 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.430530 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcd61588-08cf-49c1-9dab-febc5c612ca8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcd61588-08cf-49c1-9dab-febc5c612ca8-istio-data\") pod \"fcd61588-08cf-49c1-9dab-febc5c612ca8\" (UID: \"fcd61588-08cf-49c1-9dab-febc5c612ca8\") " Nov 01 16:33:05.431369 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.430582 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcd61588-08cf-49c1-9dab-febc5c612ca8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcd61588-08cf-49c1-9dab-febc5c612ca8-workload-certs\") pod \"fcd61588-08cf-49c1-9dab-febc5c612ca8\" (UID: \"fcd61588-08cf-49c1-9dab-febc5c612ca8\") " Nov 01 16:33:05.431369 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.430595 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcd61588-08cf-49c1-9dab-febc5c612ca8-istiod-ca-cert" pod "fcd61588-08cf-49c1-9dab-febc5c612ca8" (UID: "fcd61588-08cf-49c1-9dab-febc5c612ca8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:05.431369 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.430689 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcd61588-08cf-49c1-9dab-febc5c612ca8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd61588-08cf-49c1-9dab-febc5c612ca8-credential-socket\") pod \"fcd61588-08cf-49c1-9dab-febc5c612ca8\" (UID: \"fcd61588-08cf-49c1-9dab-febc5c612ca8\") " Nov 01 16:33:05.431369 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.430729 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcd61588-08cf-49c1-9dab-febc5c612ca8-istio-token\" (UniqueName: \"kubernetes.io/projected/fcd61588-08cf-49c1-9dab-febc5c612ca8-istio-token\") pod \"fcd61588-08cf-49c1-9dab-febc5c612ca8\" (UID: \"fcd61588-08cf-49c1-9dab-febc5c612ca8\") " Nov 01 16:33:05.431369 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.431311 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcd61588-08cf-49c1-9dab-febc5c612ca8-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:05.431369 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.431334 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd61588-08cf-49c1-9dab-febc5c612ca8-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:05.431814 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.431375 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcd61588-08cf-49c1-9dab-febc5c612ca8-workload-certs" pod "fcd61588-08cf-49c1-9dab-febc5c612ca8" (UID: "fcd61588-08cf-49c1-9dab-febc5c612ca8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:05.431814 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.431775 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcd61588-08cf-49c1-9dab-febc5c612ca8-istio-data" pod "fcd61588-08cf-49c1-9dab-febc5c612ca8" (UID: "fcd61588-08cf-49c1-9dab-febc5c612ca8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:05.431937 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.431835 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcd61588-08cf-49c1-9dab-febc5c612ca8-credential-socket" pod "fcd61588-08cf-49c1-9dab-febc5c612ca8" (UID: "fcd61588-08cf-49c1-9dab-febc5c612ca8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:05.432852 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.432818 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcd61588-08cf-49c1-9dab-febc5c612ca8-istio-envoy" pod "fcd61588-08cf-49c1-9dab-febc5c612ca8" (UID: "fcd61588-08cf-49c1-9dab-febc5c612ca8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:05.433156 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.433122 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcd61588-08cf-49c1-9dab-febc5c612ca8-kube-api-access-662ph" pod "fcd61588-08cf-49c1-9dab-febc5c612ca8" (UID: "fcd61588-08cf-49c1-9dab-febc5c612ca8"). InnerVolumeSpecName "kube-api-access-662ph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:05.433893 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.433865 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcd61588-08cf-49c1-9dab-febc5c612ca8-istio-token" pod "fcd61588-08cf-49c1-9dab-febc5c612ca8" (UID: "fcd61588-08cf-49c1-9dab-febc5c612ca8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:05.433981 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.433955 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fcd61588-08cf-49c1-9dab-febc5c612ca8-istio-podinfo" pod "fcd61588-08cf-49c1-9dab-febc5c612ca8" (UID: "fcd61588-08cf-49c1-9dab-febc5c612ca8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:33:05.532847 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.532774 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcd61588-08cf-49c1-9dab-febc5c612ca8-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:05.532847 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.532804 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-662ph\" (UniqueName: \"kubernetes.io/projected/fcd61588-08cf-49c1-9dab-febc5c612ca8-kube-api-access-662ph\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:05.532847 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.532815 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcd61588-08cf-49c1-9dab-febc5c612ca8-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:05.532847 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.532823 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcd61588-08cf-49c1-9dab-febc5c612ca8-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:05.532847 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.532832 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd61588-08cf-49c1-9dab-febc5c612ca8-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:05.532847 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.532841 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcd61588-08cf-49c1-9dab-febc5c612ca8-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:05.532847 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.532849 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcd61588-08cf-49c1-9dab-febc5c612ca8-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:05.719001 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.718961 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-734859-589cbc787-xqhjc" podUID="c56804c0-0810-4106-b393-d292cb816780" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:33:05.964568 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.964540 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" Nov 01 16:33:05.964740 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.964575 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv" event={"ID":"fcd61588-08cf-49c1-9dab-febc5c612ca8","Type":"ContainerDied","Data":"c70e4c82de8d2dbd78948603489ed46a97974541d015f1b49070e0f34cae135f"} Nov 01 16:33:05.964740 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.964648 4363 scope.go:122] "RemoveContainer" containerID="1b6fd1f6379c2ca60d8ee0af7b8bd2dd0870bec9c1f4cc2e89721e16010d4d1b" Nov 01 16:33:05.998121 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:05.998091 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv"] Nov 01 16:33:06.006371 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:06.006344 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-fxjdv"] Nov 01 16:33:06.799190 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:06.799155 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="190bf9fe-551d-4f76-8709-b637ff81d4a1" path="/var/lib/kubelet/pods/190bf9fe-551d-4f76-8709-b637ff81d4a1/volumes" Nov 01 16:33:06.799754 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:06.799737 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcd61588-08cf-49c1-9dab-febc5c612ca8" path="/var/lib/kubelet/pods/fcd61588-08cf-49c1-9dab-febc5c612ca8/volumes" Nov 01 16:33:08.608151 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:08.608098 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-501d63-d548b6d79-w9jxd" podUID="f70ac519-b330-4122-b106-2269b0bfd261" containerName="payload-processing" containerID="cri-o://7e1b1fc826f95e248d4902c7a7d46f804e7ce8be8a1005c0d54073f12e3786e0" gracePeriod=25 Nov 01 16:33:08.621177 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:08.618522 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-6689474455-mhl6q" podUID="4301c73d-6b01-40b7-a4b7-c8d4384520ca" containerName="payload-pre-processing" containerID="cri-o://6161c68e95cc62d641d6b9794f70d556f5a51a888e2755b4bba58cb66e080f3c" gracePeriod=25 Nov 01 16:33:08.761664 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:08.761283 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-734859-5fc5fb544f-n2j8r" Nov 01 16:33:08.888734 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:08.888698 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2970538a-8b72-47f6-a76b-455f828c8d83-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2970538a-8b72-47f6-a76b-455f828c8d83-plugins-config-volume\") pod \"2970538a-8b72-47f6-a76b-455f828c8d83\" (UID: \"2970538a-8b72-47f6-a76b-455f828c8d83\") " Nov 01 16:33:08.888927 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:08.888885 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2970538a-8b72-47f6-a76b-455f828c8d83-kube-api-access-bcbmq\" (UniqueName: \"kubernetes.io/projected/2970538a-8b72-47f6-a76b-455f828c8d83-kube-api-access-bcbmq\") pod \"2970538a-8b72-47f6-a76b-455f828c8d83\" (UID: \"2970538a-8b72-47f6-a76b-455f828c8d83\") " Nov 01 16:33:08.894044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:08.890819 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2970538a-8b72-47f6-a76b-455f828c8d83-plugins-config-volume" pod "2970538a-8b72-47f6-a76b-455f828c8d83" (UID: "2970538a-8b72-47f6-a76b-455f828c8d83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:08.907632 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:08.907564 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-734859-589cbc787-xqhjc" Nov 01 16:33:08.921070 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:08.919815 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2970538a-8b72-47f6-a76b-455f828c8d83-kube-api-access-bcbmq" pod "2970538a-8b72-47f6-a76b-455f828c8d83" (UID: "2970538a-8b72-47f6-a76b-455f828c8d83"). InnerVolumeSpecName "kube-api-access-bcbmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:09.028203 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:08.995156 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcbmq\" (UniqueName: \"kubernetes.io/projected/2970538a-8b72-47f6-a76b-455f828c8d83-kube-api-access-bcbmq\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:09.028203 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:08.995194 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2970538a-8b72-47f6-a76b-455f828c8d83-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:09.115049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:09.111972 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c56804c0-0810-4106-b393-d292cb816780-kube-api-access-5vwds\" (UniqueName: \"kubernetes.io/projected/c56804c0-0810-4106-b393-d292cb816780-kube-api-access-5vwds\") pod \"c56804c0-0810-4106-b393-d292cb816780\" (UID: \"c56804c0-0810-4106-b393-d292cb816780\") " Nov 01 16:33:09.115049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:09.112107 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c56804c0-0810-4106-b393-d292cb816780-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c56804c0-0810-4106-b393-d292cb816780-plugins-config-volume\") pod \"c56804c0-0810-4106-b393-d292cb816780\" (UID: \"c56804c0-0810-4106-b393-d292cb816780\") " Nov 01 16:33:09.174088 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:09.138643 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c56804c0-0810-4106-b393-d292cb816780-plugins-config-volume" pod "c56804c0-0810-4106-b393-d292cb816780" (UID: "c56804c0-0810-4106-b393-d292cb816780"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:09.185948 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:09.184418 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c56804c0-0810-4106-b393-d292cb816780-kube-api-access-5vwds" pod "c56804c0-0810-4106-b393-d292cb816780" (UID: "c56804c0-0810-4106-b393-d292cb816780"). InnerVolumeSpecName "kube-api-access-5vwds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:09.191313 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:09.191285 4363 generic.go:356] "Generic (PLEG): container finished" podID="f70ac519-b330-4122-b106-2269b0bfd261" containerID="7e1b1fc826f95e248d4902c7a7d46f804e7ce8be8a1005c0d54073f12e3786e0" exitCode=0 Nov 01 16:33:09.191505 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:09.191354 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-501d63-d548b6d79-w9jxd" event={"ID":"f70ac519-b330-4122-b106-2269b0bfd261","Type":"ContainerDied","Data":"7e1b1fc826f95e248d4902c7a7d46f804e7ce8be8a1005c0d54073f12e3786e0"} Nov 01 16:33:09.194633 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:09.194610 4363 generic.go:356] "Generic (PLEG): container finished" podID="c56804c0-0810-4106-b393-d292cb816780" containerID="26a7ab71a9f9ca0a81874a278ac144ca1c17b3cb9a8250783630073d54b2e1cd" exitCode=0 Nov 01 16:33:09.194939 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:09.194924 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-734859-589cbc787-xqhjc" Nov 01 16:33:09.200515 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:09.200349 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-734859-589cbc787-xqhjc" event={"ID":"c56804c0-0810-4106-b393-d292cb816780","Type":"ContainerDied","Data":"26a7ab71a9f9ca0a81874a278ac144ca1c17b3cb9a8250783630073d54b2e1cd"} Nov 01 16:33:09.200515 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:09.200402 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-734859-589cbc787-xqhjc" event={"ID":"c56804c0-0810-4106-b393-d292cb816780","Type":"ContainerDied","Data":"efd65cc2489c9b60beb7c024858ceb8e39193fd2dca46820a60aab3f668f91e7"} Nov 01 16:33:09.200515 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:09.200431 4363 scope.go:122] "RemoveContainer" containerID="26a7ab71a9f9ca0a81874a278ac144ca1c17b3cb9a8250783630073d54b2e1cd" Nov 01 16:33:09.213734 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:09.213676 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vwds\" (UniqueName: \"kubernetes.io/projected/c56804c0-0810-4106-b393-d292cb816780-kube-api-access-5vwds\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:09.213734 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:09.213705 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c56804c0-0810-4106-b393-d292cb816780-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:09.219076 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:09.218649 4363 generic.go:356] "Generic (PLEG): container finished" podID="2970538a-8b72-47f6-a76b-455f828c8d83" containerID="9a2cfd0a9e4a231492630b9e50963f0372fd1dd18284b3f34d3af9e267f00136" exitCode=0 Nov 01 16:33:09.219076 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:09.218832 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-734859-5fc5fb544f-n2j8r" event={"ID":"2970538a-8b72-47f6-a76b-455f828c8d83","Type":"ContainerDied","Data":"9a2cfd0a9e4a231492630b9e50963f0372fd1dd18284b3f34d3af9e267f00136"} Nov 01 16:33:09.219076 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:09.218878 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-734859-5fc5fb544f-n2j8r" event={"ID":"2970538a-8b72-47f6-a76b-455f828c8d83","Type":"ContainerDied","Data":"caa805cc66c2359cb820fe84b4b8b735b78a94a5f87ced028190cc21740bbc69"} Nov 01 16:33:09.219076 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:09.218959 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-734859-5fc5fb544f-n2j8r" Nov 01 16:33:09.247633 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:09.247454 4363 generic.go:356] "Generic (PLEG): container finished" podID="4301c73d-6b01-40b7-a4b7-c8d4384520ca" containerID="6161c68e95cc62d641d6b9794f70d556f5a51a888e2755b4bba58cb66e080f3c" exitCode=0 Nov 01 16:33:09.247633 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:09.247548 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-6689474455-mhl6q" event={"ID":"4301c73d-6b01-40b7-a4b7-c8d4384520ca","Type":"ContainerDied","Data":"6161c68e95cc62d641d6b9794f70d556f5a51a888e2755b4bba58cb66e080f3c"} Nov 01 16:33:09.257264 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:09.257078 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-6689474455-mhl6q" Nov 01 16:33:09.274957 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:09.274924 4363 scope.go:122] "RemoveContainer" containerID="26a7ab71a9f9ca0a81874a278ac144ca1c17b3cb9a8250783630073d54b2e1cd" Nov 01 16:33:09.275855 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:33:09.275712 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26a7ab71a9f9ca0a81874a278ac144ca1c17b3cb9a8250783630073d54b2e1cd\": container with ID starting with 26a7ab71a9f9ca0a81874a278ac144ca1c17b3cb9a8250783630073d54b2e1cd not found: ID does not exist" containerID="26a7ab71a9f9ca0a81874a278ac144ca1c17b3cb9a8250783630073d54b2e1cd" Nov 01 16:33:09.275855 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:09.275752 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26a7ab71a9f9ca0a81874a278ac144ca1c17b3cb9a8250783630073d54b2e1cd"} err="failed to get container status \"26a7ab71a9f9ca0a81874a278ac144ca1c17b3cb9a8250783630073d54b2e1cd\": rpc error: code = NotFound desc = could not find container \"26a7ab71a9f9ca0a81874a278ac144ca1c17b3cb9a8250783630073d54b2e1cd\": container with ID starting with 26a7ab71a9f9ca0a81874a278ac144ca1c17b3cb9a8250783630073d54b2e1cd not found: ID does not exist" Nov 01 16:33:09.275855 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:09.275777 4363 scope.go:122] "RemoveContainer" containerID="9a2cfd0a9e4a231492630b9e50963f0372fd1dd18284b3f34d3af9e267f00136" Nov 01 16:33:09.463280 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:09.463253 4363 scope.go:122] "RemoveContainer" containerID="9a2cfd0a9e4a231492630b9e50963f0372fd1dd18284b3f34d3af9e267f00136" Nov 01 16:33:09.464058 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:33:09.464011 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a2cfd0a9e4a231492630b9e50963f0372fd1dd18284b3f34d3af9e267f00136\": container with ID starting with 9a2cfd0a9e4a231492630b9e50963f0372fd1dd18284b3f34d3af9e267f00136 not found: ID does not exist" containerID="9a2cfd0a9e4a231492630b9e50963f0372fd1dd18284b3f34d3af9e267f00136" Nov 01 16:33:09.464269 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:09.464232 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a2cfd0a9e4a231492630b9e50963f0372fd1dd18284b3f34d3af9e267f00136"} err="failed to get container status \"9a2cfd0a9e4a231492630b9e50963f0372fd1dd18284b3f34d3af9e267f00136\": rpc error: code = NotFound desc = could not find container \"9a2cfd0a9e4a231492630b9e50963f0372fd1dd18284b3f34d3af9e267f00136\": container with ID starting with 9a2cfd0a9e4a231492630b9e50963f0372fd1dd18284b3f34d3af9e267f00136 not found: ID does not exist" Nov 01 16:33:09.496006 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:09.495949 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-734859-5fc5fb544f-n2j8r"] Nov 01 16:33:09.509350 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:09.503341 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-734859-5fc5fb544f-n2j8r"] Nov 01 16:33:09.509350 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:09.504065 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-501d63-d548b6d79-w9jxd" Nov 01 16:33:09.520057 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:09.517312 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-734859-589cbc787-xqhjc"] Nov 01 16:33:09.520057 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:09.517388 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-734859-589cbc787-xqhjc"] Nov 01 16:33:09.546043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:09.544393 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4301c73d-6b01-40b7-a4b7-c8d4384520ca-kube-api-access-bhgp5\" (UniqueName: \"kubernetes.io/projected/4301c73d-6b01-40b7-a4b7-c8d4384520ca-kube-api-access-bhgp5\") pod \"4301c73d-6b01-40b7-a4b7-c8d4384520ca\" (UID: \"4301c73d-6b01-40b7-a4b7-c8d4384520ca\") " Nov 01 16:33:09.546043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:09.544504 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4301c73d-6b01-40b7-a4b7-c8d4384520ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4301c73d-6b01-40b7-a4b7-c8d4384520ca-plugins-config-volume\") pod \"4301c73d-6b01-40b7-a4b7-c8d4384520ca\" (UID: \"4301c73d-6b01-40b7-a4b7-c8d4384520ca\") " Nov 01 16:33:09.546043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:09.544628 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f70ac519-b330-4122-b106-2269b0bfd261-kube-api-access-2hqrq\" (UniqueName: \"kubernetes.io/projected/f70ac519-b330-4122-b106-2269b0bfd261-kube-api-access-2hqrq\") pod \"f70ac519-b330-4122-b106-2269b0bfd261\" (UID: \"f70ac519-b330-4122-b106-2269b0bfd261\") " Nov 01 16:33:09.546043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:09.544742 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f70ac519-b330-4122-b106-2269b0bfd261-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f70ac519-b330-4122-b106-2269b0bfd261-plugins-config-volume\") pod \"f70ac519-b330-4122-b106-2269b0bfd261\" (UID: \"f70ac519-b330-4122-b106-2269b0bfd261\") " Nov 01 16:33:09.546043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:09.545533 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4301c73d-6b01-40b7-a4b7-c8d4384520ca-plugins-config-volume" pod "4301c73d-6b01-40b7-a4b7-c8d4384520ca" (UID: "4301c73d-6b01-40b7-a4b7-c8d4384520ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:09.546043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:09.545976 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f70ac519-b330-4122-b106-2269b0bfd261-plugins-config-volume" pod "f70ac519-b330-4122-b106-2269b0bfd261" (UID: "f70ac519-b330-4122-b106-2269b0bfd261"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:09.552718 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:09.550212 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f70ac519-b330-4122-b106-2269b0bfd261-kube-api-access-2hqrq" pod "f70ac519-b330-4122-b106-2269b0bfd261" (UID: "f70ac519-b330-4122-b106-2269b0bfd261"). InnerVolumeSpecName "kube-api-access-2hqrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:09.552718 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:09.550246 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4301c73d-6b01-40b7-a4b7-c8d4384520ca-kube-api-access-bhgp5" pod "4301c73d-6b01-40b7-a4b7-c8d4384520ca" (UID: "4301c73d-6b01-40b7-a4b7-c8d4384520ca"). InnerVolumeSpecName "kube-api-access-bhgp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:09.658551 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:09.658471 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhgp5\" (UniqueName: \"kubernetes.io/projected/4301c73d-6b01-40b7-a4b7-c8d4384520ca-kube-api-access-bhgp5\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:09.658551 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:09.658515 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4301c73d-6b01-40b7-a4b7-c8d4384520ca-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:09.658551 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:09.658533 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hqrq\" (UniqueName: \"kubernetes.io/projected/f70ac519-b330-4122-b106-2269b0bfd261-kube-api-access-2hqrq\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:09.658551 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:09.658548 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f70ac519-b330-4122-b106-2269b0bfd261-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:09.918098 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:09.917941 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9a482b-86b559b58f-frnp7" podUID="e533e983-dbc0-4ce2-8d58-c74c0c0d857b" containerName="payload-processing" containerID="cri-o://7b9c924a8440d30cff88c1b74062462a018d785e9e5fa51b35dc3e3c2b251e2a" gracePeriod=25 Nov 01 16:33:09.939158 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:09.939122 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a482b-79b5ff7995-5zhdr" podUID="b8b1f822-bf88-4f11-b621-da764f27c492" containerName="payload-pre-processing" containerID="cri-o://30756e3465ba1fd2331edb2ab06ca3d66dd6dbe07de218a977971bdf4fd5be48" gracePeriod=25 Nov 01 16:33:10.256423 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:10.256339 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-501d63-d548b6d79-w9jxd" Nov 01 16:33:10.256423 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:10.256341 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-501d63-d548b6d79-w9jxd" event={"ID":"f70ac519-b330-4122-b106-2269b0bfd261","Type":"ContainerDied","Data":"37bb80a8104b366620eb28b469231dff006aecd7e9b79761b9da28760b79edc3"} Nov 01 16:33:10.256667 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:10.256480 4363 scope.go:122] "RemoveContainer" containerID="7e1b1fc826f95e248d4902c7a7d46f804e7ce8be8a1005c0d54073f12e3786e0" Nov 01 16:33:10.260775 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:10.260750 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-6689474455-mhl6q" event={"ID":"4301c73d-6b01-40b7-a4b7-c8d4384520ca","Type":"ContainerDied","Data":"acedfbea0333b30a157957a0a3b43e58eadef2b40a7650c2a65d238013f13696"} Nov 01 16:33:10.260921 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:10.260806 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-6689474455-mhl6q" Nov 01 16:33:10.269658 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:10.269643 4363 scope.go:122] "RemoveContainer" containerID="6161c68e95cc62d641d6b9794f70d556f5a51a888e2755b4bba58cb66e080f3c" Nov 01 16:33:10.306045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:10.305097 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-501d63-d548b6d79-w9jxd"] Nov 01 16:33:10.311198 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:10.311164 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-501d63-d548b6d79-w9jxd"] Nov 01 16:33:10.316233 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:10.316203 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-6689474455-mhl6q"] Nov 01 16:33:10.319036 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:10.318997 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-6689474455-mhl6q"] Nov 01 16:33:10.518350 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:10.518250 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-bd831a-56899656f5-jhvtb"] Nov 01 16:33:10.871523 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:10.871444 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2970538a-8b72-47f6-a76b-455f828c8d83" path="/var/lib/kubelet/pods/2970538a-8b72-47f6-a76b-455f828c8d83/volumes" Nov 01 16:33:10.874153 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:10.872263 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4301c73d-6b01-40b7-a4b7-c8d4384520ca" path="/var/lib/kubelet/pods/4301c73d-6b01-40b7-a4b7-c8d4384520ca/volumes" Nov 01 16:33:10.874153 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:10.872999 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c56804c0-0810-4106-b393-d292cb816780" path="/var/lib/kubelet/pods/c56804c0-0810-4106-b393-d292cb816780/volumes" Nov 01 16:33:10.874153 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:10.873663 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f70ac519-b330-4122-b106-2269b0bfd261" path="/var/lib/kubelet/pods/f70ac519-b330-4122-b106-2269b0bfd261/volumes" Nov 01 16:33:11.014038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.013491 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-678bbff655-z866s"] Nov 01 16:33:11.015047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.014429 4363 cpu_manager.go:414] "containerMap: removing container" podUID="f70ac519-b330-4122-b106-2269b0bfd261" containerName="payload-processing" Nov 01 16:33:11.015047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.014454 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f70ac519-b330-4122-b106-2269b0bfd261" containerName="payload-processing" Nov 01 16:33:11.015047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.014494 4363 cpu_manager.go:414] "containerMap: removing container" podUID="fcd61588-08cf-49c1-9dab-febc5c612ca8" containerName="istio-proxy" Nov 01 16:33:11.015047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.014503 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcd61588-08cf-49c1-9dab-febc5c612ca8" containerName="istio-proxy" Nov 01 16:33:11.015047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.014523 4363 cpu_manager.go:414] "containerMap: removing container" podUID="2970538a-8b72-47f6-a76b-455f828c8d83" containerName="payload-processing" Nov 01 16:33:11.015047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.014532 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2970538a-8b72-47f6-a76b-455f828c8d83" containerName="payload-processing" Nov 01 16:33:11.015047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.014554 4363 cpu_manager.go:414] "containerMap: removing container" podUID="4301c73d-6b01-40b7-a4b7-c8d4384520ca" containerName="payload-pre-processing" Nov 01 16:33:11.015047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.014562 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4301c73d-6b01-40b7-a4b7-c8d4384520ca" containerName="payload-pre-processing" Nov 01 16:33:11.015047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.014572 4363 cpu_manager.go:414] "containerMap: removing container" podUID="c56804c0-0810-4106-b393-d292cb816780" containerName="payload-pre-processing" Nov 01 16:33:11.015047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.014580 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c56804c0-0810-4106-b393-d292cb816780" containerName="payload-pre-processing" Nov 01 16:33:11.015047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.014593 4363 cpu_manager.go:414] "containerMap: removing container" podUID="7c97faba-9728-4c13-b99d-19fa20b7e85b" containerName="istio-proxy" Nov 01 16:33:11.015047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.014601 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c97faba-9728-4c13-b99d-19fa20b7e85b" containerName="istio-proxy" Nov 01 16:33:11.015047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.014622 4363 cpu_manager.go:414] "containerMap: removing container" podUID="7c97faba-9728-4c13-b99d-19fa20b7e85b" containerName="istio-proxy" Nov 01 16:33:11.015047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.014629 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c97faba-9728-4c13-b99d-19fa20b7e85b" containerName="istio-proxy" Nov 01 16:33:11.015047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.014652 4363 cpu_manager.go:414] "containerMap: removing container" podUID="190bf9fe-551d-4f76-8709-b637ff81d4a1" containerName="main" Nov 01 16:33:11.015047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.014659 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="190bf9fe-551d-4f76-8709-b637ff81d4a1" containerName="main" Nov 01 16:33:11.015047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.014778 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="2970538a-8b72-47f6-a76b-455f828c8d83" containerName="payload-processing" Nov 01 16:33:11.015047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.014793 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="4301c73d-6b01-40b7-a4b7-c8d4384520ca" containerName="payload-pre-processing" Nov 01 16:33:11.015047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.014803 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="c56804c0-0810-4106-b393-d292cb816780" containerName="payload-pre-processing" Nov 01 16:33:11.015047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.014816 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c97faba-9728-4c13-b99d-19fa20b7e85b" containerName="istio-proxy" Nov 01 16:33:11.015047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.014831 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c97faba-9728-4c13-b99d-19fa20b7e85b" containerName="istio-proxy" Nov 01 16:33:11.015047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.014845 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="190bf9fe-551d-4f76-8709-b637ff81d4a1" containerName="main" Nov 01 16:33:11.015047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.014855 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="f70ac519-b330-4122-b106-2269b0bfd261" containerName="payload-processing" Nov 01 16:33:11.015047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.014869 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcd61588-08cf-49c1-9dab-febc5c612ca8" containerName="istio-proxy" Nov 01 16:33:11.058131 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.058093 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-678bbff655-z866s"] Nov 01 16:33:11.058322 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.058227 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-678bbff655-z866s" Nov 01 16:33:11.083042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.074181 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-501d63-5fd9cc8bfb-krlt9"] Nov 01 16:33:11.132754 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.132644 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4324048b-b48c-452d-91df-8e48f5e8e97a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-501d63-678bbff655-z866s\" (UID: \"4324048b-b48c-452d-91df-8e48f5e8e97a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-678bbff655-z866s" Nov 01 16:33:11.144671 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.144623 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwglg\" (UniqueName: \"kubernetes.io/projected/4324048b-b48c-452d-91df-8e48f5e8e97a-kube-api-access-pwglg\") pod \"payload-pre-processing-e2e-ipp-a-501d63-678bbff655-z866s\" (UID: \"4324048b-b48c-452d-91df-8e48f5e8e97a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-678bbff655-z866s" Nov 01 16:33:11.183771 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.183631 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-501d63-dockercfg-s57g6\"" Nov 01 16:33:11.183771 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.183737 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-501d63\"" Nov 01 16:33:11.210431 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.200175 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-501d63-5fd9cc8bfb-krlt9" Nov 01 16:33:11.243072 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.242809 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-501d63-5fd9cc8bfb-krlt9"] Nov 01 16:33:11.308151 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.294005 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k69ks\" (UniqueName: \"kubernetes.io/projected/cb31d5ec-ff4f-4fbf-b78f-ab022f851e78-kube-api-access-k69ks\") pod \"payload-processing-e2e-ipp-a-501d63-5fd9cc8bfb-krlt9\" (UID: \"cb31d5ec-ff4f-4fbf-b78f-ab022f851e78\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-501d63-5fd9cc8bfb-krlt9" Nov 01 16:33:11.308336 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.308247 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwglg\" (UniqueName: \"kubernetes.io/projected/4324048b-b48c-452d-91df-8e48f5e8e97a-kube-api-access-pwglg\") pod \"payload-pre-processing-e2e-ipp-a-501d63-678bbff655-z866s\" (UID: \"4324048b-b48c-452d-91df-8e48f5e8e97a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-678bbff655-z866s" Nov 01 16:33:11.308460 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.308428 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb31d5ec-ff4f-4fbf-b78f-ab022f851e78-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-501d63-5fd9cc8bfb-krlt9\" (UID: \"cb31d5ec-ff4f-4fbf-b78f-ab022f851e78\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-501d63-5fd9cc8bfb-krlt9" Nov 01 16:33:11.308534 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.308469 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4324048b-b48c-452d-91df-8e48f5e8e97a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-501d63-678bbff655-z866s\" (UID: \"4324048b-b48c-452d-91df-8e48f5e8e97a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-678bbff655-z866s" Nov 01 16:33:11.309925 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.309895 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4324048b-b48c-452d-91df-8e48f5e8e97a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-501d63-678bbff655-z866s\" (UID: \"4324048b-b48c-452d-91df-8e48f5e8e97a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-678bbff655-z866s" Nov 01 16:33:11.316368 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.316337 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwglg\" (UniqueName: \"kubernetes.io/projected/4324048b-b48c-452d-91df-8e48f5e8e97a-kube-api-access-pwglg\") pod \"payload-pre-processing-e2e-ipp-a-501d63-678bbff655-z866s\" (UID: \"4324048b-b48c-452d-91df-8e48f5e8e97a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-678bbff655-z866s" Nov 01 16:33:11.431092 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.426959 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb31d5ec-ff4f-4fbf-b78f-ab022f851e78-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-501d63-5fd9cc8bfb-krlt9\" (UID: \"cb31d5ec-ff4f-4fbf-b78f-ab022f851e78\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-501d63-5fd9cc8bfb-krlt9" Nov 01 16:33:11.431092 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.427168 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k69ks\" (UniqueName: \"kubernetes.io/projected/cb31d5ec-ff4f-4fbf-b78f-ab022f851e78-kube-api-access-k69ks\") pod \"payload-processing-e2e-ipp-a-501d63-5fd9cc8bfb-krlt9\" (UID: \"cb31d5ec-ff4f-4fbf-b78f-ab022f851e78\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-501d63-5fd9cc8bfb-krlt9" Nov 01 16:33:11.431092 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.428436 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb31d5ec-ff4f-4fbf-b78f-ab022f851e78-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-501d63-5fd9cc8bfb-krlt9\" (UID: \"cb31d5ec-ff4f-4fbf-b78f-ab022f851e78\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-501d63-5fd9cc8bfb-krlt9" Nov 01 16:33:11.459041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.458538 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k69ks\" (UniqueName: \"kubernetes.io/projected/cb31d5ec-ff4f-4fbf-b78f-ab022f851e78-kube-api-access-k69ks\") pod \"payload-processing-e2e-ipp-a-501d63-5fd9cc8bfb-krlt9\" (UID: \"cb31d5ec-ff4f-4fbf-b78f-ab022f851e78\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-501d63-5fd9cc8bfb-krlt9" Nov 01 16:33:11.525353 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.525314 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-678bbff655-z866s" Nov 01 16:33:11.535451 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.535427 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-501d63-5fd9cc8bfb-krlt9" Nov 01 16:33:11.867421 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.867392 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-678bbff655-z866s"] Nov 01 16:33:11.871255 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:11.871229 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-501d63-5fd9cc8bfb-krlt9"] Nov 01 16:33:12.304098 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:12.302836 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-678bbff655-z866s" event={"ID":"4324048b-b48c-452d-91df-8e48f5e8e97a","Type":"ContainerStarted","Data":"8ec4cf5de10cf531018ee05f14532add7ad643104f09334d57e8800d305c34ac"} Nov 01 16:33:12.304098 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:12.302946 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-678bbff655-z866s" event={"ID":"4324048b-b48c-452d-91df-8e48f5e8e97a","Type":"ContainerStarted","Data":"58938c9014f65263d985a957599bbf0997d784570f2825e667465cf6a6f89030"} Nov 01 16:33:12.304098 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:12.303967 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-678bbff655-z866s" Nov 01 16:33:12.312367 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:12.306694 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-501d63-5fd9cc8bfb-krlt9" event={"ID":"cb31d5ec-ff4f-4fbf-b78f-ab022f851e78","Type":"ContainerStarted","Data":"b267c720a2cffea3538cef5a39e37536ff1eb682d81d6e630e423362a02ec8bf"} Nov 01 16:33:12.312650 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:12.312608 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-501d63-5fd9cc8bfb-krlt9" event={"ID":"cb31d5ec-ff4f-4fbf-b78f-ab022f851e78","Type":"ContainerStarted","Data":"b4c2f5f82d1f2cc7c28eb8676ffd1bf63c735c8080f5df91444448c0c8c3636d"} Nov 01 16:33:12.314749 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:12.314731 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-501d63-5fd9cc8bfb-krlt9" Nov 01 16:33:12.332999 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:12.332951 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-678bbff655-z866s" podStartSLOduration=2.332932924 podStartE2EDuration="2.332932924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:33:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:33:12.330411037 +0000 UTC m=+2725.662148690" watchObservedRunningTime="2026-11-01 16:33:12.332932924 +0000 UTC m=+2725.664670577" Nov 01 16:33:12.351344 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:12.351295 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-501d63-5fd9cc8bfb-krlt9" podStartSLOduration=1.351279711 podStartE2EDuration="1.351279711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:33:12.34990559 +0000 UTC m=+2725.681643266" watchObservedRunningTime="2026-11-01 16:33:12.351279711 +0000 UTC m=+2725.683017364" Nov 01 16:33:15.534090 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:15.527168 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bd831a-56899656f5-jhvtb" podUID="bc5eb74e-c5d9-4210-a626-94e8279c3a36" containerName="payload-pre-processing" containerID="cri-o://f1b7825cb139ffad509aba5ec68f675537e4175cd6a5b8bbd89b67c7316cebe0" gracePeriod=25 Nov 01 16:33:17.788162 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:17.788116 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9a482b-86b559b58f-frnp7" podUID="e533e983-dbc0-4ce2-8d58-c74c0c0d857b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:33:18.320044 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:33:18.319990 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:33:18.800235 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:18.800157 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a482b-79b5ff7995-5zhdr" podUID="b8b1f822-bf88-4f11-b621-da764f27c492" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:33:18.868699 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:18.868665 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4"] Nov 01 16:33:18.869205 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:18.869167 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" podUID="5f69b8bb-4aa6-4713-8b2f-138c5d358858" containerName="istio-proxy" containerID="cri-o://5c0cf1794a919c29141a45a01845845fd0431e56c59229210667d72142b0010f" gracePeriod=30 Nov 01 16:33:19.379314 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:19.379286 4363 generic.go:356] "Generic (PLEG): container finished" podID="b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697" containerID="11d85b9dc8a77601e7240682f3127f7464c911c041076576559958705ce20c57" exitCode=0 Nov 01 16:33:19.379529 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:19.379360 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" event={"ID":"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697","Type":"ContainerDied","Data":"11d85b9dc8a77601e7240682f3127f7464c911c041076576559958705ce20c57"} Nov 01 16:33:19.379900 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:19.379882 4363 scope.go:122] "RemoveContainer" containerID="11d85b9dc8a77601e7240682f3127f7464c911c041076576559958705ce20c57" Nov 01 16:33:19.381594 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:19.381563 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:33:19.381716 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:19.381637 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:33:19.381716 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:19.381675 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:33:20.022604 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:20.022574 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6992258b-cx54c"] Nov 01 16:33:20.027870 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:20.027843 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6992258b-cx54c"] Nov 01 16:33:20.303498 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:20.303390 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" containerID="cri-o://70aa55c1cb35798a69558f65ca80eff605edd2047accc60f77fccfec2c48da27" gracePeriod=45 Nov 01 16:33:20.414385 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:20.414343 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" event={"ID":"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697","Type":"ContainerStarted","Data":"c4c8270ffe66f1ce2618d639554942cdf1db53eeef7aee617e58e4ef7eeda99a"} Nov 01 16:33:20.564229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:20.563722 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": dial tcp 10.134.0.60:8000: connect: connection refused" Nov 01 16:33:20.766863 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:20.766836 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:33:20.834686 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:20.834604 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b034b9f6-e3e0-4682-9323-f99ce63ddaf1" path="/var/lib/kubelet/pods/b034b9f6-e3e0-4682-9323-f99ce63ddaf1/volumes" Nov 01 16:33:20.875665 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:20.875630 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7be529d-6534-4d74-a436-6d02335f9e5f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7be529d-6534-4d74-a436-6d02335f9e5f-dshm\") pod \"f7be529d-6534-4d74-a436-6d02335f9e5f\" (UID: \"f7be529d-6534-4d74-a436-6d02335f9e5f\") " Nov 01 16:33:20.875856 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:20.875733 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7be529d-6534-4d74-a436-6d02335f9e5f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7be529d-6534-4d74-a436-6d02335f9e5f-model-cache\") pod \"f7be529d-6534-4d74-a436-6d02335f9e5f\" (UID: \"f7be529d-6534-4d74-a436-6d02335f9e5f\") " Nov 01 16:33:20.875856 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:20.875783 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7be529d-6534-4d74-a436-6d02335f9e5f-tls-certs\" (UniqueName: \"kubernetes.io/secret/f7be529d-6534-4d74-a436-6d02335f9e5f-tls-certs\") pod \"f7be529d-6534-4d74-a436-6d02335f9e5f\" (UID: \"f7be529d-6534-4d74-a436-6d02335f9e5f\") " Nov 01 16:33:20.875856 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:20.875826 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7be529d-6534-4d74-a436-6d02335f9e5f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7be529d-6534-4d74-a436-6d02335f9e5f-tmp-dir\") pod \"f7be529d-6534-4d74-a436-6d02335f9e5f\" (UID: \"f7be529d-6534-4d74-a436-6d02335f9e5f\") " Nov 01 16:33:20.876070 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:20.875872 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7be529d-6534-4d74-a436-6d02335f9e5f-home\" (UniqueName: \"kubernetes.io/empty-dir/f7be529d-6534-4d74-a436-6d02335f9e5f-home\") pod \"f7be529d-6534-4d74-a436-6d02335f9e5f\" (UID: \"f7be529d-6534-4d74-a436-6d02335f9e5f\") " Nov 01 16:33:20.876070 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:20.875939 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7be529d-6534-4d74-a436-6d02335f9e5f-kube-api-access-jkdd6\" (UniqueName: \"kubernetes.io/projected/f7be529d-6534-4d74-a436-6d02335f9e5f-kube-api-access-jkdd6\") pod \"f7be529d-6534-4d74-a436-6d02335f9e5f\" (UID: \"f7be529d-6534-4d74-a436-6d02335f9e5f\") " Nov 01 16:33:20.892863 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:20.877064 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7be529d-6534-4d74-a436-6d02335f9e5f-tmp-dir" pod "f7be529d-6534-4d74-a436-6d02335f9e5f" (UID: "f7be529d-6534-4d74-a436-6d02335f9e5f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:20.892863 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:20.882141 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7be529d-6534-4d74-a436-6d02335f9e5f-kube-api-access-jkdd6" pod "f7be529d-6534-4d74-a436-6d02335f9e5f" (UID: "f7be529d-6534-4d74-a436-6d02335f9e5f"). InnerVolumeSpecName "kube-api-access-jkdd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:20.893167 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:20.893102 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7be529d-6534-4d74-a436-6d02335f9e5f-home" pod "f7be529d-6534-4d74-a436-6d02335f9e5f" (UID: "f7be529d-6534-4d74-a436-6d02335f9e5f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:20.893817 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:20.893780 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7be529d-6534-4d74-a436-6d02335f9e5f-dshm" pod "f7be529d-6534-4d74-a436-6d02335f9e5f" (UID: "f7be529d-6534-4d74-a436-6d02335f9e5f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:20.912438 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:20.897367 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7be529d-6534-4d74-a436-6d02335f9e5f-model-cache" pod "f7be529d-6534-4d74-a436-6d02335f9e5f" (UID: "f7be529d-6534-4d74-a436-6d02335f9e5f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:20.932753 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:20.932681 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7be529d-6534-4d74-a436-6d02335f9e5f-tls-certs" pod "f7be529d-6534-4d74-a436-6d02335f9e5f" (UID: "f7be529d-6534-4d74-a436-6d02335f9e5f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:33:20.978065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:20.978004 4363 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7be529d-6534-4d74-a436-6d02335f9e5f-model-cache\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:20.978065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:20.978058 4363 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7be529d-6534-4d74-a436-6d02335f9e5f-tls-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:20.978065 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:20.978072 4363 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7be529d-6534-4d74-a436-6d02335f9e5f-tmp-dir\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:20.978393 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:20.978086 4363 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7be529d-6534-4d74-a436-6d02335f9e5f-home\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:20.978393 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:20.978100 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkdd6\" (UniqueName: \"kubernetes.io/projected/f7be529d-6534-4d74-a436-6d02335f9e5f-kube-api-access-jkdd6\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:20.978393 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:20.978114 4363 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7be529d-6534-4d74-a436-6d02335f9e5f-dshm\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:21.267829 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:21.267796 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" Nov 01 16:33:21.268774 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:21.268749 4363 patch_prober.go:29] interesting pod/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body= Nov 01 16:33:21.268895 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:21.268816 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" podUID="b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" Nov 01 16:33:21.423517 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:21.423476 4363 generic.go:356] "Generic (PLEG): container finished" podID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerID="70aa55c1cb35798a69558f65ca80eff605edd2047accc60f77fccfec2c48da27" exitCode=0 Nov 01 16:33:21.423711 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:21.423525 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" Nov 01 16:33:21.423711 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:21.423530 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" event={"ID":"f7be529d-6534-4d74-a436-6d02335f9e5f","Type":"ContainerDied","Data":"70aa55c1cb35798a69558f65ca80eff605edd2047accc60f77fccfec2c48da27"} Nov 01 16:33:21.423711 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:21.423585 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm" event={"ID":"f7be529d-6534-4d74-a436-6d02335f9e5f","Type":"ContainerDied","Data":"557105b00d4c771cbf6dcfadc8369b30dff23d201968cbe4a0a69211fe51bcb6"} Nov 01 16:33:21.423711 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:21.423612 4363 scope.go:122] "RemoveContainer" containerID="70aa55c1cb35798a69558f65ca80eff605edd2047accc60f77fccfec2c48da27" Nov 01 16:33:21.434797 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:21.434780 4363 scope.go:122] "RemoveContainer" containerID="70aa55c1cb35798a69558f65ca80eff605edd2047accc60f77fccfec2c48da27" Nov 01 16:33:21.435078 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:33:21.435058 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70aa55c1cb35798a69558f65ca80eff605edd2047accc60f77fccfec2c48da27\": container with ID starting with 70aa55c1cb35798a69558f65ca80eff605edd2047accc60f77fccfec2c48da27 not found: ID does not exist" containerID="70aa55c1cb35798a69558f65ca80eff605edd2047accc60f77fccfec2c48da27" Nov 01 16:33:21.435159 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:21.435086 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70aa55c1cb35798a69558f65ca80eff605edd2047accc60f77fccfec2c48da27"} err="failed to get container status \"70aa55c1cb35798a69558f65ca80eff605edd2047accc60f77fccfec2c48da27\": rpc error: code = NotFound desc = could not find container \"70aa55c1cb35798a69558f65ca80eff605edd2047accc60f77fccfec2c48da27\": container with ID starting with 70aa55c1cb35798a69558f65ca80eff605edd2047accc60f77fccfec2c48da27 not found: ID does not exist" Nov 01 16:33:21.448003 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:21.447775 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm"] Nov 01 16:33:21.451480 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:21.451458 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-bd831a/distinct-w3-bd831a-kserve-546b5dfbc9-4d7tm"] Nov 01 16:33:22.267955 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:22.267918 4363 patch_prober.go:29] interesting pod/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body= Nov 01 16:33:22.268464 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:22.267972 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" podUID="b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" Nov 01 16:33:22.327765 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:22.327724 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" Nov 01 16:33:22.432808 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:22.432768 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg"] Nov 01 16:33:22.433158 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:22.433125 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" podUID="a6505861-2940-41bb-8dbd-d71b797cd2c6" containerName="istio-proxy" containerID="cri-o://9582032c5133eac85e5b978a2473e0a66e33af27b2bd142e961cbea19ae34faa" gracePeriod=30 Nov 01 16:33:22.447526 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:22.441737 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq"] Nov 01 16:33:22.447526 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:22.441801 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf"] Nov 01 16:33:22.447526 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:22.441978 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" podUID="d7babd48-8700-4398-b87b-c282cbc99c4e" containerName="istio-proxy" containerID="cri-o://3d297cf78921df8bc352150246b9065dfbdb424a0fa844ea5e1a94143043b5de" gracePeriod=30 Nov 01 16:33:22.447526 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:22.443657 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" podUID="9c37d94d-deb1-4a00-9d58-d91bb4b2dd95" containerName="istio-proxy" containerID="cri-o://e53343934f58a222255dca8b3bf908e425bb95ccac95033c90c154ea64b54c59" gracePeriod=30 Nov 01 16:33:22.451042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:22.449987 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7"] Nov 01 16:33:22.451042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:22.450213 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" podUID="f32c7fe0-caa2-4a6d-80db-aec48cd9ddda" containerName="istio-proxy" containerID="cri-o://eb89bd1fca927c7b5dd5be025577f78abb01193fdf94913158b4335fc43bea95" gracePeriod=30 Nov 01 16:33:22.815438 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:22.815404 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" path="/var/lib/kubelet/pods/f7be529d-6534-4d74-a436-6d02335f9e5f/volumes" Nov 01 16:33:23.268132 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:23.268096 4363 patch_prober.go:29] interesting pod/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body= Nov 01 16:33:23.268541 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:23.268150 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" podUID="b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" Nov 01 16:33:23.549880 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:23.549795 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bd831a-56899656f5-jhvtb" podUID="bc5eb74e-c5d9-4210-a626-94e8279c3a36" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:33:24.240890 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.240862 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" Nov 01 16:33:24.267996 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.267968 4363 patch_prober.go:29] interesting pod/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body= Nov 01 16:33:24.268170 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.268036 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" podUID="b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" Nov 01 16:33:24.309631 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.309588 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f69b8bb-4aa6-4713-8b2f-138c5d358858-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f69b8bb-4aa6-4713-8b2f-138c5d358858-istio-podinfo\") pod \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\" (UID: \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\") " Nov 01 16:33:24.309834 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.309657 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f69b8bb-4aa6-4713-8b2f-138c5d358858-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f69b8bb-4aa6-4713-8b2f-138c5d358858-istio-envoy\") pod \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\" (UID: \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\") " Nov 01 16:33:24.309834 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.309738 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f69b8bb-4aa6-4713-8b2f-138c5d358858-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f69b8bb-4aa6-4713-8b2f-138c5d358858-credential-socket\") pod \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\" (UID: \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\") " Nov 01 16:33:24.309834 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.309784 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f69b8bb-4aa6-4713-8b2f-138c5d358858-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f69b8bb-4aa6-4713-8b2f-138c5d358858-istio-data\") pod \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\" (UID: \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\") " Nov 01 16:33:24.310029 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.309855 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f69b8bb-4aa6-4713-8b2f-138c5d358858-kube-api-access-t26mt\" (UniqueName: \"kubernetes.io/projected/5f69b8bb-4aa6-4713-8b2f-138c5d358858-kube-api-access-t26mt\") pod \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\" (UID: \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\") " Nov 01 16:33:24.310029 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.309890 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f69b8bb-4aa6-4713-8b2f-138c5d358858-istio-token\" (UniqueName: \"kubernetes.io/projected/5f69b8bb-4aa6-4713-8b2f-138c5d358858-istio-token\") pod \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\" (UID: \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\") " Nov 01 16:33:24.310029 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.309933 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f69b8bb-4aa6-4713-8b2f-138c5d358858-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f69b8bb-4aa6-4713-8b2f-138c5d358858-istiod-ca-cert\") pod \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\" (UID: \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\") " Nov 01 16:33:24.310029 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.309971 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f69b8bb-4aa6-4713-8b2f-138c5d358858-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f69b8bb-4aa6-4713-8b2f-138c5d358858-workload-certs\") pod \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\" (UID: \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\") " Nov 01 16:33:24.310349 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.310323 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f69b8bb-4aa6-4713-8b2f-138c5d358858-workload-certs" pod "5f69b8bb-4aa6-4713-8b2f-138c5d358858" (UID: "5f69b8bb-4aa6-4713-8b2f-138c5d358858"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:24.310655 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.310620 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f69b8bb-4aa6-4713-8b2f-138c5d358858-istiod-ca-cert" pod "5f69b8bb-4aa6-4713-8b2f-138c5d358858" (UID: "5f69b8bb-4aa6-4713-8b2f-138c5d358858"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:24.310851 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.310831 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f69b8bb-4aa6-4713-8b2f-138c5d358858-credential-socket" pod "5f69b8bb-4aa6-4713-8b2f-138c5d358858" (UID: "5f69b8bb-4aa6-4713-8b2f-138c5d358858"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:24.311067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.310981 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f69b8bb-4aa6-4713-8b2f-138c5d358858-istio-data" pod "5f69b8bb-4aa6-4713-8b2f-138c5d358858" (UID: "5f69b8bb-4aa6-4713-8b2f-138c5d358858"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:24.311165 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.311119 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f69b8bb-4aa6-4713-8b2f-138c5d358858-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f69b8bb-4aa6-4713-8b2f-138c5d358858-workload-socket\") pod \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\" (UID: \"5f69b8bb-4aa6-4713-8b2f-138c5d358858\") " Nov 01 16:33:24.311592 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.311571 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f69b8bb-4aa6-4713-8b2f-138c5d358858-workload-socket" pod "5f69b8bb-4aa6-4713-8b2f-138c5d358858" (UID: "5f69b8bb-4aa6-4713-8b2f-138c5d358858"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:24.311884 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.311840 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f69b8bb-4aa6-4713-8b2f-138c5d358858-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:24.311884 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.311877 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f69b8bb-4aa6-4713-8b2f-138c5d358858-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:24.312093 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.311895 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f69b8bb-4aa6-4713-8b2f-138c5d358858-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:24.312093 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.311911 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f69b8bb-4aa6-4713-8b2f-138c5d358858-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:24.312093 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.311928 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f69b8bb-4aa6-4713-8b2f-138c5d358858-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:24.314774 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.314745 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f69b8bb-4aa6-4713-8b2f-138c5d358858-istio-envoy" pod "5f69b8bb-4aa6-4713-8b2f-138c5d358858" (UID: "5f69b8bb-4aa6-4713-8b2f-138c5d358858"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:24.317049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.316879 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f69b8bb-4aa6-4713-8b2f-138c5d358858-istio-token" pod "5f69b8bb-4aa6-4713-8b2f-138c5d358858" (UID: "5f69b8bb-4aa6-4713-8b2f-138c5d358858"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:24.317255 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.317233 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f69b8bb-4aa6-4713-8b2f-138c5d358858-istio-podinfo" pod "5f69b8bb-4aa6-4713-8b2f-138c5d358858" (UID: "5f69b8bb-4aa6-4713-8b2f-138c5d358858"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:33:24.317793 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.317775 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f69b8bb-4aa6-4713-8b2f-138c5d358858-kube-api-access-t26mt" pod "5f69b8bb-4aa6-4713-8b2f-138c5d358858" (UID: "5f69b8bb-4aa6-4713-8b2f-138c5d358858"). InnerVolumeSpecName "kube-api-access-t26mt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:24.346047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.344316 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-501d63-5fd9cc8bfb-krlt9" Nov 01 16:33:24.346047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.344868 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-678bbff655-z866s" Nov 01 16:33:24.421959 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.421924 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f69b8bb-4aa6-4713-8b2f-138c5d358858-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:24.421959 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.421962 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f69b8bb-4aa6-4713-8b2f-138c5d358858-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:24.422235 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.421980 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t26mt\" (UniqueName: \"kubernetes.io/projected/5f69b8bb-4aa6-4713-8b2f-138c5d358858-kube-api-access-t26mt\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:24.422235 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.421996 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f69b8bb-4aa6-4713-8b2f-138c5d358858-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:24.613909 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.613868 4363 generic.go:356] "Generic (PLEG): container finished" podID="5f69b8bb-4aa6-4713-8b2f-138c5d358858" containerID="5c0cf1794a919c29141a45a01845845fd0431e56c59229210667d72142b0010f" exitCode=0 Nov 01 16:33:24.613909 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.613923 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" event={"ID":"5f69b8bb-4aa6-4713-8b2f-138c5d358858","Type":"ContainerDied","Data":"5c0cf1794a919c29141a45a01845845fd0431e56c59229210667d72142b0010f"} Nov 01 16:33:24.614271 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.613929 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" Nov 01 16:33:24.614271 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.613977 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4" event={"ID":"5f69b8bb-4aa6-4713-8b2f-138c5d358858","Type":"ContainerDied","Data":"879fa1d7d9ec9c2481483ddb5d6901e822d827b4b9256a615b30276a1cfd9845"} Nov 01 16:33:24.614271 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.614007 4363 scope.go:122] "RemoveContainer" containerID="5c0cf1794a919c29141a45a01845845fd0431e56c59229210667d72142b0010f" Nov 01 16:33:24.633353 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.633323 4363 scope.go:122] "RemoveContainer" containerID="68171b1bd2001cb00c9191f8272dfe0efca9dd69364dfbdaad7ff19e1bd3b8a4" Nov 01 16:33:24.644626 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.643720 4363 scope.go:122] "RemoveContainer" containerID="5c0cf1794a919c29141a45a01845845fd0431e56c59229210667d72142b0010f" Nov 01 16:33:24.644766 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:33:24.644740 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c0cf1794a919c29141a45a01845845fd0431e56c59229210667d72142b0010f\": container with ID starting with 5c0cf1794a919c29141a45a01845845fd0431e56c59229210667d72142b0010f not found: ID does not exist" containerID="5c0cf1794a919c29141a45a01845845fd0431e56c59229210667d72142b0010f" Nov 01 16:33:24.644825 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.644777 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c0cf1794a919c29141a45a01845845fd0431e56c59229210667d72142b0010f"} err="failed to get container status \"5c0cf1794a919c29141a45a01845845fd0431e56c59229210667d72142b0010f\": rpc error: code = NotFound desc = could not find container \"5c0cf1794a919c29141a45a01845845fd0431e56c59229210667d72142b0010f\": container with ID starting with 5c0cf1794a919c29141a45a01845845fd0431e56c59229210667d72142b0010f not found: ID does not exist" Nov 01 16:33:24.644825 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.644801 4363 scope.go:122] "RemoveContainer" containerID="68171b1bd2001cb00c9191f8272dfe0efca9dd69364dfbdaad7ff19e1bd3b8a4" Nov 01 16:33:24.645117 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:33:24.645091 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68171b1bd2001cb00c9191f8272dfe0efca9dd69364dfbdaad7ff19e1bd3b8a4\": container with ID starting with 68171b1bd2001cb00c9191f8272dfe0efca9dd69364dfbdaad7ff19e1bd3b8a4 not found: ID does not exist" containerID="68171b1bd2001cb00c9191f8272dfe0efca9dd69364dfbdaad7ff19e1bd3b8a4" Nov 01 16:33:24.645215 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.645125 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68171b1bd2001cb00c9191f8272dfe0efca9dd69364dfbdaad7ff19e1bd3b8a4"} err="failed to get container status \"68171b1bd2001cb00c9191f8272dfe0efca9dd69364dfbdaad7ff19e1bd3b8a4\": rpc error: code = NotFound desc = could not find container \"68171b1bd2001cb00c9191f8272dfe0efca9dd69364dfbdaad7ff19e1bd3b8a4\": container with ID starting with 68171b1bd2001cb00c9191f8272dfe0efca9dd69364dfbdaad7ff19e1bd3b8a4 not found: ID does not exist" Nov 01 16:33:24.648697 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.648678 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4"] Nov 01 16:33:24.651594 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.651573 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-5ccd7b-openshift-default-766649b8f-rg2z4"] Nov 01 16:33:24.812937 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:24.812904 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f69b8bb-4aa6-4713-8b2f-138c5d358858" path="/var/lib/kubelet/pods/5f69b8bb-4aa6-4713-8b2f-138c5d358858/volumes" Nov 01 16:33:25.268075 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:25.268043 4363 patch_prober.go:29] interesting pod/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body= Nov 01 16:33:25.268264 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:25.268095 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" podUID="b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" Nov 01 16:33:25.629841 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:25.629805 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d"] Nov 01 16:33:25.630583 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:25.630561 4363 cpu_manager.go:414] "containerMap: removing container" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" Nov 01 16:33:25.630750 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:25.630733 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" Nov 01 16:33:25.630858 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:25.630840 4363 cpu_manager.go:414] "containerMap: removing container" podUID="5f69b8bb-4aa6-4713-8b2f-138c5d358858" containerName="istio-proxy" Nov 01 16:33:25.630937 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:25.630858 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f69b8bb-4aa6-4713-8b2f-138c5d358858" containerName="istio-proxy" Nov 01 16:33:25.631142 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:25.631125 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7be529d-6534-4d74-a436-6d02335f9e5f" containerName="main" Nov 01 16:33:25.631271 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:25.631258 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f69b8bb-4aa6-4713-8b2f-138c5d358858" containerName="istio-proxy" Nov 01 16:33:25.631363 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:25.631353 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f69b8bb-4aa6-4713-8b2f-138c5d358858" containerName="istio-proxy" Nov 01 16:33:25.637177 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:25.637052 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" Nov 01 16:33:25.645875 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:25.644899 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-f4a265-openshift-default-dockercfg-psjvg\"" Nov 01 16:33:25.646566 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:25.646540 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-workload-socket\") pod \"e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d\" (UID: \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\") " pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" Nov 01 16:33:25.646695 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:25.646648 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-istiod-ca-cert\") pod \"e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d\" (UID: \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\") " pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" Nov 01 16:33:25.646773 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:25.646715 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-workload-certs\") pod \"e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d\" (UID: \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\") " pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" Nov 01 16:33:25.646773 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:25.646740 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-istio-envoy\") pod \"e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d\" (UID: \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\") " pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" Nov 01 16:33:25.646855 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:25.646817 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-credential-socket\") pod \"e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d\" (UID: \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\") " pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" Nov 01 16:33:25.646890 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:25.646855 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-istio-token\") pod \"e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d\" (UID: \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\") " pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" Nov 01 16:33:25.646958 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:25.646938 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-istio-data\") pod \"e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d\" (UID: \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\") " pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" Nov 01 16:33:25.647188 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:25.647167 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qqsc\" (UniqueName: \"kubernetes.io/projected/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-kube-api-access-8qqsc\") pod \"e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d\" (UID: \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\") " pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" Nov 01 16:33:25.647282 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:25.647205 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-istio-podinfo\") pod \"e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d\" (UID: \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\") " pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" Nov 01 16:33:25.648226 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:25.648202 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d"] Nov 01 16:33:25.754170 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:25.754131 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-istiod-ca-cert\") pod \"e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d\" (UID: \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\") " pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" Nov 01 16:33:25.754447 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:25.754429 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-workload-certs\") pod \"e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d\" (UID: \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\") " pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" Nov 01 16:33:25.754580 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:25.754566 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-istio-envoy\") pod \"e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d\" (UID: \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\") " pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" Nov 01 16:33:25.754746 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:25.754729 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-credential-socket\") pod \"e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d\" (UID: \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\") " pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" Nov 01 16:33:25.754877 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:25.754863 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-istio-token\") pod \"e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d\" (UID: \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\") " pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" Nov 01 16:33:25.755077 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:25.755058 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-istio-data\") pod \"e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d\" (UID: \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\") " pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" Nov 01 16:33:25.755282 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:25.755263 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qqsc\" (UniqueName: \"kubernetes.io/projected/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-kube-api-access-8qqsc\") pod \"e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d\" (UID: \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\") " pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" Nov 01 16:33:25.755414 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:25.755394 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-istio-podinfo\") pod \"e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d\" (UID: \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\") " pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" Nov 01 16:33:25.755555 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:25.755539 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-workload-socket\") pod \"e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d\" (UID: \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\") " pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" Nov 01 16:33:25.776042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:25.756352 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-workload-socket\") pod \"e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d\" (UID: \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\") " pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" Nov 01 16:33:25.776551 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:25.766110 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-credential-socket\") pod \"e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d\" (UID: \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\") " pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" Nov 01 16:33:25.776711 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:25.766808 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-istiod-ca-cert\") pod \"e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d\" (UID: \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\") " pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" Nov 01 16:33:25.776920 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:25.767056 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-workload-certs\") pod \"e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d\" (UID: \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\") " pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" Nov 01 16:33:25.777669 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:25.777645 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-istio-data\") pod \"e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d\" (UID: \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\") " pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" Nov 01 16:33:25.811040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:25.809313 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-istio-podinfo\") pod \"e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d\" (UID: \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\") " pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" Nov 01 16:33:25.819053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:25.816354 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-istio-envoy\") pod \"e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d\" (UID: \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\") " pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" Nov 01 16:33:25.822088 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:25.819930 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qqsc\" (UniqueName: \"kubernetes.io/projected/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-kube-api-access-8qqsc\") pod \"e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d\" (UID: \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\") " pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" Nov 01 16:33:25.822088 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:25.821085 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-istio-token\") pod \"e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d\" (UID: \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\") " pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" Nov 01 16:33:25.971064 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:25.970960 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" Nov 01 16:33:26.261294 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:26.261265 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d"] Nov 01 16:33:26.262068 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:33:26.262041 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87ddacbe_42b0_42b0_8b1c_a3e6c4ed4519.slice/crio-b9e93e91ad26ae21d93336d2139099f75915a19109e1bb431da6687e7efaef51 WatchSource:0}: Error finding container b9e93e91ad26ae21d93336d2139099f75915a19109e1bb431da6687e7efaef51: Status 404 returned error can't find the container with id b9e93e91ad26ae21d93336d2139099f75915a19109e1bb431da6687e7efaef51 Nov 01 16:33:26.264481 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:26.264446 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:33:26.264599 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:26.264524 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:33:26.264599 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:26.264569 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:33:26.272567 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:26.272543 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" Nov 01 16:33:26.273927 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:26.273907 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" Nov 01 16:33:26.636435 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:26.636401 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" event={"ID":"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519","Type":"ContainerStarted","Data":"65f99fc6091a1226f46e39229bd199f9afa4fd6033a850eedb0a5ddbaed92fea"} Nov 01 16:33:26.636625 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:26.636468 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" event={"ID":"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519","Type":"ContainerStarted","Data":"b9e93e91ad26ae21d93336d2139099f75915a19109e1bb431da6687e7efaef51"} Nov 01 16:33:26.658969 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:26.658915 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" podStartSLOduration=1.658901264 podStartE2EDuration="1.658901264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:33:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:33:26.653386817 +0000 UTC m=+2739.985124466" watchObservedRunningTime="2026-11-01 16:33:26.658901264 +0000 UTC m=+2739.990638914" Nov 01 16:33:26.971764 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:26.971679 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" Nov 01 16:33:27.566005 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:33:27.565962 4363 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7babd48_8700_4398_b87b_c282cbc99c4e.slice/crio-conmon-3d297cf78921df8bc352150246b9065dfbdb424a0fa844ea5e1a94143043b5de.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6505861_2940_41bb_8dbd_d71b797cd2c6.slice/crio-9582032c5133eac85e5b978a2473e0a66e33af27b2bd142e961cbea19ae34faa.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7babd48_8700_4398_b87b_c282cbc99c4e.slice/crio-3d297cf78921df8bc352150246b9065dfbdb424a0fa844ea5e1a94143043b5de.scope\": RecentStats: unable to find data in memory cache]" Nov 01 16:33:27.651507 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.650565 4363 generic.go:356] "Generic (PLEG): container finished" podID="9c37d94d-deb1-4a00-9d58-d91bb4b2dd95" containerID="e53343934f58a222255dca8b3bf908e425bb95ccac95033c90c154ea64b54c59" exitCode=0 Nov 01 16:33:27.651507 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.650775 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" event={"ID":"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95","Type":"ContainerDied","Data":"e53343934f58a222255dca8b3bf908e425bb95ccac95033c90c154ea64b54c59"} Nov 01 16:33:27.652853 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.652667 4363 generic.go:356] "Generic (PLEG): container finished" podID="d7babd48-8700-4398-b87b-c282cbc99c4e" containerID="3d297cf78921df8bc352150246b9065dfbdb424a0fa844ea5e1a94143043b5de" exitCode=0 Nov 01 16:33:27.652853 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.652742 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" event={"ID":"d7babd48-8700-4398-b87b-c282cbc99c4e","Type":"ContainerDied","Data":"3d297cf78921df8bc352150246b9065dfbdb424a0fa844ea5e1a94143043b5de"} Nov 01 16:33:27.654716 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.654696 4363 generic.go:356] "Generic (PLEG): container finished" podID="a6505861-2940-41bb-8dbd-d71b797cd2c6" containerID="9582032c5133eac85e5b978a2473e0a66e33af27b2bd142e961cbea19ae34faa" exitCode=0 Nov 01 16:33:27.654837 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.654783 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" event={"ID":"a6505861-2940-41bb-8dbd-d71b797cd2c6","Type":"ContainerDied","Data":"9582032c5133eac85e5b978a2473e0a66e33af27b2bd142e961cbea19ae34faa"} Nov 01 16:33:27.658135 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.658109 4363 generic.go:356] "Generic (PLEG): container finished" podID="f32c7fe0-caa2-4a6d-80db-aec48cd9ddda" containerID="eb89bd1fca927c7b5dd5be025577f78abb01193fdf94913158b4335fc43bea95" exitCode=0 Nov 01 16:33:27.658243 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.658180 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" event={"ID":"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda","Type":"ContainerDied","Data":"eb89bd1fca927c7b5dd5be025577f78abb01193fdf94913158b4335fc43bea95"} Nov 01 16:33:27.703771 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.703747 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" Nov 01 16:33:27.789255 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.787863 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9a482b-86b559b58f-frnp7" podUID="e533e983-dbc0-4ce2-8d58-c74c0c0d857b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:33:27.794422 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.794390 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-istiod-ca-cert\") pod \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\" (UID: \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\") " Nov 01 16:33:27.794665 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.794648 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-workload-certs\") pod \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\" (UID: \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\") " Nov 01 16:33:27.794793 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.794778 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-workload-socket\") pod \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\" (UID: \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\") " Nov 01 16:33:27.795001 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.794988 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-istio-data\") pod \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\" (UID: \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\") " Nov 01 16:33:27.795139 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.795125 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-istio-envoy\") pod \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\" (UID: \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\") " Nov 01 16:33:27.795285 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.795272 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-credential-socket\") pod \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\" (UID: \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\") " Nov 01 16:33:27.795442 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.795429 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-istio-token\" (UniqueName: \"kubernetes.io/projected/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-istio-token\") pod \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\" (UID: \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\") " Nov 01 16:33:27.795606 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.795592 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-istio-podinfo\") pod \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\" (UID: \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\") " Nov 01 16:33:27.795716 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.795703 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-kube-api-access-xkbjm\" (UniqueName: \"kubernetes.io/projected/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-kube-api-access-xkbjm\") pod \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\" (UID: \"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95\") " Nov 01 16:33:27.799219 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.798922 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-kube-api-access-xkbjm" pod "9c37d94d-deb1-4a00-9d58-d91bb4b2dd95" (UID: "9c37d94d-deb1-4a00-9d58-d91bb4b2dd95"). InnerVolumeSpecName "kube-api-access-xkbjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:27.799672 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.799462 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-istiod-ca-cert" pod "9c37d94d-deb1-4a00-9d58-d91bb4b2dd95" (UID: "9c37d94d-deb1-4a00-9d58-d91bb4b2dd95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:27.799941 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.799792 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-workload-certs" pod "9c37d94d-deb1-4a00-9d58-d91bb4b2dd95" (UID: "9c37d94d-deb1-4a00-9d58-d91bb4b2dd95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:27.800286 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.800124 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-workload-socket" pod "9c37d94d-deb1-4a00-9d58-d91bb4b2dd95" (UID: "9c37d94d-deb1-4a00-9d58-d91bb4b2dd95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:27.800637 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.800397 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-istio-data" pod "9c37d94d-deb1-4a00-9d58-d91bb4b2dd95" (UID: "9c37d94d-deb1-4a00-9d58-d91bb4b2dd95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:27.802929 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.802763 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-istio-envoy" pod "9c37d94d-deb1-4a00-9d58-d91bb4b2dd95" (UID: "9c37d94d-deb1-4a00-9d58-d91bb4b2dd95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:27.803344 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.803102 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-credential-socket" pod "9c37d94d-deb1-4a00-9d58-d91bb4b2dd95" (UID: "9c37d94d-deb1-4a00-9d58-d91bb4b2dd95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:27.805611 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.805359 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-istio-token" pod "9c37d94d-deb1-4a00-9d58-d91bb4b2dd95" (UID: "9c37d94d-deb1-4a00-9d58-d91bb4b2dd95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:27.807623 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.807599 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-istio-podinfo" pod "9c37d94d-deb1-4a00-9d58-d91bb4b2dd95" (UID: "9c37d94d-deb1-4a00-9d58-d91bb4b2dd95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:33:27.836547 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.836520 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" Nov 01 16:33:27.897549 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.897514 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6505861-2940-41bb-8dbd-d71b797cd2c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6505861-2940-41bb-8dbd-d71b797cd2c6-workload-socket\") pod \"a6505861-2940-41bb-8dbd-d71b797cd2c6\" (UID: \"a6505861-2940-41bb-8dbd-d71b797cd2c6\") " Nov 01 16:33:27.897821 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.897800 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6505861-2940-41bb-8dbd-d71b797cd2c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6505861-2940-41bb-8dbd-d71b797cd2c6-istio-envoy\") pod \"a6505861-2940-41bb-8dbd-d71b797cd2c6\" (UID: \"a6505861-2940-41bb-8dbd-d71b797cd2c6\") " Nov 01 16:33:27.898000 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.897982 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6505861-2940-41bb-8dbd-d71b797cd2c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6505861-2940-41bb-8dbd-d71b797cd2c6-credential-socket\") pod \"a6505861-2940-41bb-8dbd-d71b797cd2c6\" (UID: \"a6505861-2940-41bb-8dbd-d71b797cd2c6\") " Nov 01 16:33:27.898180 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.898160 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6505861-2940-41bb-8dbd-d71b797cd2c6-istio-token\" (UniqueName: \"kubernetes.io/projected/a6505861-2940-41bb-8dbd-d71b797cd2c6-istio-token\") pod \"a6505861-2940-41bb-8dbd-d71b797cd2c6\" (UID: \"a6505861-2940-41bb-8dbd-d71b797cd2c6\") " Nov 01 16:33:27.898322 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.898306 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6505861-2940-41bb-8dbd-d71b797cd2c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6505861-2940-41bb-8dbd-d71b797cd2c6-workload-certs\") pod \"a6505861-2940-41bb-8dbd-d71b797cd2c6\" (UID: \"a6505861-2940-41bb-8dbd-d71b797cd2c6\") " Nov 01 16:33:27.898555 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.898537 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6505861-2940-41bb-8dbd-d71b797cd2c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6505861-2940-41bb-8dbd-d71b797cd2c6-istiod-ca-cert\") pod \"a6505861-2940-41bb-8dbd-d71b797cd2c6\" (UID: \"a6505861-2940-41bb-8dbd-d71b797cd2c6\") " Nov 01 16:33:27.898695 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.898680 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6505861-2940-41bb-8dbd-d71b797cd2c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6505861-2940-41bb-8dbd-d71b797cd2c6-istio-data\") pod \"a6505861-2940-41bb-8dbd-d71b797cd2c6\" (UID: \"a6505861-2940-41bb-8dbd-d71b797cd2c6\") " Nov 01 16:33:27.900306 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.898901 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6505861-2940-41bb-8dbd-d71b797cd2c6-kube-api-access-w69hx\" (UniqueName: \"kubernetes.io/projected/a6505861-2940-41bb-8dbd-d71b797cd2c6-kube-api-access-w69hx\") pod \"a6505861-2940-41bb-8dbd-d71b797cd2c6\" (UID: \"a6505861-2940-41bb-8dbd-d71b797cd2c6\") " Nov 01 16:33:27.900573 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.900550 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6505861-2940-41bb-8dbd-d71b797cd2c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6505861-2940-41bb-8dbd-d71b797cd2c6-istio-podinfo\") pod \"a6505861-2940-41bb-8dbd-d71b797cd2c6\" (UID: \"a6505861-2940-41bb-8dbd-d71b797cd2c6\") " Nov 01 16:33:27.901379 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.901354 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:27.901946 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.901920 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkbjm\" (UniqueName: \"kubernetes.io/projected/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-kube-api-access-xkbjm\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:27.902094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.901950 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:27.902094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.901968 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:27.902094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.901982 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:27.902094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.901997 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:27.902094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.902029 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:27.902094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.902058 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:27.902094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.902073 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:27.902094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.901928 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6505861-2940-41bb-8dbd-d71b797cd2c6-istiod-ca-cert" pod "a6505861-2940-41bb-8dbd-d71b797cd2c6" (UID: "a6505861-2940-41bb-8dbd-d71b797cd2c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:27.902567 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.902176 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6505861-2940-41bb-8dbd-d71b797cd2c6-workload-certs" pod "a6505861-2940-41bb-8dbd-d71b797cd2c6" (UID: "a6505861-2940-41bb-8dbd-d71b797cd2c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:27.902567 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.902535 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6505861-2940-41bb-8dbd-d71b797cd2c6-istio-data" pod "a6505861-2940-41bb-8dbd-d71b797cd2c6" (UID: "a6505861-2940-41bb-8dbd-d71b797cd2c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:27.902955 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.902929 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6505861-2940-41bb-8dbd-d71b797cd2c6-credential-socket" pod "a6505861-2940-41bb-8dbd-d71b797cd2c6" (UID: "a6505861-2940-41bb-8dbd-d71b797cd2c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:27.911132 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.911104 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6505861-2940-41bb-8dbd-d71b797cd2c6-workload-socket" pod "a6505861-2940-41bb-8dbd-d71b797cd2c6" (UID: "a6505861-2940-41bb-8dbd-d71b797cd2c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:27.912229 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.912202 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6505861-2940-41bb-8dbd-d71b797cd2c6-istio-podinfo" pod "a6505861-2940-41bb-8dbd-d71b797cd2c6" (UID: "a6505861-2940-41bb-8dbd-d71b797cd2c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:33:27.912649 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.912549 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6505861-2940-41bb-8dbd-d71b797cd2c6-istio-envoy" pod "a6505861-2940-41bb-8dbd-d71b797cd2c6" (UID: "a6505861-2940-41bb-8dbd-d71b797cd2c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:27.912649 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.912610 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6505861-2940-41bb-8dbd-d71b797cd2c6-kube-api-access-w69hx" pod "a6505861-2940-41bb-8dbd-d71b797cd2c6" (UID: "a6505861-2940-41bb-8dbd-d71b797cd2c6"). InnerVolumeSpecName "kube-api-access-w69hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:27.913839 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.913811 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6505861-2940-41bb-8dbd-d71b797cd2c6-istio-token" pod "a6505861-2940-41bb-8dbd-d71b797cd2c6" (UID: "a6505861-2940-41bb-8dbd-d71b797cd2c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:27.976656 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:27.976621 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" Nov 01 16:33:28.004463 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.004432 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6505861-2940-41bb-8dbd-d71b797cd2c6-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.004463 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.004460 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6505861-2940-41bb-8dbd-d71b797cd2c6-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.004680 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.004472 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6505861-2940-41bb-8dbd-d71b797cd2c6-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.004680 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.004486 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6505861-2940-41bb-8dbd-d71b797cd2c6-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.004680 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.004497 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6505861-2940-41bb-8dbd-d71b797cd2c6-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.004680 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.004505 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6505861-2940-41bb-8dbd-d71b797cd2c6-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.004680 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.004514 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w69hx\" (UniqueName: \"kubernetes.io/projected/a6505861-2940-41bb-8dbd-d71b797cd2c6-kube-api-access-w69hx\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.004680 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.004524 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6505861-2940-41bb-8dbd-d71b797cd2c6-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.004680 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.004532 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6505861-2940-41bb-8dbd-d71b797cd2c6-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.135734 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.135702 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" Nov 01 16:33:28.210393 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.210358 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-workload-socket\") pod \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\" (UID: \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\") " Nov 01 16:33:28.210562 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.210426 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-credential-socket\") pod \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\" (UID: \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\") " Nov 01 16:33:28.210562 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.210488 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-istiod-ca-cert\") pod \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\" (UID: \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\") " Nov 01 16:33:28.210562 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.210528 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-kube-api-access-fgkqv\" (UniqueName: \"kubernetes.io/projected/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-kube-api-access-fgkqv\") pod \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\" (UID: \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\") " Nov 01 16:33:28.210805 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.210582 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-istio-token\" (UniqueName: \"kubernetes.io/projected/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-istio-token\") pod \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\" (UID: \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\") " Nov 01 16:33:28.210805 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.210622 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-istio-envoy\") pod \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\" (UID: \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\") " Nov 01 16:33:28.210805 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.210709 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-istio-data\") pod \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\" (UID: \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\") " Nov 01 16:33:28.210805 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.210740 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-istio-podinfo\") pod \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\" (UID: \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\") " Nov 01 16:33:28.210805 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.210797 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-workload-certs\") pod \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\" (UID: \"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda\") " Nov 01 16:33:28.211620 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.211571 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-workload-certs" pod "f32c7fe0-caa2-4a6d-80db-aec48cd9ddda" (UID: "f32c7fe0-caa2-4a6d-80db-aec48cd9ddda"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:28.212478 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.212200 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-istiod-ca-cert" pod "f32c7fe0-caa2-4a6d-80db-aec48cd9ddda" (UID: "f32c7fe0-caa2-4a6d-80db-aec48cd9ddda"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:28.212478 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.212393 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-workload-socket" pod "f32c7fe0-caa2-4a6d-80db-aec48cd9ddda" (UID: "f32c7fe0-caa2-4a6d-80db-aec48cd9ddda"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:28.212478 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.212448 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-istio-data" pod "f32c7fe0-caa2-4a6d-80db-aec48cd9ddda" (UID: "f32c7fe0-caa2-4a6d-80db-aec48cd9ddda"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:28.214614 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.212952 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-credential-socket" pod "f32c7fe0-caa2-4a6d-80db-aec48cd9ddda" (UID: "f32c7fe0-caa2-4a6d-80db-aec48cd9ddda"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:28.214614 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.213111 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" Nov 01 16:33:28.215200 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.215159 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-istio-envoy" pod "f32c7fe0-caa2-4a6d-80db-aec48cd9ddda" (UID: "f32c7fe0-caa2-4a6d-80db-aec48cd9ddda"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:28.215200 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.215179 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-kube-api-access-fgkqv" pod "f32c7fe0-caa2-4a6d-80db-aec48cd9ddda" (UID: "f32c7fe0-caa2-4a6d-80db-aec48cd9ddda"). InnerVolumeSpecName "kube-api-access-fgkqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:28.215786 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.215762 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-istio-token" pod "f32c7fe0-caa2-4a6d-80db-aec48cd9ddda" (UID: "f32c7fe0-caa2-4a6d-80db-aec48cd9ddda"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:28.219254 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.219227 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-istio-podinfo" pod "f32c7fe0-caa2-4a6d-80db-aec48cd9ddda" (UID: "f32c7fe0-caa2-4a6d-80db-aec48cd9ddda"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:33:28.312292 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.312196 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7babd48-8700-4398-b87b-c282cbc99c4e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7babd48-8700-4398-b87b-c282cbc99c4e-istio-podinfo\") pod \"d7babd48-8700-4398-b87b-c282cbc99c4e\" (UID: \"d7babd48-8700-4398-b87b-c282cbc99c4e\") " Nov 01 16:33:28.312292 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.312259 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7babd48-8700-4398-b87b-c282cbc99c4e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7babd48-8700-4398-b87b-c282cbc99c4e-workload-socket\") pod \"d7babd48-8700-4398-b87b-c282cbc99c4e\" (UID: \"d7babd48-8700-4398-b87b-c282cbc99c4e\") " Nov 01 16:33:28.312533 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.312313 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7babd48-8700-4398-b87b-c282cbc99c4e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7babd48-8700-4398-b87b-c282cbc99c4e-credential-socket\") pod \"d7babd48-8700-4398-b87b-c282cbc99c4e\" (UID: \"d7babd48-8700-4398-b87b-c282cbc99c4e\") " Nov 01 16:33:28.312533 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.312344 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7babd48-8700-4398-b87b-c282cbc99c4e-kube-api-access-zmrqk\" (UniqueName: \"kubernetes.io/projected/d7babd48-8700-4398-b87b-c282cbc99c4e-kube-api-access-zmrqk\") pod \"d7babd48-8700-4398-b87b-c282cbc99c4e\" (UID: \"d7babd48-8700-4398-b87b-c282cbc99c4e\") " Nov 01 16:33:28.312780 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.312608 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7babd48-8700-4398-b87b-c282cbc99c4e-credential-socket" pod "d7babd48-8700-4398-b87b-c282cbc99c4e" (UID: "d7babd48-8700-4398-b87b-c282cbc99c4e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:28.312780 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.312763 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7babd48-8700-4398-b87b-c282cbc99c4e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7babd48-8700-4398-b87b-c282cbc99c4e-istio-envoy\") pod \"d7babd48-8700-4398-b87b-c282cbc99c4e\" (UID: \"d7babd48-8700-4398-b87b-c282cbc99c4e\") " Nov 01 16:33:28.312983 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.312853 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7babd48-8700-4398-b87b-c282cbc99c4e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7babd48-8700-4398-b87b-c282cbc99c4e-workload-certs\") pod \"d7babd48-8700-4398-b87b-c282cbc99c4e\" (UID: \"d7babd48-8700-4398-b87b-c282cbc99c4e\") " Nov 01 16:33:28.312983 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.312618 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7babd48-8700-4398-b87b-c282cbc99c4e-workload-socket" pod "d7babd48-8700-4398-b87b-c282cbc99c4e" (UID: "d7babd48-8700-4398-b87b-c282cbc99c4e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:28.312983 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.312960 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7babd48-8700-4398-b87b-c282cbc99c4e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7babd48-8700-4398-b87b-c282cbc99c4e-istio-data\") pod \"d7babd48-8700-4398-b87b-c282cbc99c4e\" (UID: \"d7babd48-8700-4398-b87b-c282cbc99c4e\") " Nov 01 16:33:28.313171 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.313068 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7babd48-8700-4398-b87b-c282cbc99c4e-istio-token\" (UniqueName: \"kubernetes.io/projected/d7babd48-8700-4398-b87b-c282cbc99c4e-istio-token\") pod \"d7babd48-8700-4398-b87b-c282cbc99c4e\" (UID: \"d7babd48-8700-4398-b87b-c282cbc99c4e\") " Nov 01 16:33:28.313171 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.313104 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7babd48-8700-4398-b87b-c282cbc99c4e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7babd48-8700-4398-b87b-c282cbc99c4e-istiod-ca-cert\") pod \"d7babd48-8700-4398-b87b-c282cbc99c4e\" (UID: \"d7babd48-8700-4398-b87b-c282cbc99c4e\") " Nov 01 16:33:28.313277 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.313210 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7babd48-8700-4398-b87b-c282cbc99c4e-workload-certs" pod "d7babd48-8700-4398-b87b-c282cbc99c4e" (UID: "d7babd48-8700-4398-b87b-c282cbc99c4e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:28.313699 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.313381 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7babd48-8700-4398-b87b-c282cbc99c4e-istio-data" pod "d7babd48-8700-4398-b87b-c282cbc99c4e" (UID: "d7babd48-8700-4398-b87b-c282cbc99c4e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:28.313699 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.313669 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.313699 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.313693 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.313922 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.313709 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.313922 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.313722 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7babd48-8700-4398-b87b-c282cbc99c4e-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.313922 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.313734 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7babd48-8700-4398-b87b-c282cbc99c4e-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.313922 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.313748 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.313922 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.313762 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.313922 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.313775 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.313922 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.313789 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgkqv\" (UniqueName: \"kubernetes.io/projected/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-kube-api-access-fgkqv\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.313922 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.313804 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7babd48-8700-4398-b87b-c282cbc99c4e-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.313922 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.313732 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7babd48-8700-4398-b87b-c282cbc99c4e-istiod-ca-cert" pod "d7babd48-8700-4398-b87b-c282cbc99c4e" (UID: "d7babd48-8700-4398-b87b-c282cbc99c4e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:28.313922 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.313817 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.313922 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.313854 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.313922 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.313871 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7babd48-8700-4398-b87b-c282cbc99c4e-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.315035 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.314989 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7babd48-8700-4398-b87b-c282cbc99c4e-istio-podinfo" pod "d7babd48-8700-4398-b87b-c282cbc99c4e" (UID: "d7babd48-8700-4398-b87b-c282cbc99c4e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:33:28.315035 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.314999 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7babd48-8700-4398-b87b-c282cbc99c4e-kube-api-access-zmrqk" pod "d7babd48-8700-4398-b87b-c282cbc99c4e" (UID: "d7babd48-8700-4398-b87b-c282cbc99c4e"). InnerVolumeSpecName "kube-api-access-zmrqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:28.315305 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.315276 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7babd48-8700-4398-b87b-c282cbc99c4e-istio-envoy" pod "d7babd48-8700-4398-b87b-c282cbc99c4e" (UID: "d7babd48-8700-4398-b87b-c282cbc99c4e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:28.315662 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.315642 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7babd48-8700-4398-b87b-c282cbc99c4e-istio-token" pod "d7babd48-8700-4398-b87b-c282cbc99c4e" (UID: "d7babd48-8700-4398-b87b-c282cbc99c4e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:28.417903 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.416182 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmrqk\" (UniqueName: \"kubernetes.io/projected/d7babd48-8700-4398-b87b-c282cbc99c4e-kube-api-access-zmrqk\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.417903 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.416218 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7babd48-8700-4398-b87b-c282cbc99c4e-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.417903 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.416232 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7babd48-8700-4398-b87b-c282cbc99c4e-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.417903 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.416248 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7babd48-8700-4398-b87b-c282cbc99c4e-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.417903 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.416262 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7babd48-8700-4398-b87b-c282cbc99c4e-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.666289 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.666255 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" event={"ID":"d7babd48-8700-4398-b87b-c282cbc99c4e","Type":"ContainerDied","Data":"9393fb2e8ee6cb11848b8e86a550a319241dc47e90f12cc9def5469ec4cb3993"} Nov 01 16:33:28.666773 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.666263 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf" Nov 01 16:33:28.666773 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.666323 4363 scope.go:122] "RemoveContainer" containerID="3d297cf78921df8bc352150246b9065dfbdb424a0fa844ea5e1a94143043b5de" Nov 01 16:33:28.668069 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.668044 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" Nov 01 16:33:28.668069 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.668054 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg" event={"ID":"a6505861-2940-41bb-8dbd-d71b797cd2c6","Type":"ContainerDied","Data":"f13bd2fa60d457df0157d11cd5f7ad57d7ddd4da30cf62867dd4f7e1364a2822"} Nov 01 16:33:28.669967 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.669938 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" event={"ID":"f32c7fe0-caa2-4a6d-80db-aec48cd9ddda","Type":"ContainerDied","Data":"b2d5d9326a0e87086f7dc5a454879d2c3ea916ecec8c59f6651b4407469ceb8b"} Nov 01 16:33:28.670104 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.670070 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7" Nov 01 16:33:28.674653 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.672113 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" Nov 01 16:33:28.674653 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.672686 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq" event={"ID":"9c37d94d-deb1-4a00-9d58-d91bb4b2dd95","Type":"ContainerDied","Data":"287a7142d645ddba079c56c24bfaa9e134ac02b0c711a160083b7c7ad87102c8"} Nov 01 16:33:28.674653 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.672737 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" Nov 01 16:33:28.674653 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.673617 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" Nov 01 16:33:28.680551 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.680531 4363 scope.go:122] "RemoveContainer" containerID="9582032c5133eac85e5b978a2473e0a66e33af27b2bd142e961cbea19ae34faa" Nov 01 16:33:28.694290 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.694263 4363 scope.go:122] "RemoveContainer" containerID="eb89bd1fca927c7b5dd5be025577f78abb01193fdf94913158b4335fc43bea95" Nov 01 16:33:28.705243 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.705223 4363 scope.go:122] "RemoveContainer" containerID="e53343934f58a222255dca8b3bf908e425bb95ccac95033c90c154ea64b54c59" Nov 01 16:33:28.720554 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.720506 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf"] Nov 01 16:33:28.733749 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.733720 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-wgfxf"] Nov 01 16:33:28.765185 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.765150 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7"] Nov 01 16:33:28.778190 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.778159 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-79lv7"] Nov 01 16:33:28.795705 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.793590 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq"] Nov 01 16:33:28.799457 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.799416 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a482b-79b5ff7995-5zhdr" podUID="b8b1f822-bf88-4f11-b621-da764f27c492" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:33:28.807306 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.807278 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7babd48-8700-4398-b87b-c282cbc99c4e" path="/var/lib/kubelet/pods/d7babd48-8700-4398-b87b-c282cbc99c4e/volumes" Nov 01 16:33:28.807820 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.807807 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f32c7fe0-caa2-4a6d-80db-aec48cd9ddda" path="/var/lib/kubelet/pods/f32c7fe0-caa2-4a6d-80db-aec48cd9ddda/volumes" Nov 01 16:33:28.808138 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.808123 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-sxzwq"] Nov 01 16:33:28.813499 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.813474 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg"] Nov 01 16:33:28.821212 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:28.821189 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-r5gbg"] Nov 01 16:33:30.081180 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.081139 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f4a265-5db88cf775-7jnq2"] Nov 01 16:33:30.081827 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.081808 4363 cpu_manager.go:414] "containerMap: removing container" podUID="f32c7fe0-caa2-4a6d-80db-aec48cd9ddda" containerName="istio-proxy" Nov 01 16:33:30.081922 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.081830 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f32c7fe0-caa2-4a6d-80db-aec48cd9ddda" containerName="istio-proxy" Nov 01 16:33:30.081922 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.081864 4363 cpu_manager.go:414] "containerMap: removing container" podUID="d7babd48-8700-4398-b87b-c282cbc99c4e" containerName="istio-proxy" Nov 01 16:33:30.081922 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.081872 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7babd48-8700-4398-b87b-c282cbc99c4e" containerName="istio-proxy" Nov 01 16:33:30.081922 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.081892 4363 cpu_manager.go:414] "containerMap: removing container" podUID="a6505861-2940-41bb-8dbd-d71b797cd2c6" containerName="istio-proxy" Nov 01 16:33:30.081922 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.081900 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6505861-2940-41bb-8dbd-d71b797cd2c6" containerName="istio-proxy" Nov 01 16:33:30.081922 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.081914 4363 cpu_manager.go:414] "containerMap: removing container" podUID="9c37d94d-deb1-4a00-9d58-d91bb4b2dd95" containerName="istio-proxy" Nov 01 16:33:30.081922 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.081921 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c37d94d-deb1-4a00-9d58-d91bb4b2dd95" containerName="istio-proxy" Nov 01 16:33:30.082219 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.081945 4363 cpu_manager.go:414] "containerMap: removing container" podUID="5f69b8bb-4aa6-4713-8b2f-138c5d358858" containerName="istio-proxy" Nov 01 16:33:30.082219 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.081953 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f69b8bb-4aa6-4713-8b2f-138c5d358858" containerName="istio-proxy" Nov 01 16:33:30.082219 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.082146 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="f32c7fe0-caa2-4a6d-80db-aec48cd9ddda" containerName="istio-proxy" Nov 01 16:33:30.082219 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.082163 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7babd48-8700-4398-b87b-c282cbc99c4e" containerName="istio-proxy" Nov 01 16:33:30.082219 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.082176 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6505861-2940-41bb-8dbd-d71b797cd2c6" containerName="istio-proxy" Nov 01 16:33:30.082219 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.082185 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c37d94d-deb1-4a00-9d58-d91bb4b2dd95" containerName="istio-proxy" Nov 01 16:33:30.086419 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.086390 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4a265-5db88cf775-7jnq2" Nov 01 16:33:30.096965 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.096940 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f4a265-5db88cf775-7jnq2"] Nov 01 16:33:30.143062 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.141814 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f4a265\"" Nov 01 16:33:30.143062 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.141964 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f4a265-dockercfg-t4tm8\"" Nov 01 16:33:30.143062 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.142785 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6nw5\" (UniqueName: \"kubernetes.io/projected/10f4b6b1-4845-4f52-b90c-7650dfdb27b7-kube-api-access-n6nw5\") pod \"payload-processing-e2e-ipp-b-f4a265-5db88cf775-7jnq2\" (UID: \"10f4b6b1-4845-4f52-b90c-7650dfdb27b7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f4a265-5db88cf775-7jnq2" Nov 01 16:33:30.143062 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.142922 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10f4b6b1-4845-4f52-b90c-7650dfdb27b7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f4a265-5db88cf775-7jnq2\" (UID: \"10f4b6b1-4845-4f52-b90c-7650dfdb27b7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f4a265-5db88cf775-7jnq2" Nov 01 16:33:30.229078 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.228181 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9a482b-86b559b58f-frnp7" Nov 01 16:33:30.244237 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.244197 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6nw5\" (UniqueName: \"kubernetes.io/projected/10f4b6b1-4845-4f52-b90c-7650dfdb27b7-kube-api-access-n6nw5\") pod \"payload-processing-e2e-ipp-b-f4a265-5db88cf775-7jnq2\" (UID: \"10f4b6b1-4845-4f52-b90c-7650dfdb27b7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f4a265-5db88cf775-7jnq2" Nov 01 16:33:30.244403 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.244367 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10f4b6b1-4845-4f52-b90c-7650dfdb27b7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f4a265-5db88cf775-7jnq2\" (UID: \"10f4b6b1-4845-4f52-b90c-7650dfdb27b7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f4a265-5db88cf775-7jnq2" Nov 01 16:33:30.245301 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.245165 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10f4b6b1-4845-4f52-b90c-7650dfdb27b7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f4a265-5db88cf775-7jnq2\" (UID: \"10f4b6b1-4845-4f52-b90c-7650dfdb27b7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f4a265-5db88cf775-7jnq2" Nov 01 16:33:30.277502 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.276879 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6nw5\" (UniqueName: \"kubernetes.io/projected/10f4b6b1-4845-4f52-b90c-7650dfdb27b7-kube-api-access-n6nw5\") pod \"payload-processing-e2e-ipp-b-f4a265-5db88cf775-7jnq2\" (UID: \"10f4b6b1-4845-4f52-b90c-7650dfdb27b7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f4a265-5db88cf775-7jnq2" Nov 01 16:33:30.345378 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.345148 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e533e983-dbc0-4ce2-8d58-c74c0c0d857b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e533e983-dbc0-4ce2-8d58-c74c0c0d857b-plugins-config-volume\") pod \"e533e983-dbc0-4ce2-8d58-c74c0c0d857b\" (UID: \"e533e983-dbc0-4ce2-8d58-c74c0c0d857b\") " Nov 01 16:33:30.345616 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.345403 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e533e983-dbc0-4ce2-8d58-c74c0c0d857b-kube-api-access-kz9qc\" (UniqueName: \"kubernetes.io/projected/e533e983-dbc0-4ce2-8d58-c74c0c0d857b-kube-api-access-kz9qc\") pod \"e533e983-dbc0-4ce2-8d58-c74c0c0d857b\" (UID: \"e533e983-dbc0-4ce2-8d58-c74c0c0d857b\") " Nov 01 16:33:30.346897 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.346865 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e533e983-dbc0-4ce2-8d58-c74c0c0d857b-plugins-config-volume" pod "e533e983-dbc0-4ce2-8d58-c74c0c0d857b" (UID: "e533e983-dbc0-4ce2-8d58-c74c0c0d857b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:30.350252 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.350217 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e533e983-dbc0-4ce2-8d58-c74c0c0d857b-kube-api-access-kz9qc" pod "e533e983-dbc0-4ce2-8d58-c74c0c0d857b" (UID: "e533e983-dbc0-4ce2-8d58-c74c0c0d857b"). InnerVolumeSpecName "kube-api-access-kz9qc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:30.356906 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.356883 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a482b-79b5ff7995-5zhdr" Nov 01 16:33:30.447028 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.446931 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8b1f822-bf88-4f11-b621-da764f27c492-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8b1f822-bf88-4f11-b621-da764f27c492-plugins-config-volume\") pod \"b8b1f822-bf88-4f11-b621-da764f27c492\" (UID: \"b8b1f822-bf88-4f11-b621-da764f27c492\") " Nov 01 16:33:30.447208 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.447047 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8b1f822-bf88-4f11-b621-da764f27c492-kube-api-access-fdxcw\" (UniqueName: \"kubernetes.io/projected/b8b1f822-bf88-4f11-b621-da764f27c492-kube-api-access-fdxcw\") pod \"b8b1f822-bf88-4f11-b621-da764f27c492\" (UID: \"b8b1f822-bf88-4f11-b621-da764f27c492\") " Nov 01 16:33:30.447386 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.447361 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8b1f822-bf88-4f11-b621-da764f27c492-plugins-config-volume" pod "b8b1f822-bf88-4f11-b621-da764f27c492" (UID: "b8b1f822-bf88-4f11-b621-da764f27c492"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:30.447501 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.447486 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz9qc\" (UniqueName: \"kubernetes.io/projected/e533e983-dbc0-4ce2-8d58-c74c0c0d857b-kube-api-access-kz9qc\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:30.447569 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.447510 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e533e983-dbc0-4ce2-8d58-c74c0c0d857b-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:30.447569 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.447524 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8b1f822-bf88-4f11-b621-da764f27c492-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:30.449374 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.449347 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b1f822-bf88-4f11-b621-da764f27c492-kube-api-access-fdxcw" pod "b8b1f822-bf88-4f11-b621-da764f27c492" (UID: "b8b1f822-bf88-4f11-b621-da764f27c492"). InnerVolumeSpecName "kube-api-access-fdxcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:30.510237 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.510202 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4a265-5db88cf775-7jnq2" Nov 01 16:33:30.549458 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.549415 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdxcw\" (UniqueName: \"kubernetes.io/projected/b8b1f822-bf88-4f11-b621-da764f27c492-kube-api-access-fdxcw\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:30.693979 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.693945 4363 generic.go:356] "Generic (PLEG): container finished" podID="e533e983-dbc0-4ce2-8d58-c74c0c0d857b" containerID="7b9c924a8440d30cff88c1b74062462a018d785e9e5fa51b35dc3e3c2b251e2a" exitCode=0 Nov 01 16:33:30.694164 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.693990 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9a482b-86b559b58f-frnp7" Nov 01 16:33:30.694164 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.694056 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9a482b-86b559b58f-frnp7" event={"ID":"e533e983-dbc0-4ce2-8d58-c74c0c0d857b","Type":"ContainerDied","Data":"7b9c924a8440d30cff88c1b74062462a018d785e9e5fa51b35dc3e3c2b251e2a"} Nov 01 16:33:30.694164 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.694111 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9a482b-86b559b58f-frnp7" event={"ID":"e533e983-dbc0-4ce2-8d58-c74c0c0d857b","Type":"ContainerDied","Data":"73c29e7379d48e3c35757d692221cdf9b5b6c1e001886745379723ec8bbf399b"} Nov 01 16:33:30.694164 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.694141 4363 scope.go:122] "RemoveContainer" containerID="7b9c924a8440d30cff88c1b74062462a018d785e9e5fa51b35dc3e3c2b251e2a" Nov 01 16:33:30.695649 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.695627 4363 generic.go:356] "Generic (PLEG): container finished" podID="b8b1f822-bf88-4f11-b621-da764f27c492" containerID="30756e3465ba1fd2331edb2ab06ca3d66dd6dbe07de218a977971bdf4fd5be48" exitCode=0 Nov 01 16:33:30.695753 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.695687 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a482b-79b5ff7995-5zhdr" Nov 01 16:33:30.695753 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.695708 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a482b-79b5ff7995-5zhdr" event={"ID":"b8b1f822-bf88-4f11-b621-da764f27c492","Type":"ContainerDied","Data":"30756e3465ba1fd2331edb2ab06ca3d66dd6dbe07de218a977971bdf4fd5be48"} Nov 01 16:33:30.695872 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.695776 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a482b-79b5ff7995-5zhdr" event={"ID":"b8b1f822-bf88-4f11-b621-da764f27c492","Type":"ContainerDied","Data":"8f12c47aeb5420c6e680e46f1c54a53e246982b87a43d51fae1652e7ba568081"} Nov 01 16:33:30.705047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.705004 4363 scope.go:122] "RemoveContainer" containerID="7b9c924a8440d30cff88c1b74062462a018d785e9e5fa51b35dc3e3c2b251e2a" Nov 01 16:33:30.705336 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:33:30.705316 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b9c924a8440d30cff88c1b74062462a018d785e9e5fa51b35dc3e3c2b251e2a\": container with ID starting with 7b9c924a8440d30cff88c1b74062462a018d785e9e5fa51b35dc3e3c2b251e2a not found: ID does not exist" containerID="7b9c924a8440d30cff88c1b74062462a018d785e9e5fa51b35dc3e3c2b251e2a" Nov 01 16:33:30.705414 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.705342 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b9c924a8440d30cff88c1b74062462a018d785e9e5fa51b35dc3e3c2b251e2a"} err="failed to get container status \"7b9c924a8440d30cff88c1b74062462a018d785e9e5fa51b35dc3e3c2b251e2a\": rpc error: code = NotFound desc = could not find container \"7b9c924a8440d30cff88c1b74062462a018d785e9e5fa51b35dc3e3c2b251e2a\": container with ID starting with 7b9c924a8440d30cff88c1b74062462a018d785e9e5fa51b35dc3e3c2b251e2a not found: ID does not exist" Nov 01 16:33:30.705414 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.705357 4363 scope.go:122] "RemoveContainer" containerID="30756e3465ba1fd2331edb2ab06ca3d66dd6dbe07de218a977971bdf4fd5be48" Nov 01 16:33:30.715733 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.715716 4363 scope.go:122] "RemoveContainer" containerID="30756e3465ba1fd2331edb2ab06ca3d66dd6dbe07de218a977971bdf4fd5be48" Nov 01 16:33:30.716092 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:33:30.716065 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30756e3465ba1fd2331edb2ab06ca3d66dd6dbe07de218a977971bdf4fd5be48\": container with ID starting with 30756e3465ba1fd2331edb2ab06ca3d66dd6dbe07de218a977971bdf4fd5be48 not found: ID does not exist" containerID="30756e3465ba1fd2331edb2ab06ca3d66dd6dbe07de218a977971bdf4fd5be48" Nov 01 16:33:30.716184 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.716160 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30756e3465ba1fd2331edb2ab06ca3d66dd6dbe07de218a977971bdf4fd5be48"} err="failed to get container status \"30756e3465ba1fd2331edb2ab06ca3d66dd6dbe07de218a977971bdf4fd5be48\": rpc error: code = NotFound desc = could not find container \"30756e3465ba1fd2331edb2ab06ca3d66dd6dbe07de218a977971bdf4fd5be48\": container with ID starting with 30756e3465ba1fd2331edb2ab06ca3d66dd6dbe07de218a977971bdf4fd5be48 not found: ID does not exist" Nov 01 16:33:30.719519 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.719495 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-9a482b-86b559b58f-frnp7"] Nov 01 16:33:30.723027 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.722992 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-9a482b-86b559b58f-frnp7"] Nov 01 16:33:30.733157 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.733134 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a482b-79b5ff7995-5zhdr"] Nov 01 16:33:30.735779 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.735759 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a482b-79b5ff7995-5zhdr"] Nov 01 16:33:30.811505 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.811481 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c37d94d-deb1-4a00-9d58-d91bb4b2dd95" path="/var/lib/kubelet/pods/9c37d94d-deb1-4a00-9d58-d91bb4b2dd95/volumes" Nov 01 16:33:30.812257 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.812232 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6505861-2940-41bb-8dbd-d71b797cd2c6" path="/var/lib/kubelet/pods/a6505861-2940-41bb-8dbd-d71b797cd2c6/volumes" Nov 01 16:33:30.812704 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.812684 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8b1f822-bf88-4f11-b621-da764f27c492" path="/var/lib/kubelet/pods/b8b1f822-bf88-4f11-b621-da764f27c492/volumes" Nov 01 16:33:30.814516 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.814419 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e533e983-dbc0-4ce2-8d58-c74c0c0d857b" path="/var/lib/kubelet/pods/e533e983-dbc0-4ce2-8d58-c74c0c0d857b/volumes" Nov 01 16:33:30.815174 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:30.815157 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f4a265-5db88cf775-7jnq2"] Nov 01 16:33:31.037219 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:31.037113 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a498ae42-krb6v"] Nov 01 16:33:31.041381 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:31.041343 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a498ae42-krb6v"] Nov 01 16:33:31.706738 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:31.706707 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4a265-5db88cf775-7jnq2" event={"ID":"10f4b6b1-4845-4f52-b90c-7650dfdb27b7","Type":"ContainerStarted","Data":"837faf8431a56e7b78d770e6a9f65b0d00f62cf76874b350f51badd91f0d837c"} Nov 01 16:33:31.707243 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:31.706754 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4a265-5db88cf775-7jnq2" event={"ID":"10f4b6b1-4845-4f52-b90c-7650dfdb27b7","Type":"ContainerStarted","Data":"15d59c997e2bd7dba02169eab843024bf10dbed308f7cbcc50244edba6bbad0a"} Nov 01 16:33:31.707243 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:31.706912 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4a265-5db88cf775-7jnq2" Nov 01 16:33:31.723424 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:31.723382 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4a265-5db88cf775-7jnq2" podStartSLOduration=1.723369222 podStartE2EDuration="1.723369222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:33:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:33:31.720091961 +0000 UTC m=+2745.051829611" watchObservedRunningTime="2026-11-01 16:33:31.723369222 +0000 UTC m=+2745.055106874" Nov 01 16:33:32.816205 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:32.815834 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="669d4672-f30d-4c7f-9a8c-768a472da306" path="/var/lib/kubelet/pods/669d4672-f30d-4c7f-9a8c-768a472da306/volumes" Nov 01 16:33:32.986470 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:32.986438 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-f01de5-65884f55d5-pbgx8"] Nov 01 16:33:33.549459 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:33.549419 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bd831a-56899656f5-jhvtb" podUID="bc5eb74e-c5d9-4210-a626-94e8279c3a36" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:33:35.741574 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:35.741540 4363 generic.go:356] "Generic (PLEG): container finished" podID="bc5eb74e-c5d9-4210-a626-94e8279c3a36" containerID="f1b7825cb139ffad509aba5ec68f675537e4175cd6a5b8bbd89b67c7316cebe0" exitCode=0 Nov 01 16:33:35.742073 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:35.741608 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bd831a-56899656f5-jhvtb" event={"ID":"bc5eb74e-c5d9-4210-a626-94e8279c3a36","Type":"ContainerDied","Data":"f1b7825cb139ffad509aba5ec68f675537e4175cd6a5b8bbd89b67c7316cebe0"} Nov 01 16:33:35.824431 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:35.824403 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bd831a-56899656f5-jhvtb" Nov 01 16:33:35.931632 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:35.931601 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc5eb74e-c5d9-4210-a626-94e8279c3a36-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc5eb74e-c5d9-4210-a626-94e8279c3a36-plugins-config-volume\") pod \"bc5eb74e-c5d9-4210-a626-94e8279c3a36\" (UID: \"bc5eb74e-c5d9-4210-a626-94e8279c3a36\") " Nov 01 16:33:35.931790 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:35.931667 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc5eb74e-c5d9-4210-a626-94e8279c3a36-kube-api-access-8g7pr\" (UniqueName: \"kubernetes.io/projected/bc5eb74e-c5d9-4210-a626-94e8279c3a36-kube-api-access-8g7pr\") pod \"bc5eb74e-c5d9-4210-a626-94e8279c3a36\" (UID: \"bc5eb74e-c5d9-4210-a626-94e8279c3a36\") " Nov 01 16:33:35.932099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:35.932078 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc5eb74e-c5d9-4210-a626-94e8279c3a36-plugins-config-volume" pod "bc5eb74e-c5d9-4210-a626-94e8279c3a36" (UID: "bc5eb74e-c5d9-4210-a626-94e8279c3a36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:35.933832 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:35.933807 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc5eb74e-c5d9-4210-a626-94e8279c3a36-kube-api-access-8g7pr" pod "bc5eb74e-c5d9-4210-a626-94e8279c3a36" (UID: "bc5eb74e-c5d9-4210-a626-94e8279c3a36"). InnerVolumeSpecName "kube-api-access-8g7pr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:36.040183 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:36.035933 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc5eb74e-c5d9-4210-a626-94e8279c3a36-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:36.040183 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:36.035972 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g7pr\" (UniqueName: \"kubernetes.io/projected/bc5eb74e-c5d9-4210-a626-94e8279c3a36-kube-api-access-8g7pr\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:36.749416 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:36.749387 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bd831a-56899656f5-jhvtb" Nov 01 16:33:36.749879 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:36.749414 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bd831a-56899656f5-jhvtb" event={"ID":"bc5eb74e-c5d9-4210-a626-94e8279c3a36","Type":"ContainerDied","Data":"cf89c24952a1e8f1e5f1b9519862de32539e69da91c7a168986c724c4b5169dc"} Nov 01 16:33:36.749879 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:36.749480 4363 scope.go:122] "RemoveContainer" containerID="f1b7825cb139ffad509aba5ec68f675537e4175cd6a5b8bbd89b67c7316cebe0" Nov 01 16:33:36.772483 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:36.772456 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-bd831a-56899656f5-jhvtb"] Nov 01 16:33:36.778676 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:36.778652 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-bd831a-56899656f5-jhvtb"] Nov 01 16:33:36.797647 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:36.797622 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc5eb74e-c5d9-4210-a626-94e8279c3a36" path="/var/lib/kubelet/pods/bc5eb74e-c5d9-4210-a626-94e8279c3a36/volumes" Nov 01 16:33:37.987128 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:37.987084 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f01de5-65884f55d5-pbgx8" podUID="387b6adf-9404-414e-ae75-0b48e938e77a" containerName="payload-processing" containerID="cri-o://7e12c927caea0f8d330caf6d5d8dbbf8e12e49f79150bd84d09240a2e5b52fbb" gracePeriod=25 Nov 01 16:33:42.204509 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:42.204453 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4"] Nov 01 16:33:42.205073 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:42.204671 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" podUID="5f58c340-e931-42d5-9322-2f60420216fb" containerName="istio-proxy" containerID="cri-o://6cda8bb655a705c48dbdeb3c9ed430060e56dbe87850ef256169e37234f887dc" gracePeriod=30 Nov 01 16:33:42.719430 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:42.719402 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4a265-5db88cf775-7jnq2" Nov 01 16:33:46.207203 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:46.207165 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5"] Nov 01 16:33:46.207709 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:46.207483 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" podUID="a850693a-07f7-4fdf-9ad3-4d08559c9737" containerName="istio-proxy" containerID="cri-o://50b4d3471a0b773905f9182ceb9f7b6c32429f63af7fbc8edd4e4281bc597a07" gracePeriod=30 Nov 01 16:33:47.542219 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:47.542194 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" Nov 01 16:33:47.606613 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:47.606446 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f58c340-e931-42d5-9322-2f60420216fb-kube-api-access-j944s\" (UniqueName: \"kubernetes.io/projected/5f58c340-e931-42d5-9322-2f60420216fb-kube-api-access-j944s\") pod \"5f58c340-e931-42d5-9322-2f60420216fb\" (UID: \"5f58c340-e931-42d5-9322-2f60420216fb\") " Nov 01 16:33:47.606613 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:47.606547 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f58c340-e931-42d5-9322-2f60420216fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f58c340-e931-42d5-9322-2f60420216fb-istiod-ca-cert\") pod \"5f58c340-e931-42d5-9322-2f60420216fb\" (UID: \"5f58c340-e931-42d5-9322-2f60420216fb\") " Nov 01 16:33:47.607422 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:47.607381 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f58c340-e931-42d5-9322-2f60420216fb-istiod-ca-cert" pod "5f58c340-e931-42d5-9322-2f60420216fb" (UID: "5f58c340-e931-42d5-9322-2f60420216fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:47.607650 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:47.607619 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f58c340-e931-42d5-9322-2f60420216fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f58c340-e931-42d5-9322-2f60420216fb-istio-envoy\") pod \"5f58c340-e931-42d5-9322-2f60420216fb\" (UID: \"5f58c340-e931-42d5-9322-2f60420216fb\") " Nov 01 16:33:47.607852 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:47.607838 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f58c340-e931-42d5-9322-2f60420216fb-istio-token\" (UniqueName: \"kubernetes.io/projected/5f58c340-e931-42d5-9322-2f60420216fb-istio-token\") pod \"5f58c340-e931-42d5-9322-2f60420216fb\" (UID: \"5f58c340-e931-42d5-9322-2f60420216fb\") " Nov 01 16:33:47.607973 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:47.607960 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f58c340-e931-42d5-9322-2f60420216fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f58c340-e931-42d5-9322-2f60420216fb-istio-data\") pod \"5f58c340-e931-42d5-9322-2f60420216fb\" (UID: \"5f58c340-e931-42d5-9322-2f60420216fb\") " Nov 01 16:33:47.608120 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:47.608106 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f58c340-e931-42d5-9322-2f60420216fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f58c340-e931-42d5-9322-2f60420216fb-credential-socket\") pod \"5f58c340-e931-42d5-9322-2f60420216fb\" (UID: \"5f58c340-e931-42d5-9322-2f60420216fb\") " Nov 01 16:33:47.608276 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:47.608225 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f58c340-e931-42d5-9322-2f60420216fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f58c340-e931-42d5-9322-2f60420216fb-istio-podinfo\") pod \"5f58c340-e931-42d5-9322-2f60420216fb\" (UID: \"5f58c340-e931-42d5-9322-2f60420216fb\") " Nov 01 16:33:47.611486 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:47.611177 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f58c340-e931-42d5-9322-2f60420216fb-istio-data" pod "5f58c340-e931-42d5-9322-2f60420216fb" (UID: "5f58c340-e931-42d5-9322-2f60420216fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:47.611759 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:47.611727 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f58c340-e931-42d5-9322-2f60420216fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f58c340-e931-42d5-9322-2f60420216fb-workload-socket\") pod \"5f58c340-e931-42d5-9322-2f60420216fb\" (UID: \"5f58c340-e931-42d5-9322-2f60420216fb\") " Nov 01 16:33:47.612010 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:47.611936 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f58c340-e931-42d5-9322-2f60420216fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f58c340-e931-42d5-9322-2f60420216fb-workload-certs\") pod \"5f58c340-e931-42d5-9322-2f60420216fb\" (UID: \"5f58c340-e931-42d5-9322-2f60420216fb\") " Nov 01 16:33:47.612703 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:47.612685 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f58c340-e931-42d5-9322-2f60420216fb-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:47.612834 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:47.612822 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f58c340-e931-42d5-9322-2f60420216fb-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:47.613245 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:47.613226 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f58c340-e931-42d5-9322-2f60420216fb-workload-certs" pod "5f58c340-e931-42d5-9322-2f60420216fb" (UID: "5f58c340-e931-42d5-9322-2f60420216fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:47.613964 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:47.613924 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f58c340-e931-42d5-9322-2f60420216fb-workload-socket" pod "5f58c340-e931-42d5-9322-2f60420216fb" (UID: "5f58c340-e931-42d5-9322-2f60420216fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:47.614293 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:47.614266 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f58c340-e931-42d5-9322-2f60420216fb-credential-socket" pod "5f58c340-e931-42d5-9322-2f60420216fb" (UID: "5f58c340-e931-42d5-9322-2f60420216fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:47.616067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:47.615983 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f58c340-e931-42d5-9322-2f60420216fb-istio-envoy" pod "5f58c340-e931-42d5-9322-2f60420216fb" (UID: "5f58c340-e931-42d5-9322-2f60420216fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:47.616184 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:47.616104 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f58c340-e931-42d5-9322-2f60420216fb-istio-token" pod "5f58c340-e931-42d5-9322-2f60420216fb" (UID: "5f58c340-e931-42d5-9322-2f60420216fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:47.617248 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:47.617224 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f58c340-e931-42d5-9322-2f60420216fb-istio-podinfo" pod "5f58c340-e931-42d5-9322-2f60420216fb" (UID: "5f58c340-e931-42d5-9322-2f60420216fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:33:47.617465 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:47.617440 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f58c340-e931-42d5-9322-2f60420216fb-kube-api-access-j944s" pod "5f58c340-e931-42d5-9322-2f60420216fb" (UID: "5f58c340-e931-42d5-9322-2f60420216fb"). InnerVolumeSpecName "kube-api-access-j944s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:47.714859 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:47.714797 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f58c340-e931-42d5-9322-2f60420216fb-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:47.714859 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:47.714841 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j944s\" (UniqueName: \"kubernetes.io/projected/5f58c340-e931-42d5-9322-2f60420216fb-kube-api-access-j944s\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:47.714859 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:47.714858 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f58c340-e931-42d5-9322-2f60420216fb-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:47.714859 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:47.714871 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f58c340-e931-42d5-9322-2f60420216fb-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:47.715234 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:47.714885 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f58c340-e931-42d5-9322-2f60420216fb-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:47.715234 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:47.714898 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f58c340-e931-42d5-9322-2f60420216fb-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:47.715234 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:47.714911 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f58c340-e931-42d5-9322-2f60420216fb-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:47.834319 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:47.834230 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f01de5-65884f55d5-pbgx8" podUID="387b6adf-9404-414e-ae75-0b48e938e77a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:33:47.870448 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:47.870409 4363 generic.go:356] "Generic (PLEG): container finished" podID="5f58c340-e931-42d5-9322-2f60420216fb" containerID="6cda8bb655a705c48dbdeb3c9ed430060e56dbe87850ef256169e37234f887dc" exitCode=0 Nov 01 16:33:47.870646 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:47.870473 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" Nov 01 16:33:47.870646 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:47.870490 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" event={"ID":"5f58c340-e931-42d5-9322-2f60420216fb","Type":"ContainerDied","Data":"6cda8bb655a705c48dbdeb3c9ed430060e56dbe87850ef256169e37234f887dc"} Nov 01 16:33:47.870646 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:47.870572 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4" event={"ID":"5f58c340-e931-42d5-9322-2f60420216fb","Type":"ContainerDied","Data":"6633df2e2ab7d37c373d5f76fbf4e0f615a7557ef4b9eb2a32769af0e58310b1"} Nov 01 16:33:47.870646 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:47.870605 4363 scope.go:122] "RemoveContainer" containerID="6cda8bb655a705c48dbdeb3c9ed430060e56dbe87850ef256169e37234f887dc" Nov 01 16:33:47.914425 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:47.911143 4363 scope.go:122] "RemoveContainer" containerID="87a8952f9e0890633201b4cb35a1a240e85574976da72c9b8522e064ff91ab4e" Nov 01 16:33:47.923174 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:47.923003 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4"] Nov 01 16:33:47.930621 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:47.930591 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-c8537c-openshift-default-5bb45fc5c7-fbfw4"] Nov 01 16:33:47.933602 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:47.933575 4363 scope.go:122] "RemoveContainer" containerID="6cda8bb655a705c48dbdeb3c9ed430060e56dbe87850ef256169e37234f887dc" Nov 01 16:33:47.933975 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:33:47.933953 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cda8bb655a705c48dbdeb3c9ed430060e56dbe87850ef256169e37234f887dc\": container with ID starting with 6cda8bb655a705c48dbdeb3c9ed430060e56dbe87850ef256169e37234f887dc not found: ID does not exist" containerID="6cda8bb655a705c48dbdeb3c9ed430060e56dbe87850ef256169e37234f887dc" Nov 01 16:33:47.934110 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:47.933981 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cda8bb655a705c48dbdeb3c9ed430060e56dbe87850ef256169e37234f887dc"} err="failed to get container status \"6cda8bb655a705c48dbdeb3c9ed430060e56dbe87850ef256169e37234f887dc\": rpc error: code = NotFound desc = could not find container \"6cda8bb655a705c48dbdeb3c9ed430060e56dbe87850ef256169e37234f887dc\": container with ID starting with 6cda8bb655a705c48dbdeb3c9ed430060e56dbe87850ef256169e37234f887dc not found: ID does not exist" Nov 01 16:33:47.934110 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:47.934000 4363 scope.go:122] "RemoveContainer" containerID="87a8952f9e0890633201b4cb35a1a240e85574976da72c9b8522e064ff91ab4e" Nov 01 16:33:47.934278 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:33:47.934261 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87a8952f9e0890633201b4cb35a1a240e85574976da72c9b8522e064ff91ab4e\": container with ID starting with 87a8952f9e0890633201b4cb35a1a240e85574976da72c9b8522e064ff91ab4e not found: ID does not exist" containerID="87a8952f9e0890633201b4cb35a1a240e85574976da72c9b8522e064ff91ab4e" Nov 01 16:33:47.934339 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:47.934280 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87a8952f9e0890633201b4cb35a1a240e85574976da72c9b8522e064ff91ab4e"} err="failed to get container status \"87a8952f9e0890633201b4cb35a1a240e85574976da72c9b8522e064ff91ab4e\": rpc error: code = NotFound desc = could not find container \"87a8952f9e0890633201b4cb35a1a240e85574976da72c9b8522e064ff91ab4e\": container with ID starting with 87a8952f9e0890633201b4cb35a1a240e85574976da72c9b8522e064ff91ab4e not found: ID does not exist" Nov 01 16:33:48.335120 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:33:48.335085 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:33:48.811873 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:48.811833 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f58c340-e931-42d5-9322-2f60420216fb" path="/var/lib/kubelet/pods/5f58c340-e931-42d5-9322-2f60420216fb/volumes" Nov 01 16:33:51.270514 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:51.270479 4363 scope.go:122] "RemoveContainer" containerID="3565b660bbb4998cdc35a591ec6e4775275d5dede96d5fafd564b06b72f81908" Nov 01 16:33:51.324010 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:51.323981 4363 scope.go:122] "RemoveContainer" containerID="e2f7aa1ed9e33848458123834c55d962487b1f1e737a316ce54325ee41b4f244" Nov 01 16:33:51.335888 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:51.335861 4363 scope.go:122] "RemoveContainer" containerID="c9f912e8dab655c9f6bfa3c9a5d582105b5df6bdffcd50c4c234320aa606a9f3" Nov 01 16:33:51.344901 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:51.344881 4363 scope.go:122] "RemoveContainer" containerID="6ba8a563e37b7d8b4e8f0a5869b2f43fed73a62556405e59548bfb3ffa3b15ef" Nov 01 16:33:51.631048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:51.630956 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" Nov 01 16:33:51.633100 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:51.633076 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr"] Nov 01 16:33:51.647063 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:51.647037 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6"] Nov 01 16:33:51.726125 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:51.726085 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a850693a-07f7-4fdf-9ad3-4d08559c9737-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a850693a-07f7-4fdf-9ad3-4d08559c9737-workload-socket\") pod \"a850693a-07f7-4fdf-9ad3-4d08559c9737\" (UID: \"a850693a-07f7-4fdf-9ad3-4d08559c9737\") " Nov 01 16:33:51.726455 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:51.726432 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a850693a-07f7-4fdf-9ad3-4d08559c9737-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a850693a-07f7-4fdf-9ad3-4d08559c9737-istio-data\") pod \"a850693a-07f7-4fdf-9ad3-4d08559c9737\" (UID: \"a850693a-07f7-4fdf-9ad3-4d08559c9737\") " Nov 01 16:33:51.726672 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:51.726657 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a850693a-07f7-4fdf-9ad3-4d08559c9737-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a850693a-07f7-4fdf-9ad3-4d08559c9737-istio-podinfo\") pod \"a850693a-07f7-4fdf-9ad3-4d08559c9737\" (UID: \"a850693a-07f7-4fdf-9ad3-4d08559c9737\") " Nov 01 16:33:51.726817 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:51.726802 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a850693a-07f7-4fdf-9ad3-4d08559c9737-istio-token\" (UniqueName: \"kubernetes.io/projected/a850693a-07f7-4fdf-9ad3-4d08559c9737-istio-token\") pod \"a850693a-07f7-4fdf-9ad3-4d08559c9737\" (UID: \"a850693a-07f7-4fdf-9ad3-4d08559c9737\") " Nov 01 16:33:51.727063 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:51.727044 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a850693a-07f7-4fdf-9ad3-4d08559c9737-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a850693a-07f7-4fdf-9ad3-4d08559c9737-istiod-ca-cert\") pod \"a850693a-07f7-4fdf-9ad3-4d08559c9737\" (UID: \"a850693a-07f7-4fdf-9ad3-4d08559c9737\") " Nov 01 16:33:51.727214 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:51.727200 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a850693a-07f7-4fdf-9ad3-4d08559c9737-kube-api-access-dmk4k\" (UniqueName: \"kubernetes.io/projected/a850693a-07f7-4fdf-9ad3-4d08559c9737-kube-api-access-dmk4k\") pod \"a850693a-07f7-4fdf-9ad3-4d08559c9737\" (UID: \"a850693a-07f7-4fdf-9ad3-4d08559c9737\") " Nov 01 16:33:51.727341 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:51.727328 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a850693a-07f7-4fdf-9ad3-4d08559c9737-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a850693a-07f7-4fdf-9ad3-4d08559c9737-workload-certs\") pod \"a850693a-07f7-4fdf-9ad3-4d08559c9737\" (UID: \"a850693a-07f7-4fdf-9ad3-4d08559c9737\") " Nov 01 16:33:51.727542 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:51.727525 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a850693a-07f7-4fdf-9ad3-4d08559c9737-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a850693a-07f7-4fdf-9ad3-4d08559c9737-credential-socket\") pod \"a850693a-07f7-4fdf-9ad3-4d08559c9737\" (UID: \"a850693a-07f7-4fdf-9ad3-4d08559c9737\") " Nov 01 16:33:51.727723 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:51.727700 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a850693a-07f7-4fdf-9ad3-4d08559c9737-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a850693a-07f7-4fdf-9ad3-4d08559c9737-istio-envoy\") pod \"a850693a-07f7-4fdf-9ad3-4d08559c9737\" (UID: \"a850693a-07f7-4fdf-9ad3-4d08559c9737\") " Nov 01 16:33:51.728578 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:51.728274 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a850693a-07f7-4fdf-9ad3-4d08559c9737-workload-socket" pod "a850693a-07f7-4fdf-9ad3-4d08559c9737" (UID: "a850693a-07f7-4fdf-9ad3-4d08559c9737"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:51.729283 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:51.728942 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a850693a-07f7-4fdf-9ad3-4d08559c9737-istio-data" pod "a850693a-07f7-4fdf-9ad3-4d08559c9737" (UID: "a850693a-07f7-4fdf-9ad3-4d08559c9737"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:51.730027 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:51.729990 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a850693a-07f7-4fdf-9ad3-4d08559c9737-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:51.730173 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:51.730158 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a850693a-07f7-4fdf-9ad3-4d08559c9737-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:51.731073 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:51.731046 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a850693a-07f7-4fdf-9ad3-4d08559c9737-istiod-ca-cert" pod "a850693a-07f7-4fdf-9ad3-4d08559c9737" (UID: "a850693a-07f7-4fdf-9ad3-4d08559c9737"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:51.731492 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:51.731303 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a850693a-07f7-4fdf-9ad3-4d08559c9737-workload-certs" pod "a850693a-07f7-4fdf-9ad3-4d08559c9737" (UID: "a850693a-07f7-4fdf-9ad3-4d08559c9737"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:51.731492 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:51.731452 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a850693a-07f7-4fdf-9ad3-4d08559c9737-credential-socket" pod "a850693a-07f7-4fdf-9ad3-4d08559c9737" (UID: "a850693a-07f7-4fdf-9ad3-4d08559c9737"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:51.731774 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:51.731750 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a850693a-07f7-4fdf-9ad3-4d08559c9737-istio-token" pod "a850693a-07f7-4fdf-9ad3-4d08559c9737" (UID: "a850693a-07f7-4fdf-9ad3-4d08559c9737"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:51.734242 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:51.733940 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a850693a-07f7-4fdf-9ad3-4d08559c9737-istio-podinfo" pod "a850693a-07f7-4fdf-9ad3-4d08559c9737" (UID: "a850693a-07f7-4fdf-9ad3-4d08559c9737"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:33:51.735353 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:51.735210 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a850693a-07f7-4fdf-9ad3-4d08559c9737-istio-envoy" pod "a850693a-07f7-4fdf-9ad3-4d08559c9737" (UID: "a850693a-07f7-4fdf-9ad3-4d08559c9737"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:51.735353 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:51.735323 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a850693a-07f7-4fdf-9ad3-4d08559c9737-kube-api-access-dmk4k" pod "a850693a-07f7-4fdf-9ad3-4d08559c9737" (UID: "a850693a-07f7-4fdf-9ad3-4d08559c9737"). InnerVolumeSpecName "kube-api-access-dmk4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:51.832253 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:51.832076 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a850693a-07f7-4fdf-9ad3-4d08559c9737-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:51.832253 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:51.832109 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a850693a-07f7-4fdf-9ad3-4d08559c9737-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:51.832253 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:51.832119 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmk4k\" (UniqueName: \"kubernetes.io/projected/a850693a-07f7-4fdf-9ad3-4d08559c9737-kube-api-access-dmk4k\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:51.832253 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:51.832128 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a850693a-07f7-4fdf-9ad3-4d08559c9737-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:51.832253 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:51.832138 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a850693a-07f7-4fdf-9ad3-4d08559c9737-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:51.832253 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:51.832150 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a850693a-07f7-4fdf-9ad3-4d08559c9737-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:51.832253 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:51.832178 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a850693a-07f7-4fdf-9ad3-4d08559c9737-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:51.952667 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:51.952626 4363 generic.go:356] "Generic (PLEG): container finished" podID="a850693a-07f7-4fdf-9ad3-4d08559c9737" containerID="50b4d3471a0b773905f9182ceb9f7b6c32429f63af7fbc8edd4e4281bc597a07" exitCode=0 Nov 01 16:33:51.952857 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:51.952685 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" Nov 01 16:33:51.952857 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:51.952689 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" event={"ID":"a850693a-07f7-4fdf-9ad3-4d08559c9737","Type":"ContainerDied","Data":"50b4d3471a0b773905f9182ceb9f7b6c32429f63af7fbc8edd4e4281bc597a07"} Nov 01 16:33:51.952857 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:51.952742 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5" event={"ID":"a850693a-07f7-4fdf-9ad3-4d08559c9737","Type":"ContainerDied","Data":"857eb2e018576cd281fb6632826220eb3dbb12d76d675f72671369ba84bb7bc7"} Nov 01 16:33:51.952857 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:51.952774 4363 scope.go:122] "RemoveContainer" containerID="50b4d3471a0b773905f9182ceb9f7b6c32429f63af7fbc8edd4e4281bc597a07" Nov 01 16:33:51.965049 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:51.965011 4363 scope.go:122] "RemoveContainer" containerID="50b4d3471a0b773905f9182ceb9f7b6c32429f63af7fbc8edd4e4281bc597a07" Nov 01 16:33:51.965366 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:33:51.965341 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50b4d3471a0b773905f9182ceb9f7b6c32429f63af7fbc8edd4e4281bc597a07\": container with ID starting with 50b4d3471a0b773905f9182ceb9f7b6c32429f63af7fbc8edd4e4281bc597a07 not found: ID does not exist" containerID="50b4d3471a0b773905f9182ceb9f7b6c32429f63af7fbc8edd4e4281bc597a07" Nov 01 16:33:51.965420 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:51.965378 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50b4d3471a0b773905f9182ceb9f7b6c32429f63af7fbc8edd4e4281bc597a07"} err="failed to get container status \"50b4d3471a0b773905f9182ceb9f7b6c32429f63af7fbc8edd4e4281bc597a07\": rpc error: code = NotFound desc = could not find container \"50b4d3471a0b773905f9182ceb9f7b6c32429f63af7fbc8edd4e4281bc597a07\": container with ID starting with 50b4d3471a0b773905f9182ceb9f7b6c32429f63af7fbc8edd4e4281bc597a07 not found: ID does not exist" Nov 01 16:33:51.986040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:51.978803 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5"] Nov 01 16:33:52.006752 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:52.006723 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-qtrd5"] Nov 01 16:33:52.799276 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:52.799238 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a850693a-07f7-4fdf-9ad3-4d08559c9737" path="/var/lib/kubelet/pods/a850693a-07f7-4fdf-9ad3-4d08559c9737/volumes" Nov 01 16:33:53.076581 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:53.076490 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-dfff59-688dc9df89-4vk2r"] Nov 01 16:33:53.883757 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:53.883720 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f4a265-5db88cf775-7jnq2"] Nov 01 16:33:55.986289 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:55.986199 4363 generic.go:356] "Generic (PLEG): container finished" podID="87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519" containerID="65f99fc6091a1226f46e39229bd199f9afa4fd6033a850eedb0a5ddbaed92fea" exitCode=0 Nov 01 16:33:55.986289 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:55.986231 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" event={"ID":"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519","Type":"ContainerDied","Data":"65f99fc6091a1226f46e39229bd199f9afa4fd6033a850eedb0a5ddbaed92fea"} Nov 01 16:33:55.986856 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:55.986638 4363 scope.go:122] "RemoveContainer" containerID="65f99fc6091a1226f46e39229bd199f9afa4fd6033a850eedb0a5ddbaed92fea" Nov 01 16:33:55.988230 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:55.988203 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:33:55.988375 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:55.988289 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:33:55.988375 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:55.988342 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:33:56.318509 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:56.318424 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-7c6884bdff-d444d"] Nov 01 16:33:56.319359 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:56.319338 4363 cpu_manager.go:414] "containerMap: removing container" podUID="5f58c340-e931-42d5-9322-2f60420216fb" containerName="istio-proxy" Nov 01 16:33:56.319420 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:56.319364 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f58c340-e931-42d5-9322-2f60420216fb" containerName="istio-proxy" Nov 01 16:33:56.319420 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:56.319383 4363 cpu_manager.go:414] "containerMap: removing container" podUID="5f58c340-e931-42d5-9322-2f60420216fb" containerName="istio-proxy" Nov 01 16:33:56.319420 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:56.319391 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f58c340-e931-42d5-9322-2f60420216fb" containerName="istio-proxy" Nov 01 16:33:56.319420 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:56.319410 4363 cpu_manager.go:414] "containerMap: removing container" podUID="a850693a-07f7-4fdf-9ad3-4d08559c9737" containerName="istio-proxy" Nov 01 16:33:56.319420 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:56.319417 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a850693a-07f7-4fdf-9ad3-4d08559c9737" containerName="istio-proxy" Nov 01 16:33:56.319592 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:56.319431 4363 cpu_manager.go:414] "containerMap: removing container" podUID="b8b1f822-bf88-4f11-b621-da764f27c492" containerName="payload-pre-processing" Nov 01 16:33:56.319592 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:56.319439 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8b1f822-bf88-4f11-b621-da764f27c492" containerName="payload-pre-processing" Nov 01 16:33:56.319592 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:56.319456 4363 cpu_manager.go:414] "containerMap: removing container" podUID="bc5eb74e-c5d9-4210-a626-94e8279c3a36" containerName="payload-pre-processing" Nov 01 16:33:56.319592 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:56.319464 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc5eb74e-c5d9-4210-a626-94e8279c3a36" containerName="payload-pre-processing" Nov 01 16:33:56.319592 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:56.319473 4363 cpu_manager.go:414] "containerMap: removing container" podUID="e533e983-dbc0-4ce2-8d58-c74c0c0d857b" containerName="payload-processing" Nov 01 16:33:56.319592 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:56.319480 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e533e983-dbc0-4ce2-8d58-c74c0c0d857b" containerName="payload-processing" Nov 01 16:33:56.319788 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:56.319620 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f58c340-e931-42d5-9322-2f60420216fb" containerName="istio-proxy" Nov 01 16:33:56.319788 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:56.319633 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f58c340-e931-42d5-9322-2f60420216fb" containerName="istio-proxy" Nov 01 16:33:56.319788 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:56.319643 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="a850693a-07f7-4fdf-9ad3-4d08559c9737" containerName="istio-proxy" Nov 01 16:33:56.319788 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:56.319652 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8b1f822-bf88-4f11-b621-da764f27c492" containerName="payload-pre-processing" Nov 01 16:33:56.319788 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:56.319665 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc5eb74e-c5d9-4210-a626-94e8279c3a36" containerName="payload-pre-processing" Nov 01 16:33:56.319788 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:56.319672 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="e533e983-dbc0-4ce2-8d58-c74c0c0d857b" containerName="payload-processing" Nov 01 16:33:56.323635 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:56.323610 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-7c6884bdff-d444d" Nov 01 16:33:56.329201 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:56.329177 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f4a265-dockercfg-qppjd\"" Nov 01 16:33:56.333459 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:56.333433 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-7c6884bdff-d444d"] Nov 01 16:33:56.336371 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:56.336347 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f4a265-dc856b996-7fdck"] Nov 01 16:33:56.340534 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:56.340511 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4a265-dc856b996-7fdck" Nov 01 16:33:56.350700 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:56.350513 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f4a265-dc856b996-7fdck"] Nov 01 16:33:56.403999 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:56.403962 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcrbh\" (UniqueName: \"kubernetes.io/projected/c2d00eaa-b184-4176-9944-ef381e8eaec0-kube-api-access-vcrbh\") pod \"payload-processing-e2e-ipp-b-f4a265-dc856b996-7fdck\" (UID: \"c2d00eaa-b184-4176-9944-ef381e8eaec0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f4a265-dc856b996-7fdck" Nov 01 16:33:56.404210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:56.404030 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0240fa16-cc94-4d43-a3a8-ae27ca53f6ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f4a265-7c6884bdff-d444d\" (UID: \"0240fa16-cc94-4d43-a3a8-ae27ca53f6ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-7c6884bdff-d444d" Nov 01 16:33:56.404210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:56.404186 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7f97\" (UniqueName: \"kubernetes.io/projected/0240fa16-cc94-4d43-a3a8-ae27ca53f6ec-kube-api-access-m7f97\") pod \"payload-pre-processing-e2e-ipp-b-f4a265-7c6884bdff-d444d\" (UID: \"0240fa16-cc94-4d43-a3a8-ae27ca53f6ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-7c6884bdff-d444d" Nov 01 16:33:56.404350 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:56.404231 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2d00eaa-b184-4176-9944-ef381e8eaec0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f4a265-dc856b996-7fdck\" (UID: \"c2d00eaa-b184-4176-9944-ef381e8eaec0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f4a265-dc856b996-7fdck" Nov 01 16:33:56.506088 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:56.506032 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcrbh\" (UniqueName: \"kubernetes.io/projected/c2d00eaa-b184-4176-9944-ef381e8eaec0-kube-api-access-vcrbh\") pod \"payload-processing-e2e-ipp-b-f4a265-dc856b996-7fdck\" (UID: \"c2d00eaa-b184-4176-9944-ef381e8eaec0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f4a265-dc856b996-7fdck" Nov 01 16:33:56.506289 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:56.506131 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0240fa16-cc94-4d43-a3a8-ae27ca53f6ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f4a265-7c6884bdff-d444d\" (UID: \"0240fa16-cc94-4d43-a3a8-ae27ca53f6ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-7c6884bdff-d444d" Nov 01 16:33:56.506289 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:56.506222 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7f97\" (UniqueName: \"kubernetes.io/projected/0240fa16-cc94-4d43-a3a8-ae27ca53f6ec-kube-api-access-m7f97\") pod \"payload-pre-processing-e2e-ipp-b-f4a265-7c6884bdff-d444d\" (UID: \"0240fa16-cc94-4d43-a3a8-ae27ca53f6ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-7c6884bdff-d444d" Nov 01 16:33:56.506289 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:56.506250 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2d00eaa-b184-4176-9944-ef381e8eaec0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f4a265-dc856b996-7fdck\" (UID: \"c2d00eaa-b184-4176-9944-ef381e8eaec0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f4a265-dc856b996-7fdck" Nov 01 16:33:56.507223 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:56.507181 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2d00eaa-b184-4176-9944-ef381e8eaec0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f4a265-dc856b996-7fdck\" (UID: \"c2d00eaa-b184-4176-9944-ef381e8eaec0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f4a265-dc856b996-7fdck" Nov 01 16:33:56.507365 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:56.507219 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0240fa16-cc94-4d43-a3a8-ae27ca53f6ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f4a265-7c6884bdff-d444d\" (UID: \"0240fa16-cc94-4d43-a3a8-ae27ca53f6ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-7c6884bdff-d444d" Nov 01 16:33:56.514926 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:56.514892 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcrbh\" (UniqueName: \"kubernetes.io/projected/c2d00eaa-b184-4176-9944-ef381e8eaec0-kube-api-access-vcrbh\") pod \"payload-processing-e2e-ipp-b-f4a265-dc856b996-7fdck\" (UID: \"c2d00eaa-b184-4176-9944-ef381e8eaec0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f4a265-dc856b996-7fdck" Nov 01 16:33:56.515647 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:56.515624 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7f97\" (UniqueName: \"kubernetes.io/projected/0240fa16-cc94-4d43-a3a8-ae27ca53f6ec-kube-api-access-m7f97\") pod \"payload-pre-processing-e2e-ipp-b-f4a265-7c6884bdff-d444d\" (UID: \"0240fa16-cc94-4d43-a3a8-ae27ca53f6ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-7c6884bdff-d444d" Nov 01 16:33:56.652092 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:56.652062 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-7c6884bdff-d444d" Nov 01 16:33:56.663216 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:56.663174 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4a265-dc856b996-7fdck" Nov 01 16:33:56.952200 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:56.952169 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-7c6884bdff-d444d"] Nov 01 16:33:56.997058 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:56.980309 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f4a265-dc856b996-7fdck"] Nov 01 16:33:57.021065 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:33:57.010303 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2d00eaa_b184_4176_9944_ef381e8eaec0.slice/crio-c9d6f87dbcca3799bb387a7891b03773a5c35565c9abfdf364e51f67bccf3dbd WatchSource:0}: Error finding container c9d6f87dbcca3799bb387a7891b03773a5c35565c9abfdf364e51f67bccf3dbd: Status 404 returned error can't find the container with id c9d6f87dbcca3799bb387a7891b03773a5c35565c9abfdf364e51f67bccf3dbd Nov 01 16:33:57.031252 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:57.030155 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" event={"ID":"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519","Type":"ContainerStarted","Data":"500a1c7400013ba1a920f48ce08ce87680262ace9435db82d30a8ea72c34cd9e"} Nov 01 16:33:57.035058 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:57.035008 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-7c6884bdff-d444d" event={"ID":"0240fa16-cc94-4d43-a3a8-ae27ca53f6ec","Type":"ContainerStarted","Data":"631153c23440b0fa70fe871c15cad7f379dac01cd5ebdb6d7369249857bf50fb"} Nov 01 16:33:57.834517 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:57.834476 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f01de5-65884f55d5-pbgx8" podUID="387b6adf-9404-414e-ae75-0b48e938e77a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:33:57.971764 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:57.971731 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" Nov 01 16:33:57.972710 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:57.972689 4363 patch_prober.go:29] interesting pod/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body= Nov 01 16:33:57.972787 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:57.972735 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" podUID="87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" Nov 01 16:33:58.044041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:58.043987 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4a265-dc856b996-7fdck" event={"ID":"c2d00eaa-b184-4176-9944-ef381e8eaec0","Type":"ContainerStarted","Data":"81e22bea5dfcc4b2c55cd463a1de1c16f1886853535f5b50e142c3715c80573a"} Nov 01 16:33:58.044459 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:58.044053 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4a265-dc856b996-7fdck" event={"ID":"c2d00eaa-b184-4176-9944-ef381e8eaec0","Type":"ContainerStarted","Data":"c9d6f87dbcca3799bb387a7891b03773a5c35565c9abfdf364e51f67bccf3dbd"} Nov 01 16:33:58.044459 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:58.044082 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4a265-dc856b996-7fdck" Nov 01 16:33:58.045670 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:58.045646 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-7c6884bdff-d444d" event={"ID":"0240fa16-cc94-4d43-a3a8-ae27ca53f6ec","Type":"ContainerStarted","Data":"bfb866b45de778ea21533405137525e43e102d4c78d6ad5913b4b87e936bfa8f"} Nov 01 16:33:58.045846 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:58.045828 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-7c6884bdff-d444d" Nov 01 16:33:58.061326 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:58.061287 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4a265-dc856b996-7fdck" podStartSLOduration=2.061275064 podStartE2EDuration="2.061275064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:33:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:33:58.060087188 +0000 UTC m=+2771.391824838" watchObservedRunningTime="2026-11-01 16:33:58.061275064 +0000 UTC m=+2771.393012711" Nov 01 16:33:58.076114 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:58.076065 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-7c6884bdff-d444d" podStartSLOduration=2.076048634 podStartE2EDuration="2.076048634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:33:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:33:58.074707695 +0000 UTC m=+2771.406445346" watchObservedRunningTime="2026-11-01 16:33:58.076048634 +0000 UTC m=+2771.407786285" Nov 01 16:33:58.094100 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:58.090346 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-dfff59-688dc9df89-4vk2r" podUID="247d3008-5891-46e2-843b-da07c4329f01" containerName="payload-processing" containerID="cri-o://396c50fadb78b5123057932d296478c829edabd3df4fbb67afcf7e0a79780955" gracePeriod=25 Nov 01 16:33:58.332205 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:58.328308 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f01de5-65884f55d5-pbgx8" Nov 01 16:33:58.455947 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:58.455851 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/387b6adf-9404-414e-ae75-0b48e938e77a-kube-api-access-zpjwn\" (UniqueName: \"kubernetes.io/projected/387b6adf-9404-414e-ae75-0b48e938e77a-kube-api-access-zpjwn\") pod \"387b6adf-9404-414e-ae75-0b48e938e77a\" (UID: \"387b6adf-9404-414e-ae75-0b48e938e77a\") " Nov 01 16:33:58.456157 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:58.456136 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/387b6adf-9404-414e-ae75-0b48e938e77a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/387b6adf-9404-414e-ae75-0b48e938e77a-plugins-config-volume\") pod \"387b6adf-9404-414e-ae75-0b48e938e77a\" (UID: \"387b6adf-9404-414e-ae75-0b48e938e77a\") " Nov 01 16:33:58.456567 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:58.456543 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/387b6adf-9404-414e-ae75-0b48e938e77a-plugins-config-volume" pod "387b6adf-9404-414e-ae75-0b48e938e77a" (UID: "387b6adf-9404-414e-ae75-0b48e938e77a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:58.458180 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:58.458159 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/387b6adf-9404-414e-ae75-0b48e938e77a-kube-api-access-zpjwn" pod "387b6adf-9404-414e-ae75-0b48e938e77a" (UID: "387b6adf-9404-414e-ae75-0b48e938e77a"). InnerVolumeSpecName "kube-api-access-zpjwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:58.557969 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:58.557928 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/387b6adf-9404-414e-ae75-0b48e938e77a-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:58.557969 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:58.557961 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpjwn\" (UniqueName: \"kubernetes.io/projected/387b6adf-9404-414e-ae75-0b48e938e77a-kube-api-access-zpjwn\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:58.672835 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:58.672743 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" Nov 01 16:33:58.911042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:58.902513 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4a265-5db88cf775-7jnq2" podUID="10f4b6b1-4845-4f52-b90c-7650dfdb27b7" containerName="payload-processing" containerID="cri-o://837faf8431a56e7b78d770e6a9f65b0d00f62cf76874b350f51badd91f0d837c" gracePeriod=25 Nov 01 16:33:58.971672 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:58.971641 4363 patch_prober.go:29] interesting pod/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body= Nov 01 16:33:58.971812 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:58.971696 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" podUID="87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" Nov 01 16:33:59.059306 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:59.059270 4363 generic.go:356] "Generic (PLEG): container finished" podID="10f4b6b1-4845-4f52-b90c-7650dfdb27b7" containerID="837faf8431a56e7b78d770e6a9f65b0d00f62cf76874b350f51badd91f0d837c" exitCode=0 Nov 01 16:33:59.059752 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:59.059413 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4a265-5db88cf775-7jnq2" event={"ID":"10f4b6b1-4845-4f52-b90c-7650dfdb27b7","Type":"ContainerDied","Data":"837faf8431a56e7b78d770e6a9f65b0d00f62cf76874b350f51badd91f0d837c"} Nov 01 16:33:59.061984 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:59.061948 4363 generic.go:356] "Generic (PLEG): container finished" podID="387b6adf-9404-414e-ae75-0b48e938e77a" containerID="7e12c927caea0f8d330caf6d5d8dbbf8e12e49f79150bd84d09240a2e5b52fbb" exitCode=0 Nov 01 16:33:59.062393 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:59.062362 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f01de5-65884f55d5-pbgx8" event={"ID":"387b6adf-9404-414e-ae75-0b48e938e77a","Type":"ContainerDied","Data":"7e12c927caea0f8d330caf6d5d8dbbf8e12e49f79150bd84d09240a2e5b52fbb"} Nov 01 16:33:59.062541 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:59.062435 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f01de5-65884f55d5-pbgx8" event={"ID":"387b6adf-9404-414e-ae75-0b48e938e77a","Type":"ContainerDied","Data":"869a02c2ab39ce8338032fdc5ba7ea3b0db3b4e17745f931c603f04bd9b1bada"} Nov 01 16:33:59.062541 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:59.062501 4363 scope.go:122] "RemoveContainer" containerID="7e12c927caea0f8d330caf6d5d8dbbf8e12e49f79150bd84d09240a2e5b52fbb" Nov 01 16:33:59.063990 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:59.063963 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f01de5-65884f55d5-pbgx8" Nov 01 16:33:59.074135 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:59.074114 4363 scope.go:122] "RemoveContainer" containerID="7e12c927caea0f8d330caf6d5d8dbbf8e12e49f79150bd84d09240a2e5b52fbb" Nov 01 16:33:59.074417 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:33:59.074389 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e12c927caea0f8d330caf6d5d8dbbf8e12e49f79150bd84d09240a2e5b52fbb\": container with ID starting with 7e12c927caea0f8d330caf6d5d8dbbf8e12e49f79150bd84d09240a2e5b52fbb not found: ID does not exist" containerID="7e12c927caea0f8d330caf6d5d8dbbf8e12e49f79150bd84d09240a2e5b52fbb" Nov 01 16:33:59.074511 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:59.074427 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e12c927caea0f8d330caf6d5d8dbbf8e12e49f79150bd84d09240a2e5b52fbb"} err="failed to get container status \"7e12c927caea0f8d330caf6d5d8dbbf8e12e49f79150bd84d09240a2e5b52fbb\": rpc error: code = NotFound desc = could not find container \"7e12c927caea0f8d330caf6d5d8dbbf8e12e49f79150bd84d09240a2e5b52fbb\": container with ID starting with 7e12c927caea0f8d330caf6d5d8dbbf8e12e49f79150bd84d09240a2e5b52fbb not found: ID does not exist" Nov 01 16:33:59.114025 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:59.113972 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-f01de5-65884f55d5-pbgx8"] Nov 01 16:33:59.116520 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:59.116491 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-f01de5-65884f55d5-pbgx8"] Nov 01 16:33:59.322801 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:59.322773 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4a265-5db88cf775-7jnq2" Nov 01 16:33:59.367654 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:59.367618 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10f4b6b1-4845-4f52-b90c-7650dfdb27b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10f4b6b1-4845-4f52-b90c-7650dfdb27b7-plugins-config-volume\") pod \"10f4b6b1-4845-4f52-b90c-7650dfdb27b7\" (UID: \"10f4b6b1-4845-4f52-b90c-7650dfdb27b7\") " Nov 01 16:33:59.367807 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:59.367773 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10f4b6b1-4845-4f52-b90c-7650dfdb27b7-kube-api-access-n6nw5\" (UniqueName: \"kubernetes.io/projected/10f4b6b1-4845-4f52-b90c-7650dfdb27b7-kube-api-access-n6nw5\") pod \"10f4b6b1-4845-4f52-b90c-7650dfdb27b7\" (UID: \"10f4b6b1-4845-4f52-b90c-7650dfdb27b7\") " Nov 01 16:33:59.368005 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:59.367980 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10f4b6b1-4845-4f52-b90c-7650dfdb27b7-plugins-config-volume" pod "10f4b6b1-4845-4f52-b90c-7650dfdb27b7" (UID: "10f4b6b1-4845-4f52-b90c-7650dfdb27b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:59.368342 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:59.368317 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10f4b6b1-4845-4f52-b90c-7650dfdb27b7-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:59.369876 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:59.369855 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10f4b6b1-4845-4f52-b90c-7650dfdb27b7-kube-api-access-n6nw5" pod "10f4b6b1-4845-4f52-b90c-7650dfdb27b7" (UID: "10f4b6b1-4845-4f52-b90c-7650dfdb27b7"). InnerVolumeSpecName "kube-api-access-n6nw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:59.469248 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:59.469216 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6nw5\" (UniqueName: \"kubernetes.io/projected/10f4b6b1-4845-4f52-b90c-7650dfdb27b7-kube-api-access-n6nw5\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:59.972268 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:59.972231 4363 patch_prober.go:29] interesting pod/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body= Nov 01 16:33:59.972462 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:33:59.972294 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" podUID="87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" Nov 01 16:34:00.070420 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:00.070385 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4a265-5db88cf775-7jnq2" Nov 01 16:34:00.070420 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:00.070403 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4a265-5db88cf775-7jnq2" event={"ID":"10f4b6b1-4845-4f52-b90c-7650dfdb27b7","Type":"ContainerDied","Data":"15d59c997e2bd7dba02169eab843024bf10dbed308f7cbcc50244edba6bbad0a"} Nov 01 16:34:00.070933 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:00.070490 4363 scope.go:122] "RemoveContainer" containerID="837faf8431a56e7b78d770e6a9f65b0d00f62cf76874b350f51badd91f0d837c" Nov 01 16:34:00.118999 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:00.118970 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f4a265-5db88cf775-7jnq2"] Nov 01 16:34:00.124225 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:00.124200 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f4a265-5db88cf775-7jnq2"] Nov 01 16:34:00.808042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:00.807089 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10f4b6b1-4845-4f52-b90c-7650dfdb27b7" path="/var/lib/kubelet/pods/10f4b6b1-4845-4f52-b90c-7650dfdb27b7/volumes" Nov 01 16:34:00.808042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:00.807500 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="387b6adf-9404-414e-ae75-0b48e938e77a" path="/var/lib/kubelet/pods/387b6adf-9404-414e-ae75-0b48e938e77a/volumes" Nov 01 16:34:00.971694 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:00.971659 4363 patch_prober.go:29] interesting pod/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body= Nov 01 16:34:00.971909 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:00.971716 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" podUID="87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" Nov 01 16:34:01.971767 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:01.971726 4363 patch_prober.go:29] interesting pod/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body= Nov 01 16:34:01.972217 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:01.971791 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" podUID="87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" Nov 01 16:34:02.976127 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:02.976099 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" Nov 01 16:34:02.977378 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:02.977357 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" Nov 01 16:34:03.549395 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:03.549346 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-dfff59-688dc9df89-4vk2r" podUID="247d3008-5891-46e2-843b-da07c4329f01" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:34:06.660255 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:06.660216 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" containerID="cri-o://8ba72283f27f9372064628a2ecedd172c117b3952b1ffc4144a9a6c2fa1b32ce" gracePeriod=45 Nov 01 16:34:06.677481 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:06.677432 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" containerID="cri-o://80fad3514371311b623cb629207f77dbfa1c75ae0e3450e815d375c94a31b156" gracePeriod=45 Nov 01 16:34:06.955445 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:06.955418 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 16:34:07.001412 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.001387 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 16:34:07.095344 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.095317 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6476aee-e7a4-497f-84f0-ef0d63ede742-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6476aee-e7a4-497f-84f0-ef0d63ede742-model-cache\") pod \"a6476aee-e7a4-497f-84f0-ef0d63ede742\" (UID: \"a6476aee-e7a4-497f-84f0-ef0d63ede742\") " Nov 01 16:34:07.095515 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.095357 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6476aee-e7a4-497f-84f0-ef0d63ede742-home\" (UniqueName: \"kubernetes.io/empty-dir/a6476aee-e7a4-497f-84f0-ef0d63ede742-home\") pod \"a6476aee-e7a4-497f-84f0-ef0d63ede742\" (UID: \"a6476aee-e7a4-497f-84f0-ef0d63ede742\") " Nov 01 16:34:07.095515 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.095409 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8261b28f-6536-4ba3-88f6-0713370eb6d3-kube-api-access-f782x\" (UniqueName: \"kubernetes.io/projected/8261b28f-6536-4ba3-88f6-0713370eb6d3-kube-api-access-f782x\") pod \"8261b28f-6536-4ba3-88f6-0713370eb6d3\" (UID: \"8261b28f-6536-4ba3-88f6-0713370eb6d3\") " Nov 01 16:34:07.095515 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.095447 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6476aee-e7a4-497f-84f0-ef0d63ede742-kube-api-access-v97s9\" (UniqueName: \"kubernetes.io/projected/a6476aee-e7a4-497f-84f0-ef0d63ede742-kube-api-access-v97s9\") pod \"a6476aee-e7a4-497f-84f0-ef0d63ede742\" (UID: \"a6476aee-e7a4-497f-84f0-ef0d63ede742\") " Nov 01 16:34:07.095515 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.095478 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8261b28f-6536-4ba3-88f6-0713370eb6d3-home\" (UniqueName: \"kubernetes.io/empty-dir/8261b28f-6536-4ba3-88f6-0713370eb6d3-home\") pod \"8261b28f-6536-4ba3-88f6-0713370eb6d3\" (UID: \"8261b28f-6536-4ba3-88f6-0713370eb6d3\") " Nov 01 16:34:07.095736 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.095528 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6476aee-e7a4-497f-84f0-ef0d63ede742-tls-certs\" (UniqueName: \"kubernetes.io/secret/a6476aee-e7a4-497f-84f0-ef0d63ede742-tls-certs\") pod \"a6476aee-e7a4-497f-84f0-ef0d63ede742\" (UID: \"a6476aee-e7a4-497f-84f0-ef0d63ede742\") " Nov 01 16:34:07.095736 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.095568 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8261b28f-6536-4ba3-88f6-0713370eb6d3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8261b28f-6536-4ba3-88f6-0713370eb6d3-model-cache\") pod \"8261b28f-6536-4ba3-88f6-0713370eb6d3\" (UID: \"8261b28f-6536-4ba3-88f6-0713370eb6d3\") " Nov 01 16:34:07.095736 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.095621 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6476aee-e7a4-497f-84f0-ef0d63ede742-model-cache" pod "a6476aee-e7a4-497f-84f0-ef0d63ede742" (UID: "a6476aee-e7a4-497f-84f0-ef0d63ede742"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:07.095736 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.095617 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6476aee-e7a4-497f-84f0-ef0d63ede742-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6476aee-e7a4-497f-84f0-ef0d63ede742-dshm\") pod \"a6476aee-e7a4-497f-84f0-ef0d63ede742\" (UID: \"a6476aee-e7a4-497f-84f0-ef0d63ede742\") " Nov 01 16:34:07.095736 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.095712 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6476aee-e7a4-497f-84f0-ef0d63ede742-home" pod "a6476aee-e7a4-497f-84f0-ef0d63ede742" (UID: "a6476aee-e7a4-497f-84f0-ef0d63ede742"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:07.096183 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.095780 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8261b28f-6536-4ba3-88f6-0713370eb6d3-home" pod "8261b28f-6536-4ba3-88f6-0713370eb6d3" (UID: "8261b28f-6536-4ba3-88f6-0713370eb6d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:07.096183 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.095814 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6476aee-e7a4-497f-84f0-ef0d63ede742-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6476aee-e7a4-497f-84f0-ef0d63ede742-tmp-dir\") pod \"a6476aee-e7a4-497f-84f0-ef0d63ede742\" (UID: \"a6476aee-e7a4-497f-84f0-ef0d63ede742\") " Nov 01 16:34:07.096183 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.095853 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8261b28f-6536-4ba3-88f6-0713370eb6d3-model-cache" pod "8261b28f-6536-4ba3-88f6-0713370eb6d3" (UID: "8261b28f-6536-4ba3-88f6-0713370eb6d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:07.096183 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.095953 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8261b28f-6536-4ba3-88f6-0713370eb6d3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8261b28f-6536-4ba3-88f6-0713370eb6d3-dshm\") pod \"8261b28f-6536-4ba3-88f6-0713370eb6d3\" (UID: \"8261b28f-6536-4ba3-88f6-0713370eb6d3\") " Nov 01 16:34:07.096183 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.095986 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6476aee-e7a4-497f-84f0-ef0d63ede742-tmp-dir" pod "a6476aee-e7a4-497f-84f0-ef0d63ede742" (UID: "a6476aee-e7a4-497f-84f0-ef0d63ede742"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:07.096183 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.096065 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8261b28f-6536-4ba3-88f6-0713370eb6d3-tls-certs\" (UniqueName: \"kubernetes.io/secret/8261b28f-6536-4ba3-88f6-0713370eb6d3-tls-certs\") pod \"8261b28f-6536-4ba3-88f6-0713370eb6d3\" (UID: \"8261b28f-6536-4ba3-88f6-0713370eb6d3\") " Nov 01 16:34:07.096183 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.096132 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8261b28f-6536-4ba3-88f6-0713370eb6d3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8261b28f-6536-4ba3-88f6-0713370eb6d3-tmp-dir\") pod \"8261b28f-6536-4ba3-88f6-0713370eb6d3\" (UID: \"8261b28f-6536-4ba3-88f6-0713370eb6d3\") " Nov 01 16:34:07.097178 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.096622 4363 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6476aee-e7a4-497f-84f0-ef0d63ede742-model-cache\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:07.097178 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.096659 4363 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6476aee-e7a4-497f-84f0-ef0d63ede742-home\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:07.097178 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.096674 4363 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8261b28f-6536-4ba3-88f6-0713370eb6d3-home\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:07.097178 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.096687 4363 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8261b28f-6536-4ba3-88f6-0713370eb6d3-model-cache\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:07.097178 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.096702 4363 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6476aee-e7a4-497f-84f0-ef0d63ede742-tmp-dir\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:07.097178 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.096866 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8261b28f-6536-4ba3-88f6-0713370eb6d3-tmp-dir" pod "8261b28f-6536-4ba3-88f6-0713370eb6d3" (UID: "8261b28f-6536-4ba3-88f6-0713370eb6d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:07.098813 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.098792 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8261b28f-6536-4ba3-88f6-0713370eb6d3-dshm" pod "8261b28f-6536-4ba3-88f6-0713370eb6d3" (UID: "8261b28f-6536-4ba3-88f6-0713370eb6d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:07.098813 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.098804 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8261b28f-6536-4ba3-88f6-0713370eb6d3-kube-api-access-f782x" pod "8261b28f-6536-4ba3-88f6-0713370eb6d3" (UID: "8261b28f-6536-4ba3-88f6-0713370eb6d3"). InnerVolumeSpecName "kube-api-access-f782x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:34:07.098964 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.098885 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6476aee-e7a4-497f-84f0-ef0d63ede742-tls-certs" pod "a6476aee-e7a4-497f-84f0-ef0d63ede742" (UID: "a6476aee-e7a4-497f-84f0-ef0d63ede742"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:34:07.098964 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.098920 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6476aee-e7a4-497f-84f0-ef0d63ede742-dshm" pod "a6476aee-e7a4-497f-84f0-ef0d63ede742" (UID: "a6476aee-e7a4-497f-84f0-ef0d63ede742"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:07.099374 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.099357 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8261b28f-6536-4ba3-88f6-0713370eb6d3-tls-certs" pod "8261b28f-6536-4ba3-88f6-0713370eb6d3" (UID: "8261b28f-6536-4ba3-88f6-0713370eb6d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:34:07.099772 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.099750 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6476aee-e7a4-497f-84f0-ef0d63ede742-kube-api-access-v97s9" pod "a6476aee-e7a4-497f-84f0-ef0d63ede742" (UID: "a6476aee-e7a4-497f-84f0-ef0d63ede742"). InnerVolumeSpecName "kube-api-access-v97s9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:34:07.146334 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.146306 4363 generic.go:356] "Generic (PLEG): container finished" podID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerID="8ba72283f27f9372064628a2ecedd172c117b3952b1ffc4144a9a6c2fa1b32ce" exitCode=0 Nov 01 16:34:07.146508 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.146356 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" Nov 01 16:34:07.146508 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.146396 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" event={"ID":"a6476aee-e7a4-497f-84f0-ef0d63ede742","Type":"ContainerDied","Data":"8ba72283f27f9372064628a2ecedd172c117b3952b1ffc4144a9a6c2fa1b32ce"} Nov 01 16:34:07.146508 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.146444 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr" event={"ID":"a6476aee-e7a4-497f-84f0-ef0d63ede742","Type":"ContainerDied","Data":"350ce28ca5f3e8a2d56c7117ef5725810a9a9379f14c93a453f43d8dbff2398b"} Nov 01 16:34:07.146508 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.146467 4363 scope.go:122] "RemoveContainer" containerID="8ba72283f27f9372064628a2ecedd172c117b3952b1ffc4144a9a6c2fa1b32ce" Nov 01 16:34:07.147931 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.147911 4363 generic.go:356] "Generic (PLEG): container finished" podID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerID="80fad3514371311b623cb629207f77dbfa1c75ae0e3450e815d375c94a31b156" exitCode=0 Nov 01 16:34:07.148061 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.147942 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" event={"ID":"8261b28f-6536-4ba3-88f6-0713370eb6d3","Type":"ContainerDied","Data":"80fad3514371311b623cb629207f77dbfa1c75ae0e3450e815d375c94a31b156"} Nov 01 16:34:07.148061 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.147962 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" Nov 01 16:34:07.148061 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.147981 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6" event={"ID":"8261b28f-6536-4ba3-88f6-0713370eb6d3","Type":"ContainerDied","Data":"4d67939ffbb7519b4b95b9232190a3b0704277350ee140ffc0be824eed6671bd"} Nov 01 16:34:07.157912 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.157892 4363 scope.go:122] "RemoveContainer" containerID="8ba72283f27f9372064628a2ecedd172c117b3952b1ffc4144a9a6c2fa1b32ce" Nov 01 16:34:07.158238 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:34:07.158220 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ba72283f27f9372064628a2ecedd172c117b3952b1ffc4144a9a6c2fa1b32ce\": container with ID starting with 8ba72283f27f9372064628a2ecedd172c117b3952b1ffc4144a9a6c2fa1b32ce not found: ID does not exist" containerID="8ba72283f27f9372064628a2ecedd172c117b3952b1ffc4144a9a6c2fa1b32ce" Nov 01 16:34:07.158332 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.158244 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ba72283f27f9372064628a2ecedd172c117b3952b1ffc4144a9a6c2fa1b32ce"} err="failed to get container status \"8ba72283f27f9372064628a2ecedd172c117b3952b1ffc4144a9a6c2fa1b32ce\": rpc error: code = NotFound desc = could not find container \"8ba72283f27f9372064628a2ecedd172c117b3952b1ffc4144a9a6c2fa1b32ce\": container with ID starting with 8ba72283f27f9372064628a2ecedd172c117b3952b1ffc4144a9a6c2fa1b32ce not found: ID does not exist" Nov 01 16:34:07.158332 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.158260 4363 scope.go:122] "RemoveContainer" containerID="80fad3514371311b623cb629207f77dbfa1c75ae0e3450e815d375c94a31b156" Nov 01 16:34:07.167078 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.167057 4363 scope.go:122] "RemoveContainer" containerID="80fad3514371311b623cb629207f77dbfa1c75ae0e3450e815d375c94a31b156" Nov 01 16:34:07.167341 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:34:07.167319 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80fad3514371311b623cb629207f77dbfa1c75ae0e3450e815d375c94a31b156\": container with ID starting with 80fad3514371311b623cb629207f77dbfa1c75ae0e3450e815d375c94a31b156 not found: ID does not exist" containerID="80fad3514371311b623cb629207f77dbfa1c75ae0e3450e815d375c94a31b156" Nov 01 16:34:07.167391 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.167347 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80fad3514371311b623cb629207f77dbfa1c75ae0e3450e815d375c94a31b156"} err="failed to get container status \"80fad3514371311b623cb629207f77dbfa1c75ae0e3450e815d375c94a31b156\": rpc error: code = NotFound desc = could not find container \"80fad3514371311b623cb629207f77dbfa1c75ae0e3450e815d375c94a31b156\": container with ID starting with 80fad3514371311b623cb629207f77dbfa1c75ae0e3450e815d375c94a31b156 not found: ID does not exist" Nov 01 16:34:07.172705 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.172684 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr"] Nov 01 16:34:07.177955 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.177933 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-9f4dc9/premium-w4-9f4dc9-kserve-7fd4db9b8c-746rr"] Nov 01 16:34:07.206269 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.206189 4363 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8261b28f-6536-4ba3-88f6-0713370eb6d3-dshm\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:07.206486 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.206468 4363 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8261b28f-6536-4ba3-88f6-0713370eb6d3-tls-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:07.206591 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.206573 4363 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8261b28f-6536-4ba3-88f6-0713370eb6d3-tmp-dir\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:07.206685 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.206596 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f782x\" (UniqueName: \"kubernetes.io/projected/8261b28f-6536-4ba3-88f6-0713370eb6d3-kube-api-access-f782x\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:07.206685 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.206612 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v97s9\" (UniqueName: \"kubernetes.io/projected/a6476aee-e7a4-497f-84f0-ef0d63ede742-kube-api-access-v97s9\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:07.206685 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.206628 4363 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6476aee-e7a4-497f-84f0-ef0d63ede742-tls-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:07.206685 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.206644 4363 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6476aee-e7a4-497f-84f0-ef0d63ede742-dshm\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:07.216210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.216168 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6"] Nov 01 16:34:07.217650 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:07.217615 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-9f4dc9/simulator-w4-9f4dc9-kserve-6d4c6c598b-k8mc6"] Nov 01 16:34:08.797477 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:08.797444 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" path="/var/lib/kubelet/pods/8261b28f-6536-4ba3-88f6-0713370eb6d3/volumes" Nov 01 16:34:08.797928 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:08.797914 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" path="/var/lib/kubelet/pods/a6476aee-e7a4-497f-84f0-ef0d63ede742/volumes" Nov 01 16:34:09.065242 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:09.065157 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-7c6884bdff-d444d" Nov 01 16:34:09.065242 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:09.065231 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4a265-dc856b996-7fdck" Nov 01 16:34:12.894664 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:12.894628 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-l5qnm_dae26a45-6a59-4c32-a565-704e50c5f271/payload-processing/0.log" Nov 01 16:34:13.073651 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:13.073618 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-l5qnm_dae26a45-6a59-4c32-a565-704e50c5f271/payload-processing/0.log" Nov 01 16:34:13.260502 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:13.260414 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-f4a265-dc856b996-7fdck_c2d00eaa-b184-4176-9944-ef381e8eaec0/payload-processing/0.log" Nov 01 16:34:13.550044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:13.549935 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-dfff59-688dc9df89-4vk2r" podUID="247d3008-5891-46e2-843b-da07c4329f01" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:34:18.231150 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:18.231115 4363 generic.go:356] "Generic (PLEG): container finished" podID="247d3008-5891-46e2-843b-da07c4329f01" containerID="396c50fadb78b5123057932d296478c829edabd3df4fbb67afcf7e0a79780955" exitCode=0 Nov 01 16:34:18.231540 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:18.231176 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-dfff59-688dc9df89-4vk2r" event={"ID":"247d3008-5891-46e2-843b-da07c4329f01","Type":"ContainerDied","Data":"396c50fadb78b5123057932d296478c829edabd3df4fbb67afcf7e0a79780955"} Nov 01 16:34:18.270337 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:18.270314 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-dfff59-688dc9df89-4vk2r" Nov 01 16:34:18.320036 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:34:18.319516 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:34:18.429096 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:18.424395 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/247d3008-5891-46e2-843b-da07c4329f01-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/247d3008-5891-46e2-843b-da07c4329f01-plugins-config-volume\") pod \"247d3008-5891-46e2-843b-da07c4329f01\" (UID: \"247d3008-5891-46e2-843b-da07c4329f01\") " Nov 01 16:34:18.429096 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:18.424593 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/247d3008-5891-46e2-843b-da07c4329f01-kube-api-access-tx858\" (UniqueName: \"kubernetes.io/projected/247d3008-5891-46e2-843b-da07c4329f01-kube-api-access-tx858\") pod \"247d3008-5891-46e2-843b-da07c4329f01\" (UID: \"247d3008-5891-46e2-843b-da07c4329f01\") " Nov 01 16:34:18.439200 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:18.438516 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/247d3008-5891-46e2-843b-da07c4329f01-plugins-config-volume" pod "247d3008-5891-46e2-843b-da07c4329f01" (UID: "247d3008-5891-46e2-843b-da07c4329f01"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:34:18.440979 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:18.440940 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/247d3008-5891-46e2-843b-da07c4329f01-kube-api-access-tx858" pod "247d3008-5891-46e2-843b-da07c4329f01" (UID: "247d3008-5891-46e2-843b-da07c4329f01"). InnerVolumeSpecName "kube-api-access-tx858". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:34:18.526316 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:18.526280 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx858\" (UniqueName: \"kubernetes.io/projected/247d3008-5891-46e2-843b-da07c4329f01-kube-api-access-tx858\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:18.526316 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:18.526312 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/247d3008-5891-46e2-843b-da07c4329f01-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:19.190762 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:19.190650 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv"] Nov 01 16:34:19.191320 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:19.191289 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" podUID="6daeee34-6958-40f5-a360-9a58b3bd51d8" containerName="istio-proxy" containerID="cri-o://d7ae03be4b5c7e4ffa4ab1db04b0f8bd4db603819eeb69eaa27a23ada687c16d" gracePeriod=30 Nov 01 16:34:19.240852 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:19.240814 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-dfff59-688dc9df89-4vk2r" event={"ID":"247d3008-5891-46e2-843b-da07c4329f01","Type":"ContainerDied","Data":"f55db40440ffe4a416df9ec52b31580f8c5ae44abe14ed76dfefe549c5c31b7a"} Nov 01 16:34:19.241371 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:19.240848 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-dfff59-688dc9df89-4vk2r" Nov 01 16:34:19.241371 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:19.240877 4363 scope.go:122] "RemoveContainer" containerID="396c50fadb78b5123057932d296478c829edabd3df4fbb67afcf7e0a79780955" Nov 01 16:34:19.259053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:19.259004 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-dfff59-688dc9df89-4vk2r"] Nov 01 16:34:19.263439 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:19.263410 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-dfff59-688dc9df89-4vk2r"] Nov 01 16:34:20.807308 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:20.807270 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="247d3008-5891-46e2-843b-da07c4329f01" path="/var/lib/kubelet/pods/247d3008-5891-46e2-843b-da07c4329f01/volumes" Nov 01 16:34:21.555666 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:21.555623 4363 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:34:21.555879 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:21.555683 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" podUID="6daeee34-6958-40f5-a360-9a58b3bd51d8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:34:24.096067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.094099 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-501d63-56568c796c-8lg27"] Nov 01 16:34:24.096067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.094342 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-501d63-56568c796c-8lg27" podUID="159aed21-0910-4554-a841-b7c096765924" containerName="maas-api" containerID="cri-o://4799dce9509cf8342931fe38b89a5500d33bfe13610b4b8eba2a72dc08d1e3d1" gracePeriod=30 Nov 01 16:34:24.172582 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.172549 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-501d63-5fd9cc8bfb-krlt9"] Nov 01 16:34:24.172787 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.172768 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-501d63-5fd9cc8bfb-krlt9" podUID="cb31d5ec-ff4f-4fbf-b78f-ab022f851e78" containerName="payload-processing" containerID="cri-o://b267c720a2cffea3538cef5a39e37536ff1eb682d81d6e630e423362a02ec8bf" gracePeriod=30 Nov 01 16:34:24.196948 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.194255 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-678bbff655-z866s"] Nov 01 16:34:24.196948 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.194492 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-678bbff655-z866s" podUID="4324048b-b48c-452d-91df-8e48f5e8e97a" containerName="payload-pre-processing" containerID="cri-o://8ec4cf5de10cf531018ee05f14532add7ad643104f09334d57e8800d305c34ac" gracePeriod=30 Nov 01 16:34:24.273460 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.273434 4363 generic.go:356] "Generic (PLEG): container finished" podID="159aed21-0910-4554-a841-b7c096765924" containerID="4799dce9509cf8342931fe38b89a5500d33bfe13610b4b8eba2a72dc08d1e3d1" exitCode=0 Nov 01 16:34:24.273598 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.273481 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-501d63-56568c796c-8lg27" event={"ID":"159aed21-0910-4554-a841-b7c096765924","Type":"ContainerDied","Data":"4799dce9509cf8342931fe38b89a5500d33bfe13610b4b8eba2a72dc08d1e3d1"} Nov 01 16:34:24.338056 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.337715 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-501d63-5fd9cc8bfb-krlt9" podUID="cb31d5ec-ff4f-4fbf-b78f-ab022f851e78" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.1.33:9004: connect: connection refused" Nov 01 16:34:24.338056 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.337724 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-678bbff655-z866s" podUID="4324048b-b48c-452d-91df-8e48f5e8e97a" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.1.32:9004: connect: connection refused" Nov 01 16:34:24.523997 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.523707 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-501d63-56568c796c-8lg27" Nov 01 16:34:24.606130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.605432 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-678bbff655-z866s" Nov 01 16:34:24.613969 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.612717 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/159aed21-0910-4554-a841-b7c096765924-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/159aed21-0910-4554-a841-b7c096765924-openshift-service-ca\") pod \"159aed21-0910-4554-a841-b7c096765924\" (UID: \"159aed21-0910-4554-a841-b7c096765924\") " Nov 01 16:34:24.613969 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.612769 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4324048b-b48c-452d-91df-8e48f5e8e97a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4324048b-b48c-452d-91df-8e48f5e8e97a-plugins-config-volume\") pod \"4324048b-b48c-452d-91df-8e48f5e8e97a\" (UID: \"4324048b-b48c-452d-91df-8e48f5e8e97a\") " Nov 01 16:34:24.613969 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.612836 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4324048b-b48c-452d-91df-8e48f5e8e97a-kube-api-access-pwglg\" (UniqueName: \"kubernetes.io/projected/4324048b-b48c-452d-91df-8e48f5e8e97a-kube-api-access-pwglg\") pod \"4324048b-b48c-452d-91df-8e48f5e8e97a\" (UID: \"4324048b-b48c-452d-91df-8e48f5e8e97a\") " Nov 01 16:34:24.613969 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.612909 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/159aed21-0910-4554-a841-b7c096765924-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/159aed21-0910-4554-a841-b7c096765924-maas-api-tls\") pod \"159aed21-0910-4554-a841-b7c096765924\" (UID: \"159aed21-0910-4554-a841-b7c096765924\") " Nov 01 16:34:24.613969 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.613000 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/159aed21-0910-4554-a841-b7c096765924-kube-api-access-29qvs\" (UniqueName: \"kubernetes.io/projected/159aed21-0910-4554-a841-b7c096765924-kube-api-access-29qvs\") pod \"159aed21-0910-4554-a841-b7c096765924\" (UID: \"159aed21-0910-4554-a841-b7c096765924\") " Nov 01 16:34:24.613969 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.613662 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4324048b-b48c-452d-91df-8e48f5e8e97a-plugins-config-volume" pod "4324048b-b48c-452d-91df-8e48f5e8e97a" (UID: "4324048b-b48c-452d-91df-8e48f5e8e97a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:34:24.614467 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.613985 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/159aed21-0910-4554-a841-b7c096765924-metrics-certs\" (UniqueName: \"kubernetes.io/secret/159aed21-0910-4554-a841-b7c096765924-metrics-certs\") pod \"159aed21-0910-4554-a841-b7c096765924\" (UID: \"159aed21-0910-4554-a841-b7c096765924\") " Nov 01 16:34:24.614467 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.614404 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4324048b-b48c-452d-91df-8e48f5e8e97a-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:24.617861 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.614876 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/159aed21-0910-4554-a841-b7c096765924-openshift-service-ca" pod "159aed21-0910-4554-a841-b7c096765924" (UID: "159aed21-0910-4554-a841-b7c096765924"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:34:24.636075 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.631609 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/159aed21-0910-4554-a841-b7c096765924-maas-api-tls" pod "159aed21-0910-4554-a841-b7c096765924" (UID: "159aed21-0910-4554-a841-b7c096765924"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:34:24.645510 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.645286 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/159aed21-0910-4554-a841-b7c096765924-metrics-certs" pod "159aed21-0910-4554-a841-b7c096765924" (UID: "159aed21-0910-4554-a841-b7c096765924"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:34:24.646774 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.646721 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4324048b-b48c-452d-91df-8e48f5e8e97a-kube-api-access-pwglg" pod "4324048b-b48c-452d-91df-8e48f5e8e97a" (UID: "4324048b-b48c-452d-91df-8e48f5e8e97a"). InnerVolumeSpecName "kube-api-access-pwglg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:34:24.646774 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.646751 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/159aed21-0910-4554-a841-b7c096765924-kube-api-access-29qvs" pod "159aed21-0910-4554-a841-b7c096765924" (UID: "159aed21-0910-4554-a841-b7c096765924"). InnerVolumeSpecName "kube-api-access-29qvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:34:24.680909 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.680874 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-501d63-5fd9cc8bfb-krlt9" Nov 01 16:34:24.685678 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.685610 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" Nov 01 16:34:24.715289 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.715069 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6daeee34-6958-40f5-a360-9a58b3bd51d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6daeee34-6958-40f5-a360-9a58b3bd51d8-credential-socket\") pod \"6daeee34-6958-40f5-a360-9a58b3bd51d8\" (UID: \"6daeee34-6958-40f5-a360-9a58b3bd51d8\") " Nov 01 16:34:24.715289 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.715124 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6daeee34-6958-40f5-a360-9a58b3bd51d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6daeee34-6958-40f5-a360-9a58b3bd51d8-istiod-ca-cert\") pod \"6daeee34-6958-40f5-a360-9a58b3bd51d8\" (UID: \"6daeee34-6958-40f5-a360-9a58b3bd51d8\") " Nov 01 16:34:24.715289 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.715150 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6daeee34-6958-40f5-a360-9a58b3bd51d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6daeee34-6958-40f5-a360-9a58b3bd51d8-istio-data\") pod \"6daeee34-6958-40f5-a360-9a58b3bd51d8\" (UID: \"6daeee34-6958-40f5-a360-9a58b3bd51d8\") " Nov 01 16:34:24.715289 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.715167 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6daeee34-6958-40f5-a360-9a58b3bd51d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6daeee34-6958-40f5-a360-9a58b3bd51d8-istio-podinfo\") pod \"6daeee34-6958-40f5-a360-9a58b3bd51d8\" (UID: \"6daeee34-6958-40f5-a360-9a58b3bd51d8\") " Nov 01 16:34:24.715659 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.715493 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6daeee34-6958-40f5-a360-9a58b3bd51d8-istio-data" pod "6daeee34-6958-40f5-a360-9a58b3bd51d8" (UID: "6daeee34-6958-40f5-a360-9a58b3bd51d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:24.715659 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.715563 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6daeee34-6958-40f5-a360-9a58b3bd51d8-credential-socket" pod "6daeee34-6958-40f5-a360-9a58b3bd51d8" (UID: "6daeee34-6958-40f5-a360-9a58b3bd51d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:24.716965 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.715810 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb31d5ec-ff4f-4fbf-b78f-ab022f851e78-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb31d5ec-ff4f-4fbf-b78f-ab022f851e78-plugins-config-volume\") pod \"cb31d5ec-ff4f-4fbf-b78f-ab022f851e78\" (UID: \"cb31d5ec-ff4f-4fbf-b78f-ab022f851e78\") " Nov 01 16:34:24.716965 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.715892 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6daeee34-6958-40f5-a360-9a58b3bd51d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6daeee34-6958-40f5-a360-9a58b3bd51d8-istio-envoy\") pod \"6daeee34-6958-40f5-a360-9a58b3bd51d8\" (UID: \"6daeee34-6958-40f5-a360-9a58b3bd51d8\") " Nov 01 16:34:24.716965 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.715922 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6daeee34-6958-40f5-a360-9a58b3bd51d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6daeee34-6958-40f5-a360-9a58b3bd51d8-workload-certs\") pod \"6daeee34-6958-40f5-a360-9a58b3bd51d8\" (UID: \"6daeee34-6958-40f5-a360-9a58b3bd51d8\") " Nov 01 16:34:24.716965 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.715921 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb31d5ec-ff4f-4fbf-b78f-ab022f851e78-plugins-config-volume" pod "cb31d5ec-ff4f-4fbf-b78f-ab022f851e78" (UID: "cb31d5ec-ff4f-4fbf-b78f-ab022f851e78"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:34:24.716965 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.715978 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6daeee34-6958-40f5-a360-9a58b3bd51d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6daeee34-6958-40f5-a360-9a58b3bd51d8-workload-socket\") pod \"6daeee34-6958-40f5-a360-9a58b3bd51d8\" (UID: \"6daeee34-6958-40f5-a360-9a58b3bd51d8\") " Nov 01 16:34:24.716965 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.716097 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb31d5ec-ff4f-4fbf-b78f-ab022f851e78-kube-api-access-k69ks\" (UniqueName: \"kubernetes.io/projected/cb31d5ec-ff4f-4fbf-b78f-ab022f851e78-kube-api-access-k69ks\") pod \"cb31d5ec-ff4f-4fbf-b78f-ab022f851e78\" (UID: \"cb31d5ec-ff4f-4fbf-b78f-ab022f851e78\") " Nov 01 16:34:24.716965 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.716134 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6daeee34-6958-40f5-a360-9a58b3bd51d8-istio-token\" (UniqueName: \"kubernetes.io/projected/6daeee34-6958-40f5-a360-9a58b3bd51d8-istio-token\") pod \"6daeee34-6958-40f5-a360-9a58b3bd51d8\" (UID: \"6daeee34-6958-40f5-a360-9a58b3bd51d8\") " Nov 01 16:34:24.716965 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.716204 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6daeee34-6958-40f5-a360-9a58b3bd51d8-kube-api-access-6zmnv\" (UniqueName: \"kubernetes.io/projected/6daeee34-6958-40f5-a360-9a58b3bd51d8-kube-api-access-6zmnv\") pod \"6daeee34-6958-40f5-a360-9a58b3bd51d8\" (UID: \"6daeee34-6958-40f5-a360-9a58b3bd51d8\") " Nov 01 16:34:24.716965 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.716271 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6daeee34-6958-40f5-a360-9a58b3bd51d8-istiod-ca-cert" pod "6daeee34-6958-40f5-a360-9a58b3bd51d8" (UID: "6daeee34-6958-40f5-a360-9a58b3bd51d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:34:24.716965 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.716495 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6daeee34-6958-40f5-a360-9a58b3bd51d8-workload-socket" pod "6daeee34-6958-40f5-a360-9a58b3bd51d8" (UID: "6daeee34-6958-40f5-a360-9a58b3bd51d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:24.716965 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.716557 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6daeee34-6958-40f5-a360-9a58b3bd51d8-workload-certs" pod "6daeee34-6958-40f5-a360-9a58b3bd51d8" (UID: "6daeee34-6958-40f5-a360-9a58b3bd51d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:24.716965 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.716738 4363 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/159aed21-0910-4554-a841-b7c096765924-metrics-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:24.716965 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.716760 4363 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/159aed21-0910-4554-a841-b7c096765924-openshift-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:24.716965 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.716781 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6daeee34-6958-40f5-a360-9a58b3bd51d8-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:24.716965 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.716800 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6daeee34-6958-40f5-a360-9a58b3bd51d8-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:24.716965 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.716813 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6daeee34-6958-40f5-a360-9a58b3bd51d8-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:24.716965 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.716826 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb31d5ec-ff4f-4fbf-b78f-ab022f851e78-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:24.716965 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.716840 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwglg\" (UniqueName: \"kubernetes.io/projected/4324048b-b48c-452d-91df-8e48f5e8e97a-kube-api-access-pwglg\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:24.716965 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.716852 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6daeee34-6958-40f5-a360-9a58b3bd51d8-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:24.716965 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.716868 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/159aed21-0910-4554-a841-b7c096765924-maas-api-tls\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:24.716965 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.716886 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6daeee34-6958-40f5-a360-9a58b3bd51d8-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:24.716965 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.716902 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29qvs\" (UniqueName: \"kubernetes.io/projected/159aed21-0910-4554-a841-b7c096765924-kube-api-access-29qvs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:24.723055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.721757 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6daeee34-6958-40f5-a360-9a58b3bd51d8-istio-token" pod "6daeee34-6958-40f5-a360-9a58b3bd51d8" (UID: "6daeee34-6958-40f5-a360-9a58b3bd51d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:34:24.723055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.721896 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6daeee34-6958-40f5-a360-9a58b3bd51d8-istio-envoy" pod "6daeee34-6958-40f5-a360-9a58b3bd51d8" (UID: "6daeee34-6958-40f5-a360-9a58b3bd51d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:24.723055 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.722189 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6daeee34-6958-40f5-a360-9a58b3bd51d8-istio-podinfo" pod "6daeee34-6958-40f5-a360-9a58b3bd51d8" (UID: "6daeee34-6958-40f5-a360-9a58b3bd51d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:34:24.723468 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.723438 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb31d5ec-ff4f-4fbf-b78f-ab022f851e78-kube-api-access-k69ks" pod "cb31d5ec-ff4f-4fbf-b78f-ab022f851e78" (UID: "cb31d5ec-ff4f-4fbf-b78f-ab022f851e78"). InnerVolumeSpecName "kube-api-access-k69ks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:34:24.723562 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.723498 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6daeee34-6958-40f5-a360-9a58b3bd51d8-kube-api-access-6zmnv" pod "6daeee34-6958-40f5-a360-9a58b3bd51d8" (UID: "6daeee34-6958-40f5-a360-9a58b3bd51d8"). InnerVolumeSpecName "kube-api-access-6zmnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:34:24.818256 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.818160 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k69ks\" (UniqueName: \"kubernetes.io/projected/cb31d5ec-ff4f-4fbf-b78f-ab022f851e78-kube-api-access-k69ks\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:24.818256 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.818196 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6daeee34-6958-40f5-a360-9a58b3bd51d8-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:24.818256 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.818212 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zmnv\" (UniqueName: \"kubernetes.io/projected/6daeee34-6958-40f5-a360-9a58b3bd51d8-kube-api-access-6zmnv\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:24.818256 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.818227 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6daeee34-6958-40f5-a360-9a58b3bd51d8-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:24.818256 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:24.818241 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6daeee34-6958-40f5-a360-9a58b3bd51d8-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:25.281421 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:25.281384 4363 generic.go:356] "Generic (PLEG): container finished" podID="cb31d5ec-ff4f-4fbf-b78f-ab022f851e78" containerID="b267c720a2cffea3538cef5a39e37536ff1eb682d81d6e630e423362a02ec8bf" exitCode=0 Nov 01 16:34:25.281899 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:25.281426 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-501d63-5fd9cc8bfb-krlt9" Nov 01 16:34:25.281899 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:25.281469 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-501d63-5fd9cc8bfb-krlt9" event={"ID":"cb31d5ec-ff4f-4fbf-b78f-ab022f851e78","Type":"ContainerDied","Data":"b267c720a2cffea3538cef5a39e37536ff1eb682d81d6e630e423362a02ec8bf"} Nov 01 16:34:25.281899 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:25.281524 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-501d63-5fd9cc8bfb-krlt9" event={"ID":"cb31d5ec-ff4f-4fbf-b78f-ab022f851e78","Type":"ContainerDied","Data":"b4c2f5f82d1f2cc7c28eb8676ffd1bf63c735c8080f5df91444448c0c8c3636d"} Nov 01 16:34:25.281899 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:25.281550 4363 scope.go:122] "RemoveContainer" containerID="b267c720a2cffea3538cef5a39e37536ff1eb682d81d6e630e423362a02ec8bf" Nov 01 16:34:25.283043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:25.282991 4363 generic.go:356] "Generic (PLEG): container finished" podID="6daeee34-6958-40f5-a360-9a58b3bd51d8" containerID="d7ae03be4b5c7e4ffa4ab1db04b0f8bd4db603819eeb69eaa27a23ada687c16d" exitCode=0 Nov 01 16:34:25.283187 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:25.283066 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" Nov 01 16:34:25.283187 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:25.283070 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" event={"ID":"6daeee34-6958-40f5-a360-9a58b3bd51d8","Type":"ContainerDied","Data":"d7ae03be4b5c7e4ffa4ab1db04b0f8bd4db603819eeb69eaa27a23ada687c16d"} Nov 01 16:34:25.283187 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:25.283128 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv" event={"ID":"6daeee34-6958-40f5-a360-9a58b3bd51d8","Type":"ContainerDied","Data":"7afe6c52d476e530d71e7448695e4effa4e5c26b049ee4acc3a484fa27544d96"} Nov 01 16:34:25.285222 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:25.285202 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-501d63-56568c796c-8lg27" Nov 01 16:34:25.285222 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:25.285212 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-501d63-56568c796c-8lg27" event={"ID":"159aed21-0910-4554-a841-b7c096765924","Type":"ContainerDied","Data":"acb171ad1c32b47eced4829ed00e1b9fa250f435d77b810ec7dba4ab0c5998e1"} Nov 01 16:34:25.287710 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:25.287680 4363 generic.go:356] "Generic (PLEG): container finished" podID="4324048b-b48c-452d-91df-8e48f5e8e97a" containerID="8ec4cf5de10cf531018ee05f14532add7ad643104f09334d57e8800d305c34ac" exitCode=0 Nov 01 16:34:25.287811 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:25.287722 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-678bbff655-z866s" Nov 01 16:34:25.287811 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:25.287755 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-678bbff655-z866s" event={"ID":"4324048b-b48c-452d-91df-8e48f5e8e97a","Type":"ContainerDied","Data":"8ec4cf5de10cf531018ee05f14532add7ad643104f09334d57e8800d305c34ac"} Nov 01 16:34:25.287811 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:25.287795 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-678bbff655-z866s" event={"ID":"4324048b-b48c-452d-91df-8e48f5e8e97a","Type":"ContainerDied","Data":"58938c9014f65263d985a957599bbf0997d784570f2825e667465cf6a6f89030"} Nov 01 16:34:25.293791 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:25.293771 4363 scope.go:122] "RemoveContainer" containerID="b267c720a2cffea3538cef5a39e37536ff1eb682d81d6e630e423362a02ec8bf" Nov 01 16:34:25.294096 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:34:25.294076 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b267c720a2cffea3538cef5a39e37536ff1eb682d81d6e630e423362a02ec8bf\": container with ID starting with b267c720a2cffea3538cef5a39e37536ff1eb682d81d6e630e423362a02ec8bf not found: ID does not exist" containerID="b267c720a2cffea3538cef5a39e37536ff1eb682d81d6e630e423362a02ec8bf" Nov 01 16:34:25.294183 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:25.294103 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b267c720a2cffea3538cef5a39e37536ff1eb682d81d6e630e423362a02ec8bf"} err="failed to get container status \"b267c720a2cffea3538cef5a39e37536ff1eb682d81d6e630e423362a02ec8bf\": rpc error: code = NotFound desc = could not find container \"b267c720a2cffea3538cef5a39e37536ff1eb682d81d6e630e423362a02ec8bf\": container with ID starting with b267c720a2cffea3538cef5a39e37536ff1eb682d81d6e630e423362a02ec8bf not found: ID does not exist" Nov 01 16:34:25.294183 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:25.294119 4363 scope.go:122] "RemoveContainer" containerID="d7ae03be4b5c7e4ffa4ab1db04b0f8bd4db603819eeb69eaa27a23ada687c16d" Nov 01 16:34:25.310560 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:25.310513 4363 scope.go:122] "RemoveContainer" containerID="d7ae03be4b5c7e4ffa4ab1db04b0f8bd4db603819eeb69eaa27a23ada687c16d" Nov 01 16:34:25.311116 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:25.311081 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv"] Nov 01 16:34:25.311386 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:34:25.311368 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7ae03be4b5c7e4ffa4ab1db04b0f8bd4db603819eeb69eaa27a23ada687c16d\": container with ID starting with d7ae03be4b5c7e4ffa4ab1db04b0f8bd4db603819eeb69eaa27a23ada687c16d not found: ID does not exist" containerID="d7ae03be4b5c7e4ffa4ab1db04b0f8bd4db603819eeb69eaa27a23ada687c16d" Nov 01 16:34:25.311485 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:25.311396 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7ae03be4b5c7e4ffa4ab1db04b0f8bd4db603819eeb69eaa27a23ada687c16d"} err="failed to get container status \"d7ae03be4b5c7e4ffa4ab1db04b0f8bd4db603819eeb69eaa27a23ada687c16d\": rpc error: code = NotFound desc = could not find container \"d7ae03be4b5c7e4ffa4ab1db04b0f8bd4db603819eeb69eaa27a23ada687c16d\": container with ID starting with d7ae03be4b5c7e4ffa4ab1db04b0f8bd4db603819eeb69eaa27a23ada687c16d not found: ID does not exist" Nov 01 16:34:25.311485 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:25.311419 4363 scope.go:122] "RemoveContainer" containerID="4799dce9509cf8342931fe38b89a5500d33bfe13610b4b8eba2a72dc08d1e3d1" Nov 01 16:34:25.313957 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:25.313933 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-9v4pv"] Nov 01 16:34:25.322172 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:25.322147 4363 scope.go:122] "RemoveContainer" containerID="8ec4cf5de10cf531018ee05f14532add7ad643104f09334d57e8800d305c34ac" Nov 01 16:34:25.322479 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:25.322460 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-678bbff655-z866s"] Nov 01 16:34:25.329806 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:25.329786 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-501d63-678bbff655-z866s"] Nov 01 16:34:25.332528 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:25.332510 4363 scope.go:122] "RemoveContainer" containerID="8ec4cf5de10cf531018ee05f14532add7ad643104f09334d57e8800d305c34ac" Nov 01 16:34:25.332957 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:34:25.332923 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ec4cf5de10cf531018ee05f14532add7ad643104f09334d57e8800d305c34ac\": container with ID starting with 8ec4cf5de10cf531018ee05f14532add7ad643104f09334d57e8800d305c34ac not found: ID does not exist" containerID="8ec4cf5de10cf531018ee05f14532add7ad643104f09334d57e8800d305c34ac" Nov 01 16:34:25.333112 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:25.332965 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ec4cf5de10cf531018ee05f14532add7ad643104f09334d57e8800d305c34ac"} err="failed to get container status \"8ec4cf5de10cf531018ee05f14532add7ad643104f09334d57e8800d305c34ac\": rpc error: code = NotFound desc = could not find container \"8ec4cf5de10cf531018ee05f14532add7ad643104f09334d57e8800d305c34ac\": container with ID starting with 8ec4cf5de10cf531018ee05f14532add7ad643104f09334d57e8800d305c34ac not found: ID does not exist" Nov 01 16:34:25.340726 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:25.340703 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-501d63-56568c796c-8lg27"] Nov 01 16:34:25.345843 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:25.345822 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-501d63-56568c796c-8lg27"] Nov 01 16:34:25.355570 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:25.355552 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-501d63-5fd9cc8bfb-krlt9"] Nov 01 16:34:25.360915 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:25.360893 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-501d63-5fd9cc8bfb-krlt9"] Nov 01 16:34:26.798268 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:26.798232 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="159aed21-0910-4554-a841-b7c096765924" path="/var/lib/kubelet/pods/159aed21-0910-4554-a841-b7c096765924/volumes" Nov 01 16:34:26.798749 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:26.798620 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4324048b-b48c-452d-91df-8e48f5e8e97a" path="/var/lib/kubelet/pods/4324048b-b48c-452d-91df-8e48f5e8e97a/volumes" Nov 01 16:34:26.799046 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:26.799002 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6daeee34-6958-40f5-a360-9a58b3bd51d8" path="/var/lib/kubelet/pods/6daeee34-6958-40f5-a360-9a58b3bd51d8/volumes" Nov 01 16:34:26.799684 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:26.799667 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb31d5ec-ff4f-4fbf-b78f-ab022f851e78" path="/var/lib/kubelet/pods/cb31d5ec-ff4f-4fbf-b78f-ab022f851e78/volumes" Nov 01 16:34:30.019806 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:30.019774 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11450cde-vvjzv"] Nov 01 16:34:30.025092 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:30.025067 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11450cde-vvjzv"] Nov 01 16:34:30.797298 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:30.797268 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08a13416-5e1c-4bc0-a4ea-023e4960bc2f" path="/var/lib/kubelet/pods/08a13416-5e1c-4bc0-a4ea-023e4960bc2f/volumes" Nov 01 16:34:31.809628 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:31.809586 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b"] Nov 01 16:34:31.810248 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:31.809865 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" podUID="b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697" containerName="istio-proxy" containerID="cri-o://c4c8270ffe66f1ce2618d639554942cdf1db53eeef7aee617e58e4ef7eeda99a" gracePeriod=30 Nov 01 16:34:37.064326 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:37.064300 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" Nov 01 16:34:37.162474 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:37.162438 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-istiod-ca-cert\") pod \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\" (UID: \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\") " Nov 01 16:34:37.162661 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:37.162545 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-workload-socket\") pod \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\" (UID: \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\") " Nov 01 16:34:37.162661 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:37.162596 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-credential-socket\") pod \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\" (UID: \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\") " Nov 01 16:34:37.162661 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:37.162622 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-istio-token\" (UniqueName: \"kubernetes.io/projected/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-istio-token\") pod \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\" (UID: \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\") " Nov 01 16:34:37.162661 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:37.162645 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-istio-envoy\") pod \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\" (UID: \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\") " Nov 01 16:34:37.162834 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:37.162677 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-workload-certs\") pod \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\" (UID: \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\") " Nov 01 16:34:37.162834 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:37.162698 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-istio-data\") pod \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\" (UID: \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\") " Nov 01 16:34:37.162834 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:37.162729 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-istio-podinfo\") pod \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\" (UID: \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\") " Nov 01 16:34:37.162834 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:37.162766 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-kube-api-access-pp7zc\" (UniqueName: \"kubernetes.io/projected/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-kube-api-access-pp7zc\") pod \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\" (UID: \"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697\") " Nov 01 16:34:37.163101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:37.162947 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-credential-socket" pod "b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697" (UID: "b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:37.163101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:37.162965 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-istiod-ca-cert" pod "b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697" (UID: "b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:34:37.163209 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:37.163102 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-istio-data" pod "b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697" (UID: "b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:37.163209 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:37.163123 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-workload-certs" pod "b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697" (UID: "b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:37.163333 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:37.163249 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-workload-socket" pod "b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697" (UID: "b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:37.163464 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:37.163440 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:37.163537 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:37.163469 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:37.163537 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:37.163486 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:37.163537 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:37.163500 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:37.163537 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:37.163514 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:37.164899 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:37.164873 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-istio-token" pod "b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697" (UID: "b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:34:37.165057 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:37.165003 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-istio-podinfo" pod "b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697" (UID: "b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:34:37.165057 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:37.165047 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-kube-api-access-pp7zc" pod "b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697" (UID: "b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697"). InnerVolumeSpecName "kube-api-access-pp7zc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:34:37.165165 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:37.165080 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-istio-envoy" pod "b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697" (UID: "b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:37.264506 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:37.264475 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:37.264506 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:37.264500 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:37.264506 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:37.264509 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:37.264753 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:37.264519 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp7zc\" (UniqueName: \"kubernetes.io/projected/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697-kube-api-access-pp7zc\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:37.378293 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:37.378257 4363 generic.go:356] "Generic (PLEG): container finished" podID="b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697" containerID="c4c8270ffe66f1ce2618d639554942cdf1db53eeef7aee617e58e4ef7eeda99a" exitCode=0 Nov 01 16:34:37.378446 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:37.378320 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" Nov 01 16:34:37.378446 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:37.378338 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" event={"ID":"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697","Type":"ContainerDied","Data":"c4c8270ffe66f1ce2618d639554942cdf1db53eeef7aee617e58e4ef7eeda99a"} Nov 01 16:34:37.378446 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:37.378422 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b" event={"ID":"b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697","Type":"ContainerDied","Data":"b49f14168080113075111ee267d67394c7e52446585f8f045194cec6d042011d"} Nov 01 16:34:37.378446 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:37.378444 4363 scope.go:122] "RemoveContainer" containerID="c4c8270ffe66f1ce2618d639554942cdf1db53eeef7aee617e58e4ef7eeda99a" Nov 01 16:34:37.389035 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:37.388994 4363 scope.go:122] "RemoveContainer" containerID="11d85b9dc8a77601e7240682f3127f7464c911c041076576559958705ce20c57" Nov 01 16:34:37.397834 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:37.397804 4363 scope.go:122] "RemoveContainer" containerID="c4c8270ffe66f1ce2618d639554942cdf1db53eeef7aee617e58e4ef7eeda99a" Nov 01 16:34:37.398440 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:34:37.398412 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4c8270ffe66f1ce2618d639554942cdf1db53eeef7aee617e58e4ef7eeda99a\": container with ID starting with c4c8270ffe66f1ce2618d639554942cdf1db53eeef7aee617e58e4ef7eeda99a not found: ID does not exist" containerID="c4c8270ffe66f1ce2618d639554942cdf1db53eeef7aee617e58e4ef7eeda99a" Nov 01 16:34:37.398609 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:37.398449 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4c8270ffe66f1ce2618d639554942cdf1db53eeef7aee617e58e4ef7eeda99a"} err="failed to get container status \"c4c8270ffe66f1ce2618d639554942cdf1db53eeef7aee617e58e4ef7eeda99a\": rpc error: code = NotFound desc = could not find container \"c4c8270ffe66f1ce2618d639554942cdf1db53eeef7aee617e58e4ef7eeda99a\": container with ID starting with c4c8270ffe66f1ce2618d639554942cdf1db53eeef7aee617e58e4ef7eeda99a not found: ID does not exist" Nov 01 16:34:37.398609 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:37.398474 4363 scope.go:122] "RemoveContainer" containerID="11d85b9dc8a77601e7240682f3127f7464c911c041076576559958705ce20c57" Nov 01 16:34:37.398811 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:34:37.398792 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11d85b9dc8a77601e7240682f3127f7464c911c041076576559958705ce20c57\": container with ID starting with 11d85b9dc8a77601e7240682f3127f7464c911c041076576559958705ce20c57 not found: ID does not exist" containerID="11d85b9dc8a77601e7240682f3127f7464c911c041076576559958705ce20c57" Nov 01 16:34:37.398865 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:37.398821 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11d85b9dc8a77601e7240682f3127f7464c911c041076576559958705ce20c57"} err="failed to get container status \"11d85b9dc8a77601e7240682f3127f7464c911c041076576559958705ce20c57\": rpc error: code = NotFound desc = could not find container \"11d85b9dc8a77601e7240682f3127f7464c911c041076576559958705ce20c57\": container with ID starting with 11d85b9dc8a77601e7240682f3127f7464c911c041076576559958705ce20c57 not found: ID does not exist" Nov 01 16:34:37.400922 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:37.400897 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b"] Nov 01 16:34:37.404551 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:37.404528 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-501d63-openshift-default-b85b799f7-hjm4b"] Nov 01 16:34:38.795824 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:38.795795 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697" path="/var/lib/kubelet/pods/b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697/volumes" Nov 01 16:34:44.645547 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:44.645513 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f4a265-dc856b996-7fdck"] Nov 01 16:34:44.646250 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:44.645780 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4a265-dc856b996-7fdck" podUID="c2d00eaa-b184-4176-9944-ef381e8eaec0" containerName="payload-processing" containerID="cri-o://81e22bea5dfcc4b2c55cd463a1de1c16f1886853535f5b50e142c3715c80573a" gracePeriod=30 Nov 01 16:34:44.656497 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:44.656457 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-7c6884bdff-d444d"] Nov 01 16:34:44.656755 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:44.656726 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-7c6884bdff-d444d" podUID="0240fa16-cc94-4d43-a3a8-ae27ca53f6ec" containerName="payload-pre-processing" containerID="cri-o://bfb866b45de778ea21533405137525e43e102d4c78d6ad5913b4b87e936bfa8f" gracePeriod=30 Nov 01 16:34:44.941387 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:44.941355 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4a265-dc856b996-7fdck" Nov 01 16:34:44.974091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:44.974062 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-7c6884bdff-d444d" Nov 01 16:34:45.049303 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:45.049267 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2d00eaa-b184-4176-9944-ef381e8eaec0-kube-api-access-vcrbh\" (UniqueName: \"kubernetes.io/projected/c2d00eaa-b184-4176-9944-ef381e8eaec0-kube-api-access-vcrbh\") pod \"c2d00eaa-b184-4176-9944-ef381e8eaec0\" (UID: \"c2d00eaa-b184-4176-9944-ef381e8eaec0\") " Nov 01 16:34:45.050225 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:45.050195 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2d00eaa-b184-4176-9944-ef381e8eaec0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2d00eaa-b184-4176-9944-ef381e8eaec0-plugins-config-volume\") pod \"c2d00eaa-b184-4176-9944-ef381e8eaec0\" (UID: \"c2d00eaa-b184-4176-9944-ef381e8eaec0\") " Nov 01 16:34:45.050371 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:45.050252 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0240fa16-cc94-4d43-a3a8-ae27ca53f6ec-kube-api-access-m7f97\" (UniqueName: \"kubernetes.io/projected/0240fa16-cc94-4d43-a3a8-ae27ca53f6ec-kube-api-access-m7f97\") pod \"0240fa16-cc94-4d43-a3a8-ae27ca53f6ec\" (UID: \"0240fa16-cc94-4d43-a3a8-ae27ca53f6ec\") " Nov 01 16:34:45.050371 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:45.050303 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0240fa16-cc94-4d43-a3a8-ae27ca53f6ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0240fa16-cc94-4d43-a3a8-ae27ca53f6ec-plugins-config-volume\") pod \"0240fa16-cc94-4d43-a3a8-ae27ca53f6ec\" (UID: \"0240fa16-cc94-4d43-a3a8-ae27ca53f6ec\") " Nov 01 16:34:45.050812 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:45.050776 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2d00eaa-b184-4176-9944-ef381e8eaec0-plugins-config-volume" pod "c2d00eaa-b184-4176-9944-ef381e8eaec0" (UID: "c2d00eaa-b184-4176-9944-ef381e8eaec0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:34:45.050983 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:45.050957 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2d00eaa-b184-4176-9944-ef381e8eaec0-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:45.052484 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:45.051630 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0240fa16-cc94-4d43-a3a8-ae27ca53f6ec-plugins-config-volume" pod "0240fa16-cc94-4d43-a3a8-ae27ca53f6ec" (UID: "0240fa16-cc94-4d43-a3a8-ae27ca53f6ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:34:45.053325 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:45.053301 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0240fa16-cc94-4d43-a3a8-ae27ca53f6ec-kube-api-access-m7f97" pod "0240fa16-cc94-4d43-a3a8-ae27ca53f6ec" (UID: "0240fa16-cc94-4d43-a3a8-ae27ca53f6ec"). InnerVolumeSpecName "kube-api-access-m7f97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:34:45.054490 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:45.054463 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2d00eaa-b184-4176-9944-ef381e8eaec0-kube-api-access-vcrbh" pod "c2d00eaa-b184-4176-9944-ef381e8eaec0" (UID: "c2d00eaa-b184-4176-9944-ef381e8eaec0"). InnerVolumeSpecName "kube-api-access-vcrbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:34:45.151818 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:45.151771 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcrbh\" (UniqueName: \"kubernetes.io/projected/c2d00eaa-b184-4176-9944-ef381e8eaec0-kube-api-access-vcrbh\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:45.151818 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:45.151814 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7f97\" (UniqueName: \"kubernetes.io/projected/0240fa16-cc94-4d43-a3a8-ae27ca53f6ec-kube-api-access-m7f97\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:45.152133 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:45.151833 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0240fa16-cc94-4d43-a3a8-ae27ca53f6ec-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:45.432827 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:45.432793 4363 generic.go:356] "Generic (PLEG): container finished" podID="0240fa16-cc94-4d43-a3a8-ae27ca53f6ec" containerID="bfb866b45de778ea21533405137525e43e102d4c78d6ad5913b4b87e936bfa8f" exitCode=0 Nov 01 16:34:45.433070 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:45.432848 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-7c6884bdff-d444d" Nov 01 16:34:45.433070 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:45.432864 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-7c6884bdff-d444d" event={"ID":"0240fa16-cc94-4d43-a3a8-ae27ca53f6ec","Type":"ContainerDied","Data":"bfb866b45de778ea21533405137525e43e102d4c78d6ad5913b4b87e936bfa8f"} Nov 01 16:34:45.433070 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:45.432921 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-7c6884bdff-d444d" event={"ID":"0240fa16-cc94-4d43-a3a8-ae27ca53f6ec","Type":"ContainerDied","Data":"631153c23440b0fa70fe871c15cad7f379dac01cd5ebdb6d7369249857bf50fb"} Nov 01 16:34:45.433070 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:45.432943 4363 scope.go:122] "RemoveContainer" containerID="bfb866b45de778ea21533405137525e43e102d4c78d6ad5913b4b87e936bfa8f" Nov 01 16:34:45.434445 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:45.434422 4363 generic.go:356] "Generic (PLEG): container finished" podID="c2d00eaa-b184-4176-9944-ef381e8eaec0" containerID="81e22bea5dfcc4b2c55cd463a1de1c16f1886853535f5b50e142c3715c80573a" exitCode=0 Nov 01 16:34:45.434574 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:45.434476 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4a265-dc856b996-7fdck" Nov 01 16:34:45.434574 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:45.434496 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4a265-dc856b996-7fdck" event={"ID":"c2d00eaa-b184-4176-9944-ef381e8eaec0","Type":"ContainerDied","Data":"81e22bea5dfcc4b2c55cd463a1de1c16f1886853535f5b50e142c3715c80573a"} Nov 01 16:34:45.434574 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:45.434548 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f4a265-dc856b996-7fdck" event={"ID":"c2d00eaa-b184-4176-9944-ef381e8eaec0","Type":"ContainerDied","Data":"c9d6f87dbcca3799bb387a7891b03773a5c35565c9abfdf364e51f67bccf3dbd"} Nov 01 16:34:45.443849 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:45.443825 4363 scope.go:122] "RemoveContainer" containerID="bfb866b45de778ea21533405137525e43e102d4c78d6ad5913b4b87e936bfa8f" Nov 01 16:34:45.444311 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:34:45.444158 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfb866b45de778ea21533405137525e43e102d4c78d6ad5913b4b87e936bfa8f\": container with ID starting with bfb866b45de778ea21533405137525e43e102d4c78d6ad5913b4b87e936bfa8f not found: ID does not exist" containerID="bfb866b45de778ea21533405137525e43e102d4c78d6ad5913b4b87e936bfa8f" Nov 01 16:34:45.444311 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:45.444184 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfb866b45de778ea21533405137525e43e102d4c78d6ad5913b4b87e936bfa8f"} err="failed to get container status \"bfb866b45de778ea21533405137525e43e102d4c78d6ad5913b4b87e936bfa8f\": rpc error: code = NotFound desc = could not find container \"bfb866b45de778ea21533405137525e43e102d4c78d6ad5913b4b87e936bfa8f\": container with ID starting with bfb866b45de778ea21533405137525e43e102d4c78d6ad5913b4b87e936bfa8f not found: ID does not exist" Nov 01 16:34:45.444311 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:45.444200 4363 scope.go:122] "RemoveContainer" containerID="81e22bea5dfcc4b2c55cd463a1de1c16f1886853535f5b50e142c3715c80573a" Nov 01 16:34:45.454103 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:45.454083 4363 scope.go:122] "RemoveContainer" containerID="81e22bea5dfcc4b2c55cd463a1de1c16f1886853535f5b50e142c3715c80573a" Nov 01 16:34:45.454443 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:34:45.454380 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81e22bea5dfcc4b2c55cd463a1de1c16f1886853535f5b50e142c3715c80573a\": container with ID starting with 81e22bea5dfcc4b2c55cd463a1de1c16f1886853535f5b50e142c3715c80573a not found: ID does not exist" containerID="81e22bea5dfcc4b2c55cd463a1de1c16f1886853535f5b50e142c3715c80573a" Nov 01 16:34:45.454443 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:45.454416 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81e22bea5dfcc4b2c55cd463a1de1c16f1886853535f5b50e142c3715c80573a"} err="failed to get container status \"81e22bea5dfcc4b2c55cd463a1de1c16f1886853535f5b50e142c3715c80573a\": rpc error: code = NotFound desc = could not find container \"81e22bea5dfcc4b2c55cd463a1de1c16f1886853535f5b50e142c3715c80573a\": container with ID starting with 81e22bea5dfcc4b2c55cd463a1de1c16f1886853535f5b50e142c3715c80573a not found: ID does not exist" Nov 01 16:34:45.462130 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:45.462109 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f4a265-dc856b996-7fdck"] Nov 01 16:34:45.465683 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:45.465664 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f4a265-dc856b996-7fdck"] Nov 01 16:34:45.478375 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:45.478352 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-7c6884bdff-d444d"] Nov 01 16:34:45.483073 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:45.483047 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-7c6884bdff-d444d"] Nov 01 16:34:46.797200 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:46.797170 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0240fa16-cc94-4d43-a3a8-ae27ca53f6ec" path="/var/lib/kubelet/pods/0240fa16-cc94-4d43-a3a8-ae27ca53f6ec/volumes" Nov 01 16:34:46.797595 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:46.797562 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2d00eaa-b184-4176-9944-ef381e8eaec0" path="/var/lib/kubelet/pods/c2d00eaa-b184-4176-9944-ef381e8eaec0/volumes" Nov 01 16:34:48.315085 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:34:48.315043 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:34:51.670794 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:51.670760 4363 scope.go:122] "RemoveContainer" containerID="5ebf85af4e623dbba635f82d4d41892bcab41e222c59e9abe96186dbe29d9217" Nov 01 16:34:51.680776 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:51.680755 4363 scope.go:122] "RemoveContainer" containerID="e977be825bddf57fb10c25ad1f6847116d0811c06518d8add2872ce6b8d784be" Nov 01 16:34:51.690125 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:51.690088 4363 scope.go:122] "RemoveContainer" containerID="f8415a4045b543deeadededfd9d59061bb7225d0a75bdf187cf98588e9f26564" Nov 01 16:34:51.698948 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:51.698924 4363 scope.go:122] "RemoveContainer" containerID="68b9557e25d5e5185456b38789a0f131bf85af0e5ef7929ef140aac47031c22c" Nov 01 16:34:51.708945 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:51.708925 4363 scope.go:122] "RemoveContainer" containerID="e219a15223593c53cab77d5ffd00c81a0c6504b584e2112e7959169b942025e3" Nov 01 16:34:51.910044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:51.900040 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d"] Nov 01 16:34:51.910044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:51.900285 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" podUID="87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519" containerName="istio-proxy" containerID="cri-o://500a1c7400013ba1a920f48ce08ce87680262ace9435db82d30a8ea72c34cd9e" gracePeriod=30 Nov 01 16:34:57.171048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:57.170999 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" Nov 01 16:34:57.292897 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:57.292816 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-istio-data\") pod \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\" (UID: \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\") " Nov 01 16:34:57.292897 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:57.292861 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-workload-socket\") pod \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\" (UID: \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\") " Nov 01 16:34:57.293132 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:57.292909 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-istio-podinfo\") pod \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\" (UID: \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\") " Nov 01 16:34:57.293132 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:57.292963 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-credential-socket\") pod \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\" (UID: \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\") " Nov 01 16:34:57.293132 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:57.293038 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-kube-api-access-8qqsc\" (UniqueName: \"kubernetes.io/projected/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-kube-api-access-8qqsc\") pod \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\" (UID: \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\") " Nov 01 16:34:57.293132 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:57.293073 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-istiod-ca-cert\") pod \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\" (UID: \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\") " Nov 01 16:34:57.293132 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:57.293103 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-istio-envoy\") pod \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\" (UID: \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\") " Nov 01 16:34:57.293438 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:57.293140 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-workload-certs\") pod \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\" (UID: \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\") " Nov 01 16:34:57.293438 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:57.293167 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-istio-token\" (UniqueName: \"kubernetes.io/projected/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-istio-token\") pod \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\" (UID: \"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519\") " Nov 01 16:34:57.293438 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:57.293160 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-istio-data" pod "87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519" (UID: "87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:57.293556 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:57.293187 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-workload-socket" pod "87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519" (UID: "87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:57.293556 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:57.293327 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-credential-socket" pod "87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519" (UID: "87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:57.293556 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:57.293373 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-workload-certs" pod "87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519" (UID: "87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:57.293703 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:57.293614 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-istiod-ca-cert" pod "87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519" (UID: "87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:34:57.293703 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:57.293644 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:57.293703 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:57.293667 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:57.293703 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:57.293681 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:57.293703 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:57.293694 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:57.295328 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:57.295301 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-istio-token" pod "87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519" (UID: "87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:34:57.295438 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:57.295337 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-kube-api-access-8qqsc" pod "87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519" (UID: "87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519"). InnerVolumeSpecName "kube-api-access-8qqsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:34:57.295438 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:57.295366 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-istio-envoy" pod "87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519" (UID: "87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:57.295438 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:57.295371 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-istio-podinfo" pod "87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519" (UID: "87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:34:57.394400 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:57.394370 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qqsc\" (UniqueName: \"kubernetes.io/projected/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-kube-api-access-8qqsc\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:57.394400 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:57.394396 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:57.394400 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:57.394406 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:57.394613 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:57.394414 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:57.394613 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:57.394422 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:57.511773 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:57.511745 4363 generic.go:356] "Generic (PLEG): container finished" podID="87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519" containerID="500a1c7400013ba1a920f48ce08ce87680262ace9435db82d30a8ea72c34cd9e" exitCode=0 Nov 01 16:34:57.511949 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:57.511806 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" Nov 01 16:34:57.511949 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:57.511826 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" event={"ID":"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519","Type":"ContainerDied","Data":"500a1c7400013ba1a920f48ce08ce87680262ace9435db82d30a8ea72c34cd9e"} Nov 01 16:34:57.511949 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:57.511905 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d" event={"ID":"87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519","Type":"ContainerDied","Data":"b9e93e91ad26ae21d93336d2139099f75915a19109e1bb431da6687e7efaef51"} Nov 01 16:34:57.511949 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:57.511927 4363 scope.go:122] "RemoveContainer" containerID="500a1c7400013ba1a920f48ce08ce87680262ace9435db82d30a8ea72c34cd9e" Nov 01 16:34:57.521542 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:57.521523 4363 scope.go:122] "RemoveContainer" containerID="65f99fc6091a1226f46e39229bd199f9afa4fd6033a850eedb0a5ddbaed92fea" Nov 01 16:34:57.529616 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:57.529596 4363 scope.go:122] "RemoveContainer" containerID="500a1c7400013ba1a920f48ce08ce87680262ace9435db82d30a8ea72c34cd9e" Nov 01 16:34:57.529885 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:34:57.529863 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"500a1c7400013ba1a920f48ce08ce87680262ace9435db82d30a8ea72c34cd9e\": container with ID starting with 500a1c7400013ba1a920f48ce08ce87680262ace9435db82d30a8ea72c34cd9e not found: ID does not exist" containerID="500a1c7400013ba1a920f48ce08ce87680262ace9435db82d30a8ea72c34cd9e" Nov 01 16:34:57.529962 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:57.529899 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"500a1c7400013ba1a920f48ce08ce87680262ace9435db82d30a8ea72c34cd9e"} err="failed to get container status \"500a1c7400013ba1a920f48ce08ce87680262ace9435db82d30a8ea72c34cd9e\": rpc error: code = NotFound desc = could not find container \"500a1c7400013ba1a920f48ce08ce87680262ace9435db82d30a8ea72c34cd9e\": container with ID starting with 500a1c7400013ba1a920f48ce08ce87680262ace9435db82d30a8ea72c34cd9e not found: ID does not exist" Nov 01 16:34:57.529962 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:57.529922 4363 scope.go:122] "RemoveContainer" containerID="65f99fc6091a1226f46e39229bd199f9afa4fd6033a850eedb0a5ddbaed92fea" Nov 01 16:34:57.530197 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:34:57.530180 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65f99fc6091a1226f46e39229bd199f9afa4fd6033a850eedb0a5ddbaed92fea\": container with ID starting with 65f99fc6091a1226f46e39229bd199f9afa4fd6033a850eedb0a5ddbaed92fea not found: ID does not exist" containerID="65f99fc6091a1226f46e39229bd199f9afa4fd6033a850eedb0a5ddbaed92fea" Nov 01 16:34:57.530248 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:57.530204 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65f99fc6091a1226f46e39229bd199f9afa4fd6033a850eedb0a5ddbaed92fea"} err="failed to get container status \"65f99fc6091a1226f46e39229bd199f9afa4fd6033a850eedb0a5ddbaed92fea\": rpc error: code = NotFound desc = could not find container \"65f99fc6091a1226f46e39229bd199f9afa4fd6033a850eedb0a5ddbaed92fea\": container with ID starting with 65f99fc6091a1226f46e39229bd199f9afa4fd6033a850eedb0a5ddbaed92fea not found: ID does not exist" Nov 01 16:34:57.534060 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:57.534001 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d"] Nov 01 16:34:57.537699 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:57.537680 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-f4a265-openshift-default-544bb4948f-mzs9d"] Nov 01 16:34:57.562143 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:34:57.562095 4363 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87ddacbe_42b0_42b0_8b1c_a3e6c4ed4519.slice/crio-b9e93e91ad26ae21d93336d2139099f75915a19109e1bb431da6687e7efaef51\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87ddacbe_42b0_42b0_8b1c_a3e6c4ed4519.slice\": RecentStats: unable to find data in memory cache]" Nov 01 16:34:58.796438 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:34:58.796397 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519" path="/var/lib/kubelet/pods/87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519/volumes" Nov 01 16:35:02.516925 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:02.516845 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-383fde-8486d66857-b9mwl"] Nov 01 16:35:02.534612 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:02.534582 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-383fde-6f8cc9c655-m67lx"] Nov 01 16:35:03.091197 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:03.091161 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv"] Nov 01 16:35:03.092438 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:03.091589 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv" podUID="e9235a8f-5d47-4464-a45f-6344c833c25a" containerName="maas-api" containerID="cri-o://b32697768fa5b9f216d1ad2a33127efb0fd6ff1dce0870459cb6135e7284c70c" gracePeriod=30 Nov 01 16:35:03.351501 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:03.351474 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv" Nov 01 16:35:03.356729 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:03.356703 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9235a8f-5d47-4464-a45f-6344c833c25a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9235a8f-5d47-4464-a45f-6344c833c25a-openshift-service-ca\") pod \"e9235a8f-5d47-4464-a45f-6344c833c25a\" (UID: \"e9235a8f-5d47-4464-a45f-6344c833c25a\") " Nov 01 16:35:03.356895 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:03.356747 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9235a8f-5d47-4464-a45f-6344c833c25a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9235a8f-5d47-4464-a45f-6344c833c25a-metrics-certs\") pod \"e9235a8f-5d47-4464-a45f-6344c833c25a\" (UID: \"e9235a8f-5d47-4464-a45f-6344c833c25a\") " Nov 01 16:35:03.356895 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:03.356827 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9235a8f-5d47-4464-a45f-6344c833c25a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9235a8f-5d47-4464-a45f-6344c833c25a-maas-api-tls\") pod \"e9235a8f-5d47-4464-a45f-6344c833c25a\" (UID: \"e9235a8f-5d47-4464-a45f-6344c833c25a\") " Nov 01 16:35:03.356895 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:03.356889 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9235a8f-5d47-4464-a45f-6344c833c25a-kube-api-access-r759j\" (UniqueName: \"kubernetes.io/projected/e9235a8f-5d47-4464-a45f-6344c833c25a-kube-api-access-r759j\") pod \"e9235a8f-5d47-4464-a45f-6344c833c25a\" (UID: \"e9235a8f-5d47-4464-a45f-6344c833c25a\") " Nov 01 16:35:03.357255 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:03.357212 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9235a8f-5d47-4464-a45f-6344c833c25a-openshift-service-ca" pod "e9235a8f-5d47-4464-a45f-6344c833c25a" (UID: "e9235a8f-5d47-4464-a45f-6344c833c25a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:35:03.359063 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:03.359003 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9235a8f-5d47-4464-a45f-6344c833c25a-metrics-certs" pod "e9235a8f-5d47-4464-a45f-6344c833c25a" (UID: "e9235a8f-5d47-4464-a45f-6344c833c25a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:35:03.359167 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:03.359060 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9235a8f-5d47-4464-a45f-6344c833c25a-kube-api-access-r759j" pod "e9235a8f-5d47-4464-a45f-6344c833c25a" (UID: "e9235a8f-5d47-4464-a45f-6344c833c25a"). InnerVolumeSpecName "kube-api-access-r759j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:35:03.359216 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:03.359195 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9235a8f-5d47-4464-a45f-6344c833c25a-maas-api-tls" pod "e9235a8f-5d47-4464-a45f-6344c833c25a" (UID: "e9235a8f-5d47-4464-a45f-6344c833c25a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:35:03.458129 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:03.458090 4363 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9235a8f-5d47-4464-a45f-6344c833c25a-openshift-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:03.458129 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:03.458123 4363 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9235a8f-5d47-4464-a45f-6344c833c25a-metrics-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:03.458129 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:03.458133 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9235a8f-5d47-4464-a45f-6344c833c25a-maas-api-tls\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:03.458448 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:03.458142 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r759j\" (UniqueName: \"kubernetes.io/projected/e9235a8f-5d47-4464-a45f-6344c833c25a-kube-api-access-r759j\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:03.559916 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:03.559877 4363 generic.go:356] "Generic (PLEG): container finished" podID="e9235a8f-5d47-4464-a45f-6344c833c25a" containerID="b32697768fa5b9f216d1ad2a33127efb0fd6ff1dce0870459cb6135e7284c70c" exitCode=0 Nov 01 16:35:03.560455 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:03.560108 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv" event={"ID":"e9235a8f-5d47-4464-a45f-6344c833c25a","Type":"ContainerDied","Data":"b32697768fa5b9f216d1ad2a33127efb0fd6ff1dce0870459cb6135e7284c70c"} Nov 01 16:35:03.560455 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:03.560169 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv" event={"ID":"e9235a8f-5d47-4464-a45f-6344c833c25a","Type":"ContainerDied","Data":"1e2f9e2c0d21ecfa3f8e31c6088ca6b3b4e77db63695badfb20b95d3f68c8bc1"} Nov 01 16:35:03.560455 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:03.560199 4363 scope.go:122] "RemoveContainer" containerID="b32697768fa5b9f216d1ad2a33127efb0fd6ff1dce0870459cb6135e7284c70c" Nov 01 16:35:03.560455 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:03.560357 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv" Nov 01 16:35:03.599641 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:03.595186 4363 scope.go:122] "RemoveContainer" containerID="b32697768fa5b9f216d1ad2a33127efb0fd6ff1dce0870459cb6135e7284c70c" Nov 01 16:35:03.599641 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:35:03.595667 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b32697768fa5b9f216d1ad2a33127efb0fd6ff1dce0870459cb6135e7284c70c\": container with ID starting with b32697768fa5b9f216d1ad2a33127efb0fd6ff1dce0870459cb6135e7284c70c not found: ID does not exist" containerID="b32697768fa5b9f216d1ad2a33127efb0fd6ff1dce0870459cb6135e7284c70c" Nov 01 16:35:03.599641 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:03.595709 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b32697768fa5b9f216d1ad2a33127efb0fd6ff1dce0870459cb6135e7284c70c"} err="failed to get container status \"b32697768fa5b9f216d1ad2a33127efb0fd6ff1dce0870459cb6135e7284c70c\": rpc error: code = NotFound desc = could not find container \"b32697768fa5b9f216d1ad2a33127efb0fd6ff1dce0870459cb6135e7284c70c\": container with ID starting with b32697768fa5b9f216d1ad2a33127efb0fd6ff1dce0870459cb6135e7284c70c not found: ID does not exist" Nov 01 16:35:03.599641 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:03.599090 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv"] Nov 01 16:35:03.599641 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:03.599581 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-383fde-59cf57db5b-h8znv"] Nov 01 16:35:04.802441 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:04.802404 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9235a8f-5d47-4464-a45f-6344c833c25a" path="/var/lib/kubelet/pods/e9235a8f-5d47-4464-a45f-6344c833c25a/volumes" Nov 01 16:35:07.518200 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:07.518148 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-383fde-8486d66857-b9mwl" podUID="ec15b77d-c274-4eb4-bc47-6134110a0a05" containerName="payload-processing" containerID="cri-o://f84727879a636d69b433d1c9e69e314a942a751a03949fc139bc808a056dabd6" gracePeriod=25 Nov 01 16:35:07.535401 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:07.535367 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-383fde-6f8cc9c655-m67lx" podUID="79cee252-8c24-49ac-b432-49de53bb30aa" containerName="payload-pre-processing" containerID="cri-o://8654f5f4d9f3f527465e938614cabe5546bd8eef384dd6c16c9b6811215bf12d" gracePeriod=25 Nov 01 16:35:09.607256 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:09.607219 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-383fde-6f8cc9c655-m67lx" podUID="79cee252-8c24-49ac-b432-49de53bb30aa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:35:10.426607 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:10.426561 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw"] Nov 01 16:35:10.426879 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:10.426849 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" podUID="3f975360-c532-4eb2-aae5-5233d03d027f" containerName="istio-proxy" containerID="cri-o://acce3348d5e54a50e9dda71501d6beee9c94c42c9a8b964d75ed057c8beef1a6" gracePeriod=30 Nov 01 16:35:10.429919 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:10.429878 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr"] Nov 01 16:35:10.430157 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:10.430102 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" podUID="0cc3066c-d320-41fa-8a7b-933362174b48" containerName="istio-proxy" containerID="cri-o://dd974f5e7dc1362a51e357707f821518234452bfe12c8535bdc36e95ea1c624d" gracePeriod=30 Nov 01 16:35:10.617883 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:10.617846 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-383fde-8486d66857-b9mwl" podUID="ec15b77d-c274-4eb4-bc47-6134110a0a05" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:35:15.637739 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.637705 4363 generic.go:356] "Generic (PLEG): container finished" podID="0cc3066c-d320-41fa-8a7b-933362174b48" containerID="dd974f5e7dc1362a51e357707f821518234452bfe12c8535bdc36e95ea1c624d" exitCode=0 Nov 01 16:35:15.638175 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.637762 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" event={"ID":"0cc3066c-d320-41fa-8a7b-933362174b48","Type":"ContainerDied","Data":"dd974f5e7dc1362a51e357707f821518234452bfe12c8535bdc36e95ea1c624d"} Nov 01 16:35:15.639798 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.639773 4363 generic.go:356] "Generic (PLEG): container finished" podID="3f975360-c532-4eb2-aae5-5233d03d027f" containerID="acce3348d5e54a50e9dda71501d6beee9c94c42c9a8b964d75ed057c8beef1a6" exitCode=0 Nov 01 16:35:15.639922 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.639841 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" event={"ID":"3f975360-c532-4eb2-aae5-5233d03d027f","Type":"ContainerDied","Data":"acce3348d5e54a50e9dda71501d6beee9c94c42c9a8b964d75ed057c8beef1a6"} Nov 01 16:35:15.684890 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.684868 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" Nov 01 16:35:15.709518 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.709497 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" Nov 01 16:35:15.805475 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.805446 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cc3066c-d320-41fa-8a7b-933362174b48-kube-api-access-t92vm\" (UniqueName: \"kubernetes.io/projected/0cc3066c-d320-41fa-8a7b-933362174b48-kube-api-access-t92vm\") pod \"0cc3066c-d320-41fa-8a7b-933362174b48\" (UID: \"0cc3066c-d320-41fa-8a7b-933362174b48\") " Nov 01 16:35:15.805687 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.805488 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f975360-c532-4eb2-aae5-5233d03d027f-kube-api-access-z8jwg\" (UniqueName: \"kubernetes.io/projected/3f975360-c532-4eb2-aae5-5233d03d027f-kube-api-access-z8jwg\") pod \"3f975360-c532-4eb2-aae5-5233d03d027f\" (UID: \"3f975360-c532-4eb2-aae5-5233d03d027f\") " Nov 01 16:35:15.805687 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.805516 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f975360-c532-4eb2-aae5-5233d03d027f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f975360-c532-4eb2-aae5-5233d03d027f-istiod-ca-cert\") pod \"3f975360-c532-4eb2-aae5-5233d03d027f\" (UID: \"3f975360-c532-4eb2-aae5-5233d03d027f\") " Nov 01 16:35:15.805687 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.805536 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cc3066c-d320-41fa-8a7b-933362174b48-istio-token\" (UniqueName: \"kubernetes.io/projected/0cc3066c-d320-41fa-8a7b-933362174b48-istio-token\") pod \"0cc3066c-d320-41fa-8a7b-933362174b48\" (UID: \"0cc3066c-d320-41fa-8a7b-933362174b48\") " Nov 01 16:35:15.805687 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.805591 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f975360-c532-4eb2-aae5-5233d03d027f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f975360-c532-4eb2-aae5-5233d03d027f-workload-socket\") pod \"3f975360-c532-4eb2-aae5-5233d03d027f\" (UID: \"3f975360-c532-4eb2-aae5-5233d03d027f\") " Nov 01 16:35:15.805687 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.805608 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc3066c-d320-41fa-8a7b-933362174b48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc3066c-d320-41fa-8a7b-933362174b48-credential-socket\") pod \"0cc3066c-d320-41fa-8a7b-933362174b48\" (UID: \"0cc3066c-d320-41fa-8a7b-933362174b48\") " Nov 01 16:35:15.806008 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.805786 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc3066c-d320-41fa-8a7b-933362174b48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc3066c-d320-41fa-8a7b-933362174b48-workload-socket\") pod \"0cc3066c-d320-41fa-8a7b-933362174b48\" (UID: \"0cc3066c-d320-41fa-8a7b-933362174b48\") " Nov 01 16:35:15.806008 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.805846 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cc3066c-d320-41fa-8a7b-933362174b48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc3066c-d320-41fa-8a7b-933362174b48-istiod-ca-cert\") pod \"0cc3066c-d320-41fa-8a7b-933362174b48\" (UID: \"0cc3066c-d320-41fa-8a7b-933362174b48\") " Nov 01 16:35:15.806008 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.805874 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f975360-c532-4eb2-aae5-5233d03d027f-workload-socket" pod "3f975360-c532-4eb2-aae5-5233d03d027f" (UID: "3f975360-c532-4eb2-aae5-5233d03d027f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:15.806008 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.805905 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f975360-c532-4eb2-aae5-5233d03d027f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f975360-c532-4eb2-aae5-5233d03d027f-credential-socket\") pod \"3f975360-c532-4eb2-aae5-5233d03d027f\" (UID: \"3f975360-c532-4eb2-aae5-5233d03d027f\") " Nov 01 16:35:15.806008 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.805959 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f975360-c532-4eb2-aae5-5233d03d027f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f975360-c532-4eb2-aae5-5233d03d027f-workload-certs\") pod \"3f975360-c532-4eb2-aae5-5233d03d027f\" (UID: \"3f975360-c532-4eb2-aae5-5233d03d027f\") " Nov 01 16:35:15.806431 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.806044 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f975360-c532-4eb2-aae5-5233d03d027f-istio-token\" (UniqueName: \"kubernetes.io/projected/3f975360-c532-4eb2-aae5-5233d03d027f-istio-token\") pod \"3f975360-c532-4eb2-aae5-5233d03d027f\" (UID: \"3f975360-c532-4eb2-aae5-5233d03d027f\") " Nov 01 16:35:15.806431 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.806096 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f975360-c532-4eb2-aae5-5233d03d027f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f975360-c532-4eb2-aae5-5233d03d027f-istio-podinfo\") pod \"3f975360-c532-4eb2-aae5-5233d03d027f\" (UID: \"3f975360-c532-4eb2-aae5-5233d03d027f\") " Nov 01 16:35:15.806431 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.806150 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0cc3066c-d320-41fa-8a7b-933362174b48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc3066c-d320-41fa-8a7b-933362174b48-istio-podinfo\") pod \"0cc3066c-d320-41fa-8a7b-933362174b48\" (UID: \"0cc3066c-d320-41fa-8a7b-933362174b48\") " Nov 01 16:35:15.806431 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.806194 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc3066c-d320-41fa-8a7b-933362174b48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc3066c-d320-41fa-8a7b-933362174b48-istio-data\") pod \"0cc3066c-d320-41fa-8a7b-933362174b48\" (UID: \"0cc3066c-d320-41fa-8a7b-933362174b48\") " Nov 01 16:35:15.806431 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.806227 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f975360-c532-4eb2-aae5-5233d03d027f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f975360-c532-4eb2-aae5-5233d03d027f-istio-data\") pod \"3f975360-c532-4eb2-aae5-5233d03d027f\" (UID: \"3f975360-c532-4eb2-aae5-5233d03d027f\") " Nov 01 16:35:15.806431 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.806256 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc3066c-d320-41fa-8a7b-933362174b48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc3066c-d320-41fa-8a7b-933362174b48-workload-certs\") pod \"0cc3066c-d320-41fa-8a7b-933362174b48\" (UID: \"0cc3066c-d320-41fa-8a7b-933362174b48\") " Nov 01 16:35:15.806431 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.805905 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f975360-c532-4eb2-aae5-5233d03d027f-istiod-ca-cert" pod "3f975360-c532-4eb2-aae5-5233d03d027f" (UID: "3f975360-c532-4eb2-aae5-5233d03d027f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:35:15.806431 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.806296 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f975360-c532-4eb2-aae5-5233d03d027f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f975360-c532-4eb2-aae5-5233d03d027f-istio-envoy\") pod \"3f975360-c532-4eb2-aae5-5233d03d027f\" (UID: \"3f975360-c532-4eb2-aae5-5233d03d027f\") " Nov 01 16:35:15.806431 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.806324 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc3066c-d320-41fa-8a7b-933362174b48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc3066c-d320-41fa-8a7b-933362174b48-istio-envoy\") pod \"0cc3066c-d320-41fa-8a7b-933362174b48\" (UID: \"0cc3066c-d320-41fa-8a7b-933362174b48\") " Nov 01 16:35:15.806944 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.806769 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f975360-c532-4eb2-aae5-5233d03d027f-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:15.806944 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.806794 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f975360-c532-4eb2-aae5-5233d03d027f-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:15.807085 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.806248 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f975360-c532-4eb2-aae5-5233d03d027f-credential-socket" pod "3f975360-c532-4eb2-aae5-5233d03d027f" (UID: "3f975360-c532-4eb2-aae5-5233d03d027f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:15.807085 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.806274 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cc3066c-d320-41fa-8a7b-933362174b48-istiod-ca-cert" pod "0cc3066c-d320-41fa-8a7b-933362174b48" (UID: "0cc3066c-d320-41fa-8a7b-933362174b48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:35:15.807085 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.806084 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc3066c-d320-41fa-8a7b-933362174b48-workload-socket" pod "0cc3066c-d320-41fa-8a7b-933362174b48" (UID: "0cc3066c-d320-41fa-8a7b-933362174b48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:15.807085 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.806480 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f975360-c532-4eb2-aae5-5233d03d027f-workload-certs" pod "3f975360-c532-4eb2-aae5-5233d03d027f" (UID: "3f975360-c532-4eb2-aae5-5233d03d027f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:15.807085 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.806659 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f975360-c532-4eb2-aae5-5233d03d027f-istio-data" pod "3f975360-c532-4eb2-aae5-5233d03d027f" (UID: "3f975360-c532-4eb2-aae5-5233d03d027f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:15.807404 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.807141 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc3066c-d320-41fa-8a7b-933362174b48-workload-certs" pod "0cc3066c-d320-41fa-8a7b-933362174b48" (UID: "0cc3066c-d320-41fa-8a7b-933362174b48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:15.807404 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.807234 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc3066c-d320-41fa-8a7b-933362174b48-istio-data" pod "0cc3066c-d320-41fa-8a7b-933362174b48" (UID: "0cc3066c-d320-41fa-8a7b-933362174b48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:15.807404 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.807310 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc3066c-d320-41fa-8a7b-933362174b48-credential-socket" pod "0cc3066c-d320-41fa-8a7b-933362174b48" (UID: "0cc3066c-d320-41fa-8a7b-933362174b48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:15.808288 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.808265 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc3066c-d320-41fa-8a7b-933362174b48-kube-api-access-t92vm" pod "0cc3066c-d320-41fa-8a7b-933362174b48" (UID: "0cc3066c-d320-41fa-8a7b-933362174b48"). InnerVolumeSpecName "kube-api-access-t92vm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:35:15.808769 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.808743 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f975360-c532-4eb2-aae5-5233d03d027f-istio-podinfo" pod "3f975360-c532-4eb2-aae5-5233d03d027f" (UID: "3f975360-c532-4eb2-aae5-5233d03d027f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:35:15.808890 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.808825 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0cc3066c-d320-41fa-8a7b-933362174b48-istio-podinfo" pod "0cc3066c-d320-41fa-8a7b-933362174b48" (UID: "0cc3066c-d320-41fa-8a7b-933362174b48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:35:15.808972 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.808954 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc3066c-d320-41fa-8a7b-933362174b48-istio-token" pod "0cc3066c-d320-41fa-8a7b-933362174b48" (UID: "0cc3066c-d320-41fa-8a7b-933362174b48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:35:15.809745 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.809710 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc3066c-d320-41fa-8a7b-933362174b48-istio-envoy" pod "0cc3066c-d320-41fa-8a7b-933362174b48" (UID: "0cc3066c-d320-41fa-8a7b-933362174b48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:15.809862 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.809819 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f975360-c532-4eb2-aae5-5233d03d027f-kube-api-access-z8jwg" pod "3f975360-c532-4eb2-aae5-5233d03d027f" (UID: "3f975360-c532-4eb2-aae5-5233d03d027f"). InnerVolumeSpecName "kube-api-access-z8jwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:35:15.810084 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.810062 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f975360-c532-4eb2-aae5-5233d03d027f-istio-envoy" pod "3f975360-c532-4eb2-aae5-5233d03d027f" (UID: "3f975360-c532-4eb2-aae5-5233d03d027f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:15.810386 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.810366 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f975360-c532-4eb2-aae5-5233d03d027f-istio-token" pod "3f975360-c532-4eb2-aae5-5233d03d027f" (UID: "3f975360-c532-4eb2-aae5-5233d03d027f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:35:15.907685 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.907656 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f975360-c532-4eb2-aae5-5233d03d027f-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:15.907685 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.907685 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f975360-c532-4eb2-aae5-5233d03d027f-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:15.907685 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.907702 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc3066c-d320-41fa-8a7b-933362174b48-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:15.907685 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.907716 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc3066c-d320-41fa-8a7b-933362174b48-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:15.908053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.907730 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f975360-c532-4eb2-aae5-5233d03d027f-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:15.908053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.907742 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc3066c-d320-41fa-8a7b-933362174b48-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:15.908053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.907754 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f975360-c532-4eb2-aae5-5233d03d027f-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:15.908053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.907766 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc3066c-d320-41fa-8a7b-933362174b48-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:15.908053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.907779 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t92vm\" (UniqueName: \"kubernetes.io/projected/0cc3066c-d320-41fa-8a7b-933362174b48-kube-api-access-t92vm\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:15.908053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.907795 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8jwg\" (UniqueName: \"kubernetes.io/projected/3f975360-c532-4eb2-aae5-5233d03d027f-kube-api-access-z8jwg\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:15.908053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.907808 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc3066c-d320-41fa-8a7b-933362174b48-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:15.908053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.907821 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc3066c-d320-41fa-8a7b-933362174b48-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:15.908053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.907833 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc3066c-d320-41fa-8a7b-933362174b48-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:15.908053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.907845 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc3066c-d320-41fa-8a7b-933362174b48-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:15.908053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.907858 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f975360-c532-4eb2-aae5-5233d03d027f-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:15.908053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:15.907870 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f975360-c532-4eb2-aae5-5233d03d027f-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:16.647459 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:16.647422 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" Nov 01 16:35:16.647459 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:16.647432 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr" event={"ID":"0cc3066c-d320-41fa-8a7b-933362174b48","Type":"ContainerDied","Data":"4265fdc03efacd8194a0b5275e811e0d6b31bc3f4a37b9272afc8496de844eb6"} Nov 01 16:35:16.648005 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:16.647494 4363 scope.go:122] "RemoveContainer" containerID="dd974f5e7dc1362a51e357707f821518234452bfe12c8535bdc36e95ea1c624d" Nov 01 16:35:16.648973 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:16.648948 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" event={"ID":"3f975360-c532-4eb2-aae5-5233d03d027f","Type":"ContainerDied","Data":"42f6204571f554fb52cb54bf96f98512d146bef69c3961304f54dd6088800b08"} Nov 01 16:35:16.649089 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:16.648961 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw" Nov 01 16:35:16.660153 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:16.659694 4363 scope.go:122] "RemoveContainer" containerID="acce3348d5e54a50e9dda71501d6beee9c94c42c9a8b964d75ed057c8beef1a6" Nov 01 16:35:16.676132 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:16.676109 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw"] Nov 01 16:35:16.682202 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:16.682177 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-gn6sw"] Nov 01 16:35:16.691900 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:16.691875 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr"] Nov 01 16:35:16.695671 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:16.695641 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-hxkzr"] Nov 01 16:35:16.796260 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:16.796197 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cc3066c-d320-41fa-8a7b-933362174b48" path="/var/lib/kubelet/pods/0cc3066c-d320-41fa-8a7b-933362174b48/volumes" Nov 01 16:35:16.796631 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:16.796611 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f975360-c532-4eb2-aae5-5233d03d027f" path="/var/lib/kubelet/pods/3f975360-c532-4eb2-aae5-5233d03d027f/volumes" Nov 01 16:35:18.308707 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:35:18.308670 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:35:19.606867 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:19.606820 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-383fde-6f8cc9c655-m67lx" podUID="79cee252-8c24-49ac-b432-49de53bb30aa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:35:20.619434 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:20.619307 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-383fde-8486d66857-b9mwl" podUID="ec15b77d-c274-4eb4-bc47-6134110a0a05" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:35:20.992632 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:20.992545 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a9a967-6948b69c67-tgrtj"] Nov 01 16:35:25.993717 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:25.993643 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a9a967-6948b69c67-tgrtj" podUID="de4bccaf-0cc6-4d9a-8b5c-800217cff553" containerName="payload-pre-processing" containerID="cri-o://646cf3813c38dae5d27f6ea50c5de77e494d6e91c9e92e741f0d679dd66a43fb" gracePeriod=25 Nov 01 16:35:27.560609 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:35:27.560578 4363 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec15b77d_c274_4eb4_bc47_6134110a0a05.slice/crio-f84727879a636d69b433d1c9e69e314a942a751a03949fc139bc808a056dabd6.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec15b77d_c274_4eb4_bc47_6134110a0a05.slice/crio-conmon-f84727879a636d69b433d1c9e69e314a942a751a03949fc139bc808a056dabd6.scope\": RecentStats: unable to find data in memory cache]" Nov 01 16:35:27.715926 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:27.715889 4363 generic.go:356] "Generic (PLEG): container finished" podID="79cee252-8c24-49ac-b432-49de53bb30aa" containerID="8654f5f4d9f3f527465e938614cabe5546bd8eef384dd6c16c9b6811215bf12d" exitCode=0 Nov 01 16:35:27.716352 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:27.716056 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-383fde-6f8cc9c655-m67lx" event={"ID":"79cee252-8c24-49ac-b432-49de53bb30aa","Type":"ContainerDied","Data":"8654f5f4d9f3f527465e938614cabe5546bd8eef384dd6c16c9b6811215bf12d"} Nov 01 16:35:27.719071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:27.718884 4363 generic.go:356] "Generic (PLEG): container finished" podID="ec15b77d-c274-4eb4-bc47-6134110a0a05" containerID="f84727879a636d69b433d1c9e69e314a942a751a03949fc139bc808a056dabd6" exitCode=0 Nov 01 16:35:27.719071 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:27.719000 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-383fde-8486d66857-b9mwl" event={"ID":"ec15b77d-c274-4eb4-bc47-6134110a0a05","Type":"ContainerDied","Data":"f84727879a636d69b433d1c9e69e314a942a751a03949fc139bc808a056dabd6"} Nov 01 16:35:27.790842 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:27.790777 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-383fde-8486d66857-b9mwl" Nov 01 16:35:27.854261 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:27.854227 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-383fde-6f8cc9c655-m67lx" Nov 01 16:35:27.935034 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:27.934997 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79cee252-8c24-49ac-b432-49de53bb30aa-kube-api-access-wm5k4\" (UniqueName: \"kubernetes.io/projected/79cee252-8c24-49ac-b432-49de53bb30aa-kube-api-access-wm5k4\") pod \"79cee252-8c24-49ac-b432-49de53bb30aa\" (UID: \"79cee252-8c24-49ac-b432-49de53bb30aa\") " Nov 01 16:35:27.935192 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:27.935074 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec15b77d-c274-4eb4-bc47-6134110a0a05-kube-api-access-vs4sm\" (UniqueName: \"kubernetes.io/projected/ec15b77d-c274-4eb4-bc47-6134110a0a05-kube-api-access-vs4sm\") pod \"ec15b77d-c274-4eb4-bc47-6134110a0a05\" (UID: \"ec15b77d-c274-4eb4-bc47-6134110a0a05\") " Nov 01 16:35:27.935192 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:27.935097 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79cee252-8c24-49ac-b432-49de53bb30aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79cee252-8c24-49ac-b432-49de53bb30aa-plugins-config-volume\") pod \"79cee252-8c24-49ac-b432-49de53bb30aa\" (UID: \"79cee252-8c24-49ac-b432-49de53bb30aa\") " Nov 01 16:35:27.935339 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:27.935203 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec15b77d-c274-4eb4-bc47-6134110a0a05-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec15b77d-c274-4eb4-bc47-6134110a0a05-plugins-config-volume\") pod \"ec15b77d-c274-4eb4-bc47-6134110a0a05\" (UID: \"ec15b77d-c274-4eb4-bc47-6134110a0a05\") " Nov 01 16:35:27.935581 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:27.935555 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79cee252-8c24-49ac-b432-49de53bb30aa-plugins-config-volume" pod "79cee252-8c24-49ac-b432-49de53bb30aa" (UID: "79cee252-8c24-49ac-b432-49de53bb30aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:35:27.935692 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:27.935655 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec15b77d-c274-4eb4-bc47-6134110a0a05-plugins-config-volume" pod "ec15b77d-c274-4eb4-bc47-6134110a0a05" (UID: "ec15b77d-c274-4eb4-bc47-6134110a0a05"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:35:27.937225 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:27.937205 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec15b77d-c274-4eb4-bc47-6134110a0a05-kube-api-access-vs4sm" pod "ec15b77d-c274-4eb4-bc47-6134110a0a05" (UID: "ec15b77d-c274-4eb4-bc47-6134110a0a05"). InnerVolumeSpecName "kube-api-access-vs4sm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:35:27.937307 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:27.937237 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79cee252-8c24-49ac-b432-49de53bb30aa-kube-api-access-wm5k4" pod "79cee252-8c24-49ac-b432-49de53bb30aa" (UID: "79cee252-8c24-49ac-b432-49de53bb30aa"). InnerVolumeSpecName "kube-api-access-wm5k4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:35:28.036494 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:28.036465 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec15b77d-c274-4eb4-bc47-6134110a0a05-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:28.036494 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:28.036491 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm5k4\" (UniqueName: \"kubernetes.io/projected/79cee252-8c24-49ac-b432-49de53bb30aa-kube-api-access-wm5k4\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:28.036494 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:28.036501 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs4sm\" (UniqueName: \"kubernetes.io/projected/ec15b77d-c274-4eb4-bc47-6134110a0a05-kube-api-access-vs4sm\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:28.036692 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:28.036509 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79cee252-8c24-49ac-b432-49de53bb30aa-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:28.725844 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:28.725810 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-383fde-6f8cc9c655-m67lx" Nov 01 16:35:28.726358 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:28.725842 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-383fde-6f8cc9c655-m67lx" event={"ID":"79cee252-8c24-49ac-b432-49de53bb30aa","Type":"ContainerDied","Data":"1655ee3d8e082c171c872b12f7f520e77feb87496f618f68f4ef6a31cdee76fa"} Nov 01 16:35:28.726358 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:28.725913 4363 scope.go:122] "RemoveContainer" containerID="8654f5f4d9f3f527465e938614cabe5546bd8eef384dd6c16c9b6811215bf12d" Nov 01 16:35:28.727507 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:28.727489 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-383fde-8486d66857-b9mwl" Nov 01 16:35:28.727600 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:28.727525 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-383fde-8486d66857-b9mwl" event={"ID":"ec15b77d-c274-4eb4-bc47-6134110a0a05","Type":"ContainerDied","Data":"c1869c041c031f18fa6bcab78beea6d07d3c77bbc7c2bd0d4ac7bec911a9908c"} Nov 01 16:35:28.736691 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:28.736674 4363 scope.go:122] "RemoveContainer" containerID="f84727879a636d69b433d1c9e69e314a942a751a03949fc139bc808a056dabd6" Nov 01 16:35:28.755427 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:28.755401 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-383fde-8486d66857-b9mwl"] Nov 01 16:35:28.759809 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:28.759788 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-383fde-8486d66857-b9mwl"] Nov 01 16:35:28.771380 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:28.771357 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-383fde-6f8cc9c655-m67lx"] Nov 01 16:35:28.778108 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:28.778085 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-383fde-6f8cc9c655-m67lx"] Nov 01 16:35:28.797632 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:28.797567 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79cee252-8c24-49ac-b432-49de53bb30aa" path="/var/lib/kubelet/pods/79cee252-8c24-49ac-b432-49de53bb30aa/volumes" Nov 01 16:35:28.798050 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:28.798031 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec15b77d-c274-4eb4-bc47-6134110a0a05" path="/var/lib/kubelet/pods/ec15b77d-c274-4eb4-bc47-6134110a0a05/volumes" Nov 01 16:35:29.749361 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:29.749326 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p"] Nov 01 16:35:29.749915 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:29.749532 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" podUID="c176da5b-fd83-454e-bbb7-bb793807cb67" containerName="istio-proxy" containerID="cri-o://b001a90e25bfaed41da951cfe9477775e6ef5e21487f02cb3e23de8bd91803a1" gracePeriod=30 Nov 01 16:35:29.752009 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:29.751978 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72"] Nov 01 16:35:29.752216 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:29.752190 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" podUID="a27bb1f1-cb87-4003-862c-4839331f5c9e" containerName="istio-proxy" containerID="cri-o://052cf4da2e63f6ac8ee0ff54803cec24498a47a85e98dbb4c86e6417ac97a930" gracePeriod=30 Nov 01 16:35:34.527061 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:34.526991 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a9a967-6948b69c67-tgrtj" podUID="de4bccaf-0cc6-4d9a-8b5c-800217cff553" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:35:35.016958 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.016934 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" Nov 01 16:35:35.049355 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.049323 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" Nov 01 16:35:35.105714 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.105679 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c176da5b-fd83-454e-bbb7-bb793807cb67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c176da5b-fd83-454e-bbb7-bb793807cb67-istio-podinfo\") pod \"c176da5b-fd83-454e-bbb7-bb793807cb67\" (UID: \"c176da5b-fd83-454e-bbb7-bb793807cb67\") " Nov 01 16:35:35.105878 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.105770 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c176da5b-fd83-454e-bbb7-bb793807cb67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c176da5b-fd83-454e-bbb7-bb793807cb67-workload-socket\") pod \"c176da5b-fd83-454e-bbb7-bb793807cb67\" (UID: \"c176da5b-fd83-454e-bbb7-bb793807cb67\") " Nov 01 16:35:35.105878 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.105827 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c176da5b-fd83-454e-bbb7-bb793807cb67-kube-api-access-fx8dc\" (UniqueName: \"kubernetes.io/projected/c176da5b-fd83-454e-bbb7-bb793807cb67-kube-api-access-fx8dc\") pod \"c176da5b-fd83-454e-bbb7-bb793807cb67\" (UID: \"c176da5b-fd83-454e-bbb7-bb793807cb67\") " Nov 01 16:35:35.105878 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.105853 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c176da5b-fd83-454e-bbb7-bb793807cb67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c176da5b-fd83-454e-bbb7-bb793807cb67-credential-socket\") pod \"c176da5b-fd83-454e-bbb7-bb793807cb67\" (UID: \"c176da5b-fd83-454e-bbb7-bb793807cb67\") " Nov 01 16:35:35.105878 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.105874 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c176da5b-fd83-454e-bbb7-bb793807cb67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c176da5b-fd83-454e-bbb7-bb793807cb67-workload-certs\") pod \"c176da5b-fd83-454e-bbb7-bb793807cb67\" (UID: \"c176da5b-fd83-454e-bbb7-bb793807cb67\") " Nov 01 16:35:35.106133 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.105906 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c176da5b-fd83-454e-bbb7-bb793807cb67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c176da5b-fd83-454e-bbb7-bb793807cb67-istio-data\") pod \"c176da5b-fd83-454e-bbb7-bb793807cb67\" (UID: \"c176da5b-fd83-454e-bbb7-bb793807cb67\") " Nov 01 16:35:35.106133 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.105922 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c176da5b-fd83-454e-bbb7-bb793807cb67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c176da5b-fd83-454e-bbb7-bb793807cb67-istio-envoy\") pod \"c176da5b-fd83-454e-bbb7-bb793807cb67\" (UID: \"c176da5b-fd83-454e-bbb7-bb793807cb67\") " Nov 01 16:35:35.106133 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.105971 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c176da5b-fd83-454e-bbb7-bb793807cb67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c176da5b-fd83-454e-bbb7-bb793807cb67-istiod-ca-cert\") pod \"c176da5b-fd83-454e-bbb7-bb793807cb67\" (UID: \"c176da5b-fd83-454e-bbb7-bb793807cb67\") " Nov 01 16:35:35.106133 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.106012 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c176da5b-fd83-454e-bbb7-bb793807cb67-istio-token\" (UniqueName: \"kubernetes.io/projected/c176da5b-fd83-454e-bbb7-bb793807cb67-istio-token\") pod \"c176da5b-fd83-454e-bbb7-bb793807cb67\" (UID: \"c176da5b-fd83-454e-bbb7-bb793807cb67\") " Nov 01 16:35:35.106133 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.106090 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c176da5b-fd83-454e-bbb7-bb793807cb67-workload-socket" pod "c176da5b-fd83-454e-bbb7-bb793807cb67" (UID: "c176da5b-fd83-454e-bbb7-bb793807cb67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:35.106403 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.106234 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c176da5b-fd83-454e-bbb7-bb793807cb67-credential-socket" pod "c176da5b-fd83-454e-bbb7-bb793807cb67" (UID: "c176da5b-fd83-454e-bbb7-bb793807cb67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:35.106403 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.106246 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c176da5b-fd83-454e-bbb7-bb793807cb67-workload-certs" pod "c176da5b-fd83-454e-bbb7-bb793807cb67" (UID: "c176da5b-fd83-454e-bbb7-bb793807cb67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:35.106403 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.106317 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c176da5b-fd83-454e-bbb7-bb793807cb67-istio-data" pod "c176da5b-fd83-454e-bbb7-bb793807cb67" (UID: "c176da5b-fd83-454e-bbb7-bb793807cb67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:35.106589 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.106461 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c176da5b-fd83-454e-bbb7-bb793807cb67-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.106589 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.106480 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c176da5b-fd83-454e-bbb7-bb793807cb67-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.106589 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.106490 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c176da5b-fd83-454e-bbb7-bb793807cb67-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.106589 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.106498 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c176da5b-fd83-454e-bbb7-bb793807cb67-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.106855 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.106831 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c176da5b-fd83-454e-bbb7-bb793807cb67-istiod-ca-cert" pod "c176da5b-fd83-454e-bbb7-bb793807cb67" (UID: "c176da5b-fd83-454e-bbb7-bb793807cb67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:35:35.107961 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.107937 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c176da5b-fd83-454e-bbb7-bb793807cb67-istio-podinfo" pod "c176da5b-fd83-454e-bbb7-bb793807cb67" (UID: "c176da5b-fd83-454e-bbb7-bb793807cb67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:35:35.108188 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.108169 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c176da5b-fd83-454e-bbb7-bb793807cb67-istio-token" pod "c176da5b-fd83-454e-bbb7-bb793807cb67" (UID: "c176da5b-fd83-454e-bbb7-bb793807cb67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:35:35.108264 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.108209 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c176da5b-fd83-454e-bbb7-bb793807cb67-istio-envoy" pod "c176da5b-fd83-454e-bbb7-bb793807cb67" (UID: "c176da5b-fd83-454e-bbb7-bb793807cb67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:35.108418 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.108398 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c176da5b-fd83-454e-bbb7-bb793807cb67-kube-api-access-fx8dc" pod "c176da5b-fd83-454e-bbb7-bb793807cb67" (UID: "c176da5b-fd83-454e-bbb7-bb793807cb67"). InnerVolumeSpecName "kube-api-access-fx8dc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:35:35.202171 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.202140 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx"] Nov 01 16:35:35.207513 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.207485 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a27bb1f1-cb87-4003-862c-4839331f5c9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a27bb1f1-cb87-4003-862c-4839331f5c9e-istio-podinfo\") pod \"a27bb1f1-cb87-4003-862c-4839331f5c9e\" (UID: \"a27bb1f1-cb87-4003-862c-4839331f5c9e\") " Nov 01 16:35:35.207688 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.207549 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a27bb1f1-cb87-4003-862c-4839331f5c9e-kube-api-access-94cgp\" (UniqueName: \"kubernetes.io/projected/a27bb1f1-cb87-4003-862c-4839331f5c9e-kube-api-access-94cgp\") pod \"a27bb1f1-cb87-4003-862c-4839331f5c9e\" (UID: \"a27bb1f1-cb87-4003-862c-4839331f5c9e\") " Nov 01 16:35:35.207688 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.207576 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a27bb1f1-cb87-4003-862c-4839331f5c9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a27bb1f1-cb87-4003-862c-4839331f5c9e-istiod-ca-cert\") pod \"a27bb1f1-cb87-4003-862c-4839331f5c9e\" (UID: \"a27bb1f1-cb87-4003-862c-4839331f5c9e\") " Nov 01 16:35:35.207817 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.207678 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a27bb1f1-cb87-4003-862c-4839331f5c9e-istio-token\" (UniqueName: \"kubernetes.io/projected/a27bb1f1-cb87-4003-862c-4839331f5c9e-istio-token\") pod \"a27bb1f1-cb87-4003-862c-4839331f5c9e\" (UID: \"a27bb1f1-cb87-4003-862c-4839331f5c9e\") " Nov 01 16:35:35.207817 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.207732 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a27bb1f1-cb87-4003-862c-4839331f5c9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a27bb1f1-cb87-4003-862c-4839331f5c9e-istio-data\") pod \"a27bb1f1-cb87-4003-862c-4839331f5c9e\" (UID: \"a27bb1f1-cb87-4003-862c-4839331f5c9e\") " Nov 01 16:35:35.207817 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.207768 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a27bb1f1-cb87-4003-862c-4839331f5c9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27bb1f1-cb87-4003-862c-4839331f5c9e-credential-socket\") pod \"a27bb1f1-cb87-4003-862c-4839331f5c9e\" (UID: \"a27bb1f1-cb87-4003-862c-4839331f5c9e\") " Nov 01 16:35:35.207817 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.207804 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a27bb1f1-cb87-4003-862c-4839331f5c9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a27bb1f1-cb87-4003-862c-4839331f5c9e-istio-envoy\") pod \"a27bb1f1-cb87-4003-862c-4839331f5c9e\" (UID: \"a27bb1f1-cb87-4003-862c-4839331f5c9e\") " Nov 01 16:35:35.208010 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.207846 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a27bb1f1-cb87-4003-862c-4839331f5c9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27bb1f1-cb87-4003-862c-4839331f5c9e-workload-socket\") pod \"a27bb1f1-cb87-4003-862c-4839331f5c9e\" (UID: \"a27bb1f1-cb87-4003-862c-4839331f5c9e\") " Nov 01 16:35:35.208010 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.207964 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a27bb1f1-cb87-4003-862c-4839331f5c9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a27bb1f1-cb87-4003-862c-4839331f5c9e-workload-certs\") pod \"a27bb1f1-cb87-4003-862c-4839331f5c9e\" (UID: \"a27bb1f1-cb87-4003-862c-4839331f5c9e\") " Nov 01 16:35:35.209265 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.207960 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a27bb1f1-cb87-4003-862c-4839331f5c9e-istiod-ca-cert" pod "a27bb1f1-cb87-4003-862c-4839331f5c9e" (UID: "a27bb1f1-cb87-4003-862c-4839331f5c9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:35:35.209265 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.208130 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a27bb1f1-cb87-4003-862c-4839331f5c9e-credential-socket" pod "a27bb1f1-cb87-4003-862c-4839331f5c9e" (UID: "a27bb1f1-cb87-4003-862c-4839331f5c9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:35.209265 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.208331 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a27bb1f1-cb87-4003-862c-4839331f5c9e-istio-data" pod "a27bb1f1-cb87-4003-862c-4839331f5c9e" (UID: "a27bb1f1-cb87-4003-862c-4839331f5c9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:35.209265 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.208754 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a27bb1f1-cb87-4003-862c-4839331f5c9e-workload-certs" pod "a27bb1f1-cb87-4003-862c-4839331f5c9e" (UID: "a27bb1f1-cb87-4003-862c-4839331f5c9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:35.209265 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.208894 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a27bb1f1-cb87-4003-862c-4839331f5c9e-workload-socket" pod "a27bb1f1-cb87-4003-862c-4839331f5c9e" (UID: "a27bb1f1-cb87-4003-862c-4839331f5c9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:35.209729 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.209704 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a27bb1f1-cb87-4003-862c-4839331f5c9e-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.209810 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.209734 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a27bb1f1-cb87-4003-862c-4839331f5c9e-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.209810 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.209749 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27bb1f1-cb87-4003-862c-4839331f5c9e-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.209810 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.209766 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx8dc\" (UniqueName: \"kubernetes.io/projected/c176da5b-fd83-454e-bbb7-bb793807cb67-kube-api-access-fx8dc\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.209810 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.209782 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27bb1f1-cb87-4003-862c-4839331f5c9e-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.209810 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.209796 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c176da5b-fd83-454e-bbb7-bb793807cb67-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.209810 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.209809 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c176da5b-fd83-454e-bbb7-bb793807cb67-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.210230 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.209823 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c176da5b-fd83-454e-bbb7-bb793807cb67-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.210230 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.209836 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a27bb1f1-cb87-4003-862c-4839331f5c9e-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.210230 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.209852 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c176da5b-fd83-454e-bbb7-bb793807cb67-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.210411 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.210235 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a27bb1f1-cb87-4003-862c-4839331f5c9e-istio-token" pod "a27bb1f1-cb87-4003-862c-4839331f5c9e" (UID: "a27bb1f1-cb87-4003-862c-4839331f5c9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:35:35.210544 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.210523 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a27bb1f1-cb87-4003-862c-4839331f5c9e-istio-envoy" pod "a27bb1f1-cb87-4003-862c-4839331f5c9e" (UID: "a27bb1f1-cb87-4003-862c-4839331f5c9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:35.211111 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.211090 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a27bb1f1-cb87-4003-862c-4839331f5c9e-kube-api-access-94cgp" pod "a27bb1f1-cb87-4003-862c-4839331f5c9e" (UID: "a27bb1f1-cb87-4003-862c-4839331f5c9e"). InnerVolumeSpecName "kube-api-access-94cgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:35:35.213367 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.213345 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a27bb1f1-cb87-4003-862c-4839331f5c9e-istio-podinfo" pod "a27bb1f1-cb87-4003-862c-4839331f5c9e" (UID: "a27bb1f1-cb87-4003-862c-4839331f5c9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:35:35.311403 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.311316 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a27bb1f1-cb87-4003-862c-4839331f5c9e-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.311403 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.311347 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a27bb1f1-cb87-4003-862c-4839331f5c9e-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.311403 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.311361 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a27bb1f1-cb87-4003-862c-4839331f5c9e-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.311403 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.311375 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94cgp\" (UniqueName: \"kubernetes.io/projected/a27bb1f1-cb87-4003-862c-4839331f5c9e-kube-api-access-94cgp\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.783219 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.783183 4363 generic.go:356] "Generic (PLEG): container finished" podID="c176da5b-fd83-454e-bbb7-bb793807cb67" containerID="b001a90e25bfaed41da951cfe9477775e6ef5e21487f02cb3e23de8bd91803a1" exitCode=0 Nov 01 16:35:35.783645 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.783266 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" Nov 01 16:35:35.783645 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.783278 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" event={"ID":"c176da5b-fd83-454e-bbb7-bb793807cb67","Type":"ContainerDied","Data":"b001a90e25bfaed41da951cfe9477775e6ef5e21487f02cb3e23de8bd91803a1"} Nov 01 16:35:35.783645 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.783360 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p" event={"ID":"c176da5b-fd83-454e-bbb7-bb793807cb67","Type":"ContainerDied","Data":"1158919e402f00a9784f37e3298d3e33680cbc1b2753946e4c83030b28046cb4"} Nov 01 16:35:35.783645 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.783392 4363 scope.go:122] "RemoveContainer" containerID="b001a90e25bfaed41da951cfe9477775e6ef5e21487f02cb3e23de8bd91803a1" Nov 01 16:35:35.784695 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.784676 4363 generic.go:356] "Generic (PLEG): container finished" podID="a27bb1f1-cb87-4003-862c-4839331f5c9e" containerID="052cf4da2e63f6ac8ee0ff54803cec24498a47a85e98dbb4c86e6417ac97a930" exitCode=0 Nov 01 16:35:35.784768 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.784722 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" event={"ID":"a27bb1f1-cb87-4003-862c-4839331f5c9e","Type":"ContainerDied","Data":"052cf4da2e63f6ac8ee0ff54803cec24498a47a85e98dbb4c86e6417ac97a930"} Nov 01 16:35:35.784768 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.784751 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" event={"ID":"a27bb1f1-cb87-4003-862c-4839331f5c9e","Type":"ContainerDied","Data":"e341d8071458ade3c1fd32c03ba1121b15a1de65849fa47f7bf5a06385777a06"} Nov 01 16:35:35.784850 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.784808 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72" Nov 01 16:35:35.794102 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.794084 4363 scope.go:122] "RemoveContainer" containerID="b001a90e25bfaed41da951cfe9477775e6ef5e21487f02cb3e23de8bd91803a1" Nov 01 16:35:35.794355 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:35:35.794337 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b001a90e25bfaed41da951cfe9477775e6ef5e21487f02cb3e23de8bd91803a1\": container with ID starting with b001a90e25bfaed41da951cfe9477775e6ef5e21487f02cb3e23de8bd91803a1 not found: ID does not exist" containerID="b001a90e25bfaed41da951cfe9477775e6ef5e21487f02cb3e23de8bd91803a1" Nov 01 16:35:35.794440 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.794361 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b001a90e25bfaed41da951cfe9477775e6ef5e21487f02cb3e23de8bd91803a1"} err="failed to get container status \"b001a90e25bfaed41da951cfe9477775e6ef5e21487f02cb3e23de8bd91803a1\": rpc error: code = NotFound desc = could not find container \"b001a90e25bfaed41da951cfe9477775e6ef5e21487f02cb3e23de8bd91803a1\": container with ID starting with b001a90e25bfaed41da951cfe9477775e6ef5e21487f02cb3e23de8bd91803a1 not found: ID does not exist" Nov 01 16:35:35.794440 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.794377 4363 scope.go:122] "RemoveContainer" containerID="052cf4da2e63f6ac8ee0ff54803cec24498a47a85e98dbb4c86e6417ac97a930" Nov 01 16:35:35.802714 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.802699 4363 scope.go:122] "RemoveContainer" containerID="052cf4da2e63f6ac8ee0ff54803cec24498a47a85e98dbb4c86e6417ac97a930" Nov 01 16:35:35.802941 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:35:35.802922 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"052cf4da2e63f6ac8ee0ff54803cec24498a47a85e98dbb4c86e6417ac97a930\": container with ID starting with 052cf4da2e63f6ac8ee0ff54803cec24498a47a85e98dbb4c86e6417ac97a930 not found: ID does not exist" containerID="052cf4da2e63f6ac8ee0ff54803cec24498a47a85e98dbb4c86e6417ac97a930" Nov 01 16:35:35.803007 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.802949 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"052cf4da2e63f6ac8ee0ff54803cec24498a47a85e98dbb4c86e6417ac97a930"} err="failed to get container status \"052cf4da2e63f6ac8ee0ff54803cec24498a47a85e98dbb4c86e6417ac97a930\": rpc error: code = NotFound desc = could not find container \"052cf4da2e63f6ac8ee0ff54803cec24498a47a85e98dbb4c86e6417ac97a930\": container with ID starting with 052cf4da2e63f6ac8ee0ff54803cec24498a47a85e98dbb4c86e6417ac97a930 not found: ID does not exist" Nov 01 16:35:35.808480 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.808456 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p"] Nov 01 16:35:35.814079 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.814058 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-f4d8p"] Nov 01 16:35:35.823151 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.823127 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72"] Nov 01 16:35:35.828655 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:35.828636 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-55b72"] Nov 01 16:35:36.795975 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:36.795946 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a27bb1f1-cb87-4003-862c-4839331f5c9e" path="/var/lib/kubelet/pods/a27bb1f1-cb87-4003-862c-4839331f5c9e/volumes" Nov 01 16:35:36.796420 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:36.796405 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c176da5b-fd83-454e-bbb7-bb793807cb67" path="/var/lib/kubelet/pods/c176da5b-fd83-454e-bbb7-bb793807cb67/volumes" Nov 01 16:35:44.526967 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:44.526926 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a9a967-6948b69c67-tgrtj" podUID="de4bccaf-0cc6-4d9a-8b5c-800217cff553" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:35:46.245918 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:46.245890 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a9a967-6948b69c67-tgrtj" Nov 01 16:35:46.316041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:46.315926 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de4bccaf-0cc6-4d9a-8b5c-800217cff553-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de4bccaf-0cc6-4d9a-8b5c-800217cff553-plugins-config-volume\") pod \"de4bccaf-0cc6-4d9a-8b5c-800217cff553\" (UID: \"de4bccaf-0cc6-4d9a-8b5c-800217cff553\") " Nov 01 16:35:46.316041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:46.316012 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de4bccaf-0cc6-4d9a-8b5c-800217cff553-kube-api-access-mwdhg\" (UniqueName: \"kubernetes.io/projected/de4bccaf-0cc6-4d9a-8b5c-800217cff553-kube-api-access-mwdhg\") pod \"de4bccaf-0cc6-4d9a-8b5c-800217cff553\" (UID: \"de4bccaf-0cc6-4d9a-8b5c-800217cff553\") " Nov 01 16:35:46.316361 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:46.316337 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de4bccaf-0cc6-4d9a-8b5c-800217cff553-plugins-config-volume" pod "de4bccaf-0cc6-4d9a-8b5c-800217cff553" (UID: "de4bccaf-0cc6-4d9a-8b5c-800217cff553"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:35:46.318156 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:46.318134 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de4bccaf-0cc6-4d9a-8b5c-800217cff553-kube-api-access-mwdhg" pod "de4bccaf-0cc6-4d9a-8b5c-800217cff553" (UID: "de4bccaf-0cc6-4d9a-8b5c-800217cff553"). InnerVolumeSpecName "kube-api-access-mwdhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:35:46.417512 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:46.417479 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de4bccaf-0cc6-4d9a-8b5c-800217cff553-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:46.417512 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:46.417509 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwdhg\" (UniqueName: \"kubernetes.io/projected/de4bccaf-0cc6-4d9a-8b5c-800217cff553-kube-api-access-mwdhg\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:46.845892 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:46.845859 4363 generic.go:356] "Generic (PLEG): container finished" podID="de4bccaf-0cc6-4d9a-8b5c-800217cff553" containerID="646cf3813c38dae5d27f6ea50c5de77e494d6e91c9e92e741f0d679dd66a43fb" exitCode=0 Nov 01 16:35:46.846070 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:46.845922 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a9a967-6948b69c67-tgrtj" Nov 01 16:35:46.846070 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:46.845941 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a9a967-6948b69c67-tgrtj" event={"ID":"de4bccaf-0cc6-4d9a-8b5c-800217cff553","Type":"ContainerDied","Data":"646cf3813c38dae5d27f6ea50c5de77e494d6e91c9e92e741f0d679dd66a43fb"} Nov 01 16:35:46.846070 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:46.845990 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a9a967-6948b69c67-tgrtj" event={"ID":"de4bccaf-0cc6-4d9a-8b5c-800217cff553","Type":"ContainerDied","Data":"e768adca3aeb7806969a46b8c5d26cd275ef5116406a68900bd5b1ec034df435"} Nov 01 16:35:46.846070 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:46.846011 4363 scope.go:122] "RemoveContainer" containerID="646cf3813c38dae5d27f6ea50c5de77e494d6e91c9e92e741f0d679dd66a43fb" Nov 01 16:35:46.855197 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:46.855182 4363 scope.go:122] "RemoveContainer" containerID="646cf3813c38dae5d27f6ea50c5de77e494d6e91c9e92e741f0d679dd66a43fb" Nov 01 16:35:46.855478 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:35:46.855461 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"646cf3813c38dae5d27f6ea50c5de77e494d6e91c9e92e741f0d679dd66a43fb\": container with ID starting with 646cf3813c38dae5d27f6ea50c5de77e494d6e91c9e92e741f0d679dd66a43fb not found: ID does not exist" containerID="646cf3813c38dae5d27f6ea50c5de77e494d6e91c9e92e741f0d679dd66a43fb" Nov 01 16:35:46.855558 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:46.855486 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"646cf3813c38dae5d27f6ea50c5de77e494d6e91c9e92e741f0d679dd66a43fb"} err="failed to get container status \"646cf3813c38dae5d27f6ea50c5de77e494d6e91c9e92e741f0d679dd66a43fb\": rpc error: code = NotFound desc = could not find container \"646cf3813c38dae5d27f6ea50c5de77e494d6e91c9e92e741f0d679dd66a43fb\": container with ID starting with 646cf3813c38dae5d27f6ea50c5de77e494d6e91c9e92e741f0d679dd66a43fb not found: ID does not exist" Nov 01 16:35:46.868373 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:46.868333 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a9a967-6948b69c67-tgrtj"] Nov 01 16:35:46.868522 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:46.868388 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a9a967-6948b69c67-tgrtj"] Nov 01 16:35:48.308394 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:35:48.308360 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:35:48.795899 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:48.795863 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de4bccaf-0cc6-4d9a-8b5c-800217cff553" path="/var/lib/kubelet/pods/de4bccaf-0cc6-4d9a-8b5c-800217cff553/volumes" Nov 01 16:35:50.223619 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:50.223549 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" containerID="cri-o://6a0250be15169c8396b6a52f815f873c677123510048d991cafa4b288a205fa0" gracePeriod=45 Nov 01 16:35:50.469853 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:50.469829 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:35:50.558267 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:50.558184 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed53e8bc-2f4c-426d-8088-ccaac93847ff-tls-certs\" (UniqueName: \"kubernetes.io/secret/ed53e8bc-2f4c-426d-8088-ccaac93847ff-tls-certs\") pod \"ed53e8bc-2f4c-426d-8088-ccaac93847ff\" (UID: \"ed53e8bc-2f4c-426d-8088-ccaac93847ff\") " Nov 01 16:35:50.558447 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:50.558270 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed53e8bc-2f4c-426d-8088-ccaac93847ff-kube-api-access-r79xb\" (UniqueName: \"kubernetes.io/projected/ed53e8bc-2f4c-426d-8088-ccaac93847ff-kube-api-access-r79xb\") pod \"ed53e8bc-2f4c-426d-8088-ccaac93847ff\" (UID: \"ed53e8bc-2f4c-426d-8088-ccaac93847ff\") " Nov 01 16:35:50.558447 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:50.558314 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed53e8bc-2f4c-426d-8088-ccaac93847ff-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed53e8bc-2f4c-426d-8088-ccaac93847ff-tmp-dir\") pod \"ed53e8bc-2f4c-426d-8088-ccaac93847ff\" (UID: \"ed53e8bc-2f4c-426d-8088-ccaac93847ff\") " Nov 01 16:35:50.558447 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:50.558336 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed53e8bc-2f4c-426d-8088-ccaac93847ff-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed53e8bc-2f4c-426d-8088-ccaac93847ff-model-cache\") pod \"ed53e8bc-2f4c-426d-8088-ccaac93847ff\" (UID: \"ed53e8bc-2f4c-426d-8088-ccaac93847ff\") " Nov 01 16:35:50.558447 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:50.558402 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed53e8bc-2f4c-426d-8088-ccaac93847ff-home\" (UniqueName: \"kubernetes.io/empty-dir/ed53e8bc-2f4c-426d-8088-ccaac93847ff-home\") pod \"ed53e8bc-2f4c-426d-8088-ccaac93847ff\" (UID: \"ed53e8bc-2f4c-426d-8088-ccaac93847ff\") " Nov 01 16:35:50.558692 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:50.558448 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed53e8bc-2f4c-426d-8088-ccaac93847ff-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed53e8bc-2f4c-426d-8088-ccaac93847ff-dshm\") pod \"ed53e8bc-2f4c-426d-8088-ccaac93847ff\" (UID: \"ed53e8bc-2f4c-426d-8088-ccaac93847ff\") " Nov 01 16:35:50.558692 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:50.558638 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed53e8bc-2f4c-426d-8088-ccaac93847ff-tmp-dir" pod "ed53e8bc-2f4c-426d-8088-ccaac93847ff" (UID: "ed53e8bc-2f4c-426d-8088-ccaac93847ff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:50.558820 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:50.558691 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed53e8bc-2f4c-426d-8088-ccaac93847ff-home" pod "ed53e8bc-2f4c-426d-8088-ccaac93847ff" (UID: "ed53e8bc-2f4c-426d-8088-ccaac93847ff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:50.558820 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:50.558700 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed53e8bc-2f4c-426d-8088-ccaac93847ff-model-cache" pod "ed53e8bc-2f4c-426d-8088-ccaac93847ff" (UID: "ed53e8bc-2f4c-426d-8088-ccaac93847ff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:50.558927 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:50.558856 4363 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed53e8bc-2f4c-426d-8088-ccaac93847ff-tmp-dir\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:50.558927 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:50.558870 4363 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed53e8bc-2f4c-426d-8088-ccaac93847ff-model-cache\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:50.558927 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:50.558881 4363 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed53e8bc-2f4c-426d-8088-ccaac93847ff-home\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:50.560875 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:50.560850 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed53e8bc-2f4c-426d-8088-ccaac93847ff-kube-api-access-r79xb" pod "ed53e8bc-2f4c-426d-8088-ccaac93847ff" (UID: "ed53e8bc-2f4c-426d-8088-ccaac93847ff"). InnerVolumeSpecName "kube-api-access-r79xb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:35:50.561073 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:50.561050 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed53e8bc-2f4c-426d-8088-ccaac93847ff-tls-certs" pod "ed53e8bc-2f4c-426d-8088-ccaac93847ff" (UID: "ed53e8bc-2f4c-426d-8088-ccaac93847ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:35:50.561158 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:50.561066 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed53e8bc-2f4c-426d-8088-ccaac93847ff-dshm" pod "ed53e8bc-2f4c-426d-8088-ccaac93847ff" (UID: "ed53e8bc-2f4c-426d-8088-ccaac93847ff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:50.660234 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:50.660201 4363 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed53e8bc-2f4c-426d-8088-ccaac93847ff-tls-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:50.660234 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:50.660230 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r79xb\" (UniqueName: \"kubernetes.io/projected/ed53e8bc-2f4c-426d-8088-ccaac93847ff-kube-api-access-r79xb\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:50.660234 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:50.660242 4363 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed53e8bc-2f4c-426d-8088-ccaac93847ff-dshm\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:50.867515 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:50.867486 4363 generic.go:356] "Generic (PLEG): container finished" podID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerID="6a0250be15169c8396b6a52f815f873c677123510048d991cafa4b288a205fa0" exitCode=0 Nov 01 16:35:50.867687 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:50.867518 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" event={"ID":"ed53e8bc-2f4c-426d-8088-ccaac93847ff","Type":"ContainerDied","Data":"6a0250be15169c8396b6a52f815f873c677123510048d991cafa4b288a205fa0"} Nov 01 16:35:50.867687 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:50.867535 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" Nov 01 16:35:50.867687 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:50.867561 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" event={"ID":"ed53e8bc-2f4c-426d-8088-ccaac93847ff","Type":"ContainerDied","Data":"4791c407044df2b2d1b85a2525f5c28f2a8cd76d7cc3d10613076272a3569fd7"} Nov 01 16:35:50.867687 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:50.867580 4363 scope.go:122] "RemoveContainer" containerID="6a0250be15169c8396b6a52f815f873c677123510048d991cafa4b288a205fa0" Nov 01 16:35:50.878490 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:50.878473 4363 scope.go:122] "RemoveContainer" containerID="6a0250be15169c8396b6a52f815f873c677123510048d991cafa4b288a205fa0" Nov 01 16:35:50.878791 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:35:50.878765 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a0250be15169c8396b6a52f815f873c677123510048d991cafa4b288a205fa0\": container with ID starting with 6a0250be15169c8396b6a52f815f873c677123510048d991cafa4b288a205fa0 not found: ID does not exist" containerID="6a0250be15169c8396b6a52f815f873c677123510048d991cafa4b288a205fa0" Nov 01 16:35:50.878888 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:50.878797 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a0250be15169c8396b6a52f815f873c677123510048d991cafa4b288a205fa0"} err="failed to get container status \"6a0250be15169c8396b6a52f815f873c677123510048d991cafa4b288a205fa0\": rpc error: code = NotFound desc = could not find container \"6a0250be15169c8396b6a52f815f873c677123510048d991cafa4b288a205fa0\": container with ID starting with 6a0250be15169c8396b6a52f815f873c677123510048d991cafa4b288a205fa0 not found: ID does not exist" Nov 01 16:35:50.882934 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:50.882897 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx"] Nov 01 16:35:50.886796 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:50.886776 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx"] Nov 01 16:35:51.449143 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:51.449097 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/premium-w5-164db6-kserve-68b84d5f96-nbgxx" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:35:51.818754 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:51.818676 4363 scope.go:122] "RemoveContainer" containerID="528285b11d82c8fe43d425d5e388473abb8f2782f0e0bc3e00567d2bfc460c71" Nov 01 16:35:51.827840 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:51.827819 4363 scope.go:122] "RemoveContainer" containerID="135a09e4b651f948a2cb35aac342ba4828b33c07358a429fccdf08a44569003a" Nov 01 16:35:52.798399 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:52.798364 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" path="/var/lib/kubelet/pods/ed53e8bc-2f4c-426d-8088-ccaac93847ff/volumes" Nov 01 16:35:55.950562 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:55.950510 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-164db6-5746fd596c-rmbpv"] Nov 01 16:35:55.951131 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:55.950733 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-164db6-5746fd596c-rmbpv" podUID="c6e54fd2-84cf-4043-b891-ba574ddef23b" containerName="maas-api" containerID="cri-o://829758803f3353e8fcd3c0df278d7563debfbfdfbc34035d86bba6e723eafc70" gracePeriod=30 Nov 01 16:35:56.022631 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:56.022594 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96e10868-s6xp2"] Nov 01 16:35:56.024987 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:56.024962 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96e10868-s6xp2"] Nov 01 16:35:56.209137 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:56.209074 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-164db6-5746fd596c-rmbpv" Nov 01 16:35:56.313680 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:56.313650 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6e54fd2-84cf-4043-b891-ba574ddef23b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6e54fd2-84cf-4043-b891-ba574ddef23b-openshift-service-ca\") pod \"c6e54fd2-84cf-4043-b891-ba574ddef23b\" (UID: \"c6e54fd2-84cf-4043-b891-ba574ddef23b\") " Nov 01 16:35:56.313889 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:56.313749 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6e54fd2-84cf-4043-b891-ba574ddef23b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6e54fd2-84cf-4043-b891-ba574ddef23b-metrics-certs\") pod \"c6e54fd2-84cf-4043-b891-ba574ddef23b\" (UID: \"c6e54fd2-84cf-4043-b891-ba574ddef23b\") " Nov 01 16:35:56.313889 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:56.313790 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6e54fd2-84cf-4043-b891-ba574ddef23b-kube-api-access-j7q9q\" (UniqueName: \"kubernetes.io/projected/c6e54fd2-84cf-4043-b891-ba574ddef23b-kube-api-access-j7q9q\") pod \"c6e54fd2-84cf-4043-b891-ba574ddef23b\" (UID: \"c6e54fd2-84cf-4043-b891-ba574ddef23b\") " Nov 01 16:35:56.313889 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:56.313815 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6e54fd2-84cf-4043-b891-ba574ddef23b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6e54fd2-84cf-4043-b891-ba574ddef23b-maas-api-tls\") pod \"c6e54fd2-84cf-4043-b891-ba574ddef23b\" (UID: \"c6e54fd2-84cf-4043-b891-ba574ddef23b\") " Nov 01 16:35:56.314140 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:56.314116 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6e54fd2-84cf-4043-b891-ba574ddef23b-openshift-service-ca" pod "c6e54fd2-84cf-4043-b891-ba574ddef23b" (UID: "c6e54fd2-84cf-4043-b891-ba574ddef23b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:35:56.315985 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:56.315951 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6e54fd2-84cf-4043-b891-ba574ddef23b-kube-api-access-j7q9q" pod "c6e54fd2-84cf-4043-b891-ba574ddef23b" (UID: "c6e54fd2-84cf-4043-b891-ba574ddef23b"). InnerVolumeSpecName "kube-api-access-j7q9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:35:56.315985 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:56.315960 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6e54fd2-84cf-4043-b891-ba574ddef23b-metrics-certs" pod "c6e54fd2-84cf-4043-b891-ba574ddef23b" (UID: "c6e54fd2-84cf-4043-b891-ba574ddef23b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:35:56.316211 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:56.316000 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6e54fd2-84cf-4043-b891-ba574ddef23b-maas-api-tls" pod "c6e54fd2-84cf-4043-b891-ba574ddef23b" (UID: "c6e54fd2-84cf-4043-b891-ba574ddef23b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:35:56.414569 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:56.414539 4363 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6e54fd2-84cf-4043-b891-ba574ddef23b-openshift-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:56.414569 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:56.414566 4363 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6e54fd2-84cf-4043-b891-ba574ddef23b-metrics-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:56.414801 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:56.414577 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7q9q\" (UniqueName: \"kubernetes.io/projected/c6e54fd2-84cf-4043-b891-ba574ddef23b-kube-api-access-j7q9q\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:56.414801 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:56.414587 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6e54fd2-84cf-4043-b891-ba574ddef23b-maas-api-tls\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:56.796069 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:56.796035 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6cdf1c8-f85e-4abf-af8a-8a695f7302de" path="/var/lib/kubelet/pods/e6cdf1c8-f85e-4abf-af8a-8a695f7302de/volumes" Nov 01 16:35:56.904148 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:56.904116 4363 generic.go:356] "Generic (PLEG): container finished" podID="c6e54fd2-84cf-4043-b891-ba574ddef23b" containerID="829758803f3353e8fcd3c0df278d7563debfbfdfbc34035d86bba6e723eafc70" exitCode=0 Nov 01 16:35:56.904508 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:56.904158 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-164db6-5746fd596c-rmbpv" Nov 01 16:35:56.904508 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:56.904203 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-164db6-5746fd596c-rmbpv" event={"ID":"c6e54fd2-84cf-4043-b891-ba574ddef23b","Type":"ContainerDied","Data":"829758803f3353e8fcd3c0df278d7563debfbfdfbc34035d86bba6e723eafc70"} Nov 01 16:35:56.904508 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:56.904262 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-164db6-5746fd596c-rmbpv" event={"ID":"c6e54fd2-84cf-4043-b891-ba574ddef23b","Type":"ContainerDied","Data":"a55ab93182a831be377ea7e9b55b309406723a92dbfb37650fa7431e621e0edc"} Nov 01 16:35:56.904508 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:56.904290 4363 scope.go:122] "RemoveContainer" containerID="829758803f3353e8fcd3c0df278d7563debfbfdfbc34035d86bba6e723eafc70" Nov 01 16:35:56.913657 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:56.913638 4363 scope.go:122] "RemoveContainer" containerID="829758803f3353e8fcd3c0df278d7563debfbfdfbc34035d86bba6e723eafc70" Nov 01 16:35:56.913938 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:35:56.913915 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"829758803f3353e8fcd3c0df278d7563debfbfdfbc34035d86bba6e723eafc70\": container with ID starting with 829758803f3353e8fcd3c0df278d7563debfbfdfbc34035d86bba6e723eafc70 not found: ID does not exist" containerID="829758803f3353e8fcd3c0df278d7563debfbfdfbc34035d86bba6e723eafc70" Nov 01 16:35:56.914089 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:56.913950 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"829758803f3353e8fcd3c0df278d7563debfbfdfbc34035d86bba6e723eafc70"} err="failed to get container status \"829758803f3353e8fcd3c0df278d7563debfbfdfbc34035d86bba6e723eafc70\": rpc error: code = NotFound desc = could not find container \"829758803f3353e8fcd3c0df278d7563debfbfdfbc34035d86bba6e723eafc70\": container with ID starting with 829758803f3353e8fcd3c0df278d7563debfbfdfbc34035d86bba6e723eafc70 not found: ID does not exist" Nov 01 16:35:56.919588 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:56.919561 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-164db6-5746fd596c-rmbpv"] Nov 01 16:35:56.926003 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:56.925981 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-164db6-5746fd596c-rmbpv"] Nov 01 16:35:58.797330 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:35:58.797288 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6e54fd2-84cf-4043-b891-ba574ddef23b" path="/var/lib/kubelet/pods/c6e54fd2-84cf-4043-b891-ba574ddef23b/volumes" Nov 01 16:36:07.081011 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.080976 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-646677ff99-r8pcz"] Nov 01 16:36:07.081585 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.081565 4363 cpu_manager.go:414] "containerMap: removing container" podUID="3f975360-c532-4eb2-aae5-5233d03d027f" containerName="istio-proxy" Nov 01 16:36:07.081678 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.081588 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f975360-c532-4eb2-aae5-5233d03d027f" containerName="istio-proxy" Nov 01 16:36:07.081678 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.081604 4363 cpu_manager.go:414] "containerMap: removing container" podUID="b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697" containerName="istio-proxy" Nov 01 16:36:07.081678 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.081611 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697" containerName="istio-proxy" Nov 01 16:36:07.081678 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.081622 4363 cpu_manager.go:414] "containerMap: removing container" podUID="cb31d5ec-ff4f-4fbf-b78f-ab022f851e78" containerName="payload-processing" Nov 01 16:36:07.081678 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.081629 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb31d5ec-ff4f-4fbf-b78f-ab022f851e78" containerName="payload-processing" Nov 01 16:36:07.081678 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.081645 4363 cpu_manager.go:414] "containerMap: removing container" podUID="c6e54fd2-84cf-4043-b891-ba574ddef23b" containerName="maas-api" Nov 01 16:36:07.081678 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.081653 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6e54fd2-84cf-4043-b891-ba574ddef23b" containerName="maas-api" Nov 01 16:36:07.081678 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.081663 4363 cpu_manager.go:414] "containerMap: removing container" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" Nov 01 16:36:07.081678 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.081670 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" Nov 01 16:36:07.082210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.081683 4363 cpu_manager.go:414] "containerMap: removing container" podUID="e9235a8f-5d47-4464-a45f-6344c833c25a" containerName="maas-api" Nov 01 16:36:07.082210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.081690 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9235a8f-5d47-4464-a45f-6344c833c25a" containerName="maas-api" Nov 01 16:36:07.082210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.081704 4363 cpu_manager.go:414] "containerMap: removing container" podUID="4324048b-b48c-452d-91df-8e48f5e8e97a" containerName="payload-pre-processing" Nov 01 16:36:07.082210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.081712 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4324048b-b48c-452d-91df-8e48f5e8e97a" containerName="payload-pre-processing" Nov 01 16:36:07.082210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.081727 4363 cpu_manager.go:414] "containerMap: removing container" podUID="247d3008-5891-46e2-843b-da07c4329f01" containerName="payload-processing" Nov 01 16:36:07.082210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.081735 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="247d3008-5891-46e2-843b-da07c4329f01" containerName="payload-processing" Nov 01 16:36:07.082210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.081744 4363 cpu_manager.go:414] "containerMap: removing container" podUID="0cc3066c-d320-41fa-8a7b-933362174b48" containerName="istio-proxy" Nov 01 16:36:07.082210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.081752 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cc3066c-d320-41fa-8a7b-933362174b48" containerName="istio-proxy" Nov 01 16:36:07.082210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.081765 4363 cpu_manager.go:414] "containerMap: removing container" podUID="87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519" containerName="istio-proxy" Nov 01 16:36:07.082210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.081772 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519" containerName="istio-proxy" Nov 01 16:36:07.082210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.081785 4363 cpu_manager.go:414] "containerMap: removing container" podUID="a27bb1f1-cb87-4003-862c-4839331f5c9e" containerName="istio-proxy" Nov 01 16:36:07.082210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.081791 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a27bb1f1-cb87-4003-862c-4839331f5c9e" containerName="istio-proxy" Nov 01 16:36:07.082210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.081805 4363 cpu_manager.go:414] "containerMap: removing container" podUID="de4bccaf-0cc6-4d9a-8b5c-800217cff553" containerName="payload-pre-processing" Nov 01 16:36:07.082210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.081813 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de4bccaf-0cc6-4d9a-8b5c-800217cff553" containerName="payload-pre-processing" Nov 01 16:36:07.082210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.081826 4363 cpu_manager.go:414] "containerMap: removing container" podUID="79cee252-8c24-49ac-b432-49de53bb30aa" containerName="payload-pre-processing" Nov 01 16:36:07.082210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.081833 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79cee252-8c24-49ac-b432-49de53bb30aa" containerName="payload-pre-processing" Nov 01 16:36:07.082210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.081849 4363 cpu_manager.go:414] "containerMap: removing container" podUID="10f4b6b1-4845-4f52-b90c-7650dfdb27b7" containerName="payload-processing" Nov 01 16:36:07.082210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.081856 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10f4b6b1-4845-4f52-b90c-7650dfdb27b7" containerName="payload-processing" Nov 01 16:36:07.082210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.081866 4363 cpu_manager.go:414] "containerMap: removing container" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" Nov 01 16:36:07.082210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.081873 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" Nov 01 16:36:07.082210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.081885 4363 cpu_manager.go:414] "containerMap: removing container" podUID="c2d00eaa-b184-4176-9944-ef381e8eaec0" containerName="payload-processing" Nov 01 16:36:07.082210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.081893 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2d00eaa-b184-4176-9944-ef381e8eaec0" containerName="payload-processing" Nov 01 16:36:07.082210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.081904 4363 cpu_manager.go:414] "containerMap: removing container" podUID="c176da5b-fd83-454e-bbb7-bb793807cb67" containerName="istio-proxy" Nov 01 16:36:07.082210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.081911 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c176da5b-fd83-454e-bbb7-bb793807cb67" containerName="istio-proxy" Nov 01 16:36:07.082210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.081925 4363 cpu_manager.go:414] "containerMap: removing container" podUID="0240fa16-cc94-4d43-a3a8-ae27ca53f6ec" containerName="payload-pre-processing" Nov 01 16:36:07.082210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.081932 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0240fa16-cc94-4d43-a3a8-ae27ca53f6ec" containerName="payload-pre-processing" Nov 01 16:36:07.082210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.081942 4363 cpu_manager.go:414] "containerMap: removing container" podUID="159aed21-0910-4554-a841-b7c096765924" containerName="maas-api" Nov 01 16:36:07.082210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.081950 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="159aed21-0910-4554-a841-b7c096765924" containerName="maas-api" Nov 01 16:36:07.082210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.081960 4363 cpu_manager.go:414] "containerMap: removing container" podUID="b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697" containerName="istio-proxy" Nov 01 16:36:07.082210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.081967 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697" containerName="istio-proxy" Nov 01 16:36:07.082210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.081988 4363 cpu_manager.go:414] "containerMap: removing container" podUID="387b6adf-9404-414e-ae75-0b48e938e77a" containerName="payload-processing" Nov 01 16:36:07.082210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.081995 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="387b6adf-9404-414e-ae75-0b48e938e77a" containerName="payload-processing" Nov 01 16:36:07.082210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.082010 4363 cpu_manager.go:414] "containerMap: removing container" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" Nov 01 16:36:07.082210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.082046 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" Nov 01 16:36:07.082210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.082059 4363 cpu_manager.go:414] "containerMap: removing container" podUID="ec15b77d-c274-4eb4-bc47-6134110a0a05" containerName="payload-processing" Nov 01 16:36:07.082210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.082066 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec15b77d-c274-4eb4-bc47-6134110a0a05" containerName="payload-processing" Nov 01 16:36:07.082210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.082084 4363 cpu_manager.go:414] "containerMap: removing container" podUID="87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519" containerName="istio-proxy" Nov 01 16:36:07.082210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.082091 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519" containerName="istio-proxy" Nov 01 16:36:07.082210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.082099 4363 cpu_manager.go:414] "containerMap: removing container" podUID="6daeee34-6958-40f5-a360-9a58b3bd51d8" containerName="istio-proxy" Nov 01 16:36:07.082210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.082107 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6daeee34-6958-40f5-a360-9a58b3bd51d8" containerName="istio-proxy" Nov 01 16:36:07.082210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.082199 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="a27bb1f1-cb87-4003-862c-4839331f5c9e" containerName="istio-proxy" Nov 01 16:36:07.082210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.082215 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="de4bccaf-0cc6-4d9a-8b5c-800217cff553" containerName="payload-pre-processing" Nov 01 16:36:07.082210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.082227 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="79cee252-8c24-49ac-b432-49de53bb30aa" containerName="payload-pre-processing" Nov 01 16:36:07.084207 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.082238 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="10f4b6b1-4845-4f52-b90c-7650dfdb27b7" containerName="payload-processing" Nov 01 16:36:07.084207 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.082248 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6476aee-e7a4-497f-84f0-ef0d63ede742" containerName="main" Nov 01 16:36:07.084207 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.082257 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2d00eaa-b184-4176-9944-ef381e8eaec0" containerName="payload-processing" Nov 01 16:36:07.084207 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.082266 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="c176da5b-fd83-454e-bbb7-bb793807cb67" containerName="istio-proxy" Nov 01 16:36:07.084207 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.082276 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="0240fa16-cc94-4d43-a3a8-ae27ca53f6ec" containerName="payload-pre-processing" Nov 01 16:36:07.084207 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.082285 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="159aed21-0910-4554-a841-b7c096765924" containerName="maas-api" Nov 01 16:36:07.084207 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.082294 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697" containerName="istio-proxy" Nov 01 16:36:07.084207 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.082304 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="387b6adf-9404-414e-ae75-0b48e938e77a" containerName="payload-processing" Nov 01 16:36:07.084207 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.082385 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="8261b28f-6536-4ba3-88f6-0713370eb6d3" containerName="main" Nov 01 16:36:07.084207 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.082398 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec15b77d-c274-4eb4-bc47-6134110a0a05" containerName="payload-processing" Nov 01 16:36:07.084207 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.082408 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519" containerName="istio-proxy" Nov 01 16:36:07.084207 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.082416 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="6daeee34-6958-40f5-a360-9a58b3bd51d8" containerName="istio-proxy" Nov 01 16:36:07.084207 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.082426 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f975360-c532-4eb2-aae5-5233d03d027f" containerName="istio-proxy" Nov 01 16:36:07.084207 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.082434 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2b69d5a-3eef-4aaf-a3ad-454ac3dd6697" containerName="istio-proxy" Nov 01 16:36:07.084207 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.082443 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb31d5ec-ff4f-4fbf-b78f-ab022f851e78" containerName="payload-processing" Nov 01 16:36:07.084207 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.082451 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6e54fd2-84cf-4043-b891-ba574ddef23b" containerName="maas-api" Nov 01 16:36:07.084207 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.082459 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed53e8bc-2f4c-426d-8088-ccaac93847ff" containerName="main" Nov 01 16:36:07.084207 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.082470 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9235a8f-5d47-4464-a45f-6344c833c25a" containerName="maas-api" Nov 01 16:36:07.084207 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.082479 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="4324048b-b48c-452d-91df-8e48f5e8e97a" containerName="payload-pre-processing" Nov 01 16:36:07.084207 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.082488 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="247d3008-5891-46e2-843b-da07c4329f01" containerName="payload-processing" Nov 01 16:36:07.084207 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.082496 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cc3066c-d320-41fa-8a7b-933362174b48" containerName="istio-proxy" Nov 01 16:36:07.084207 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.082505 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="87ddacbe-42b0-42b0-8b1c-a3e6c4ed4519" containerName="istio-proxy" Nov 01 16:36:07.087339 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.087316 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-646677ff99-r8pcz" Nov 01 16:36:07.093393 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.093359 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 01 16:36:07.097854 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.096884 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-646677ff99-r8pcz"] Nov 01 16:36:07.097854 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.096945 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 01 16:36:07.097854 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.097084 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 01 16:36:07.097854 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.097180 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-kqkln\"" Nov 01 16:36:07.222459 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.222420 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd02427d-9f19-445c-8201-d3523ae9a81e-metrics-certs\") pod \"maas-controller-646677ff99-r8pcz\" (UID: \"cd02427d-9f19-445c-8201-d3523ae9a81e\") " pod="opendatahub/maas-controller-646677ff99-r8pcz" Nov 01 16:36:07.222650 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.222473 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cd02427d-9f19-445c-8201-d3523ae9a81e-webhook-cert\") pod \"maas-controller-646677ff99-r8pcz\" (UID: \"cd02427d-9f19-445c-8201-d3523ae9a81e\") " pod="opendatahub/maas-controller-646677ff99-r8pcz" Nov 01 16:36:07.222650 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.222613 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpx2m\" (UniqueName: \"kubernetes.io/projected/cd02427d-9f19-445c-8201-d3523ae9a81e-kube-api-access-rpx2m\") pod \"maas-controller-646677ff99-r8pcz\" (UID: \"cd02427d-9f19-445c-8201-d3523ae9a81e\") " pod="opendatahub/maas-controller-646677ff99-r8pcz" Nov 01 16:36:07.323943 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.323905 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpx2m\" (UniqueName: \"kubernetes.io/projected/cd02427d-9f19-445c-8201-d3523ae9a81e-kube-api-access-rpx2m\") pod \"maas-controller-646677ff99-r8pcz\" (UID: \"cd02427d-9f19-445c-8201-d3523ae9a81e\") " pod="opendatahub/maas-controller-646677ff99-r8pcz" Nov 01 16:36:07.324149 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.324010 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd02427d-9f19-445c-8201-d3523ae9a81e-metrics-certs\") pod \"maas-controller-646677ff99-r8pcz\" (UID: \"cd02427d-9f19-445c-8201-d3523ae9a81e\") " pod="opendatahub/maas-controller-646677ff99-r8pcz" Nov 01 16:36:07.324149 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.324066 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cd02427d-9f19-445c-8201-d3523ae9a81e-webhook-cert\") pod \"maas-controller-646677ff99-r8pcz\" (UID: \"cd02427d-9f19-445c-8201-d3523ae9a81e\") " pod="opendatahub/maas-controller-646677ff99-r8pcz" Nov 01 16:36:07.326761 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.326735 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd02427d-9f19-445c-8201-d3523ae9a81e-metrics-certs\") pod \"maas-controller-646677ff99-r8pcz\" (UID: \"cd02427d-9f19-445c-8201-d3523ae9a81e\") " pod="opendatahub/maas-controller-646677ff99-r8pcz" Nov 01 16:36:07.326872 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.326735 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cd02427d-9f19-445c-8201-d3523ae9a81e-webhook-cert\") pod \"maas-controller-646677ff99-r8pcz\" (UID: \"cd02427d-9f19-445c-8201-d3523ae9a81e\") " pod="opendatahub/maas-controller-646677ff99-r8pcz" Nov 01 16:36:07.331210 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.331153 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpx2m\" (UniqueName: \"kubernetes.io/projected/cd02427d-9f19-445c-8201-d3523ae9a81e-kube-api-access-rpx2m\") pod \"maas-controller-646677ff99-r8pcz\" (UID: \"cd02427d-9f19-445c-8201-d3523ae9a81e\") " pod="opendatahub/maas-controller-646677ff99-r8pcz" Nov 01 16:36:07.402882 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.402851 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-646677ff99-r8pcz" Nov 01 16:36:07.695586 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.694098 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-646677ff99-r8pcz"] Nov 01 16:36:07.961361 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:07.961272 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-646677ff99-r8pcz" event={"ID":"cd02427d-9f19-445c-8201-d3523ae9a81e","Type":"ContainerStarted","Data":"6a480ce5c0a9530369b2412d315a1c016fa11353e1db3174e37c8f8f91eb6890"} Nov 01 16:36:10.976654 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:10.976554 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-646677ff99-r8pcz" event={"ID":"cd02427d-9f19-445c-8201-d3523ae9a81e","Type":"ContainerStarted","Data":"d31127fbebc1e6494b9d99befb9403b52db1e6707525815ae11e7725d125fcfb"} Nov 01 16:36:10.977073 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:10.976680 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-646677ff99-r8pcz" Nov 01 16:36:10.995518 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:10.995470 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-646677ff99-r8pcz" podStartSLOduration=1.096423547 podStartE2EDuration="3.995456672s" totalImagesPullingTime="2.899033125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:36:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:36:10.993199967 +0000 UTC m=+2904.324937614" watchObservedRunningTime="2026-11-01 16:36:10.995456672 +0000 UTC m=+2904.327194322" Nov 01 16:36:18.307138 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:36:18.307101 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:36:20.165499 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:20.165464 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6"] Nov 01 16:36:20.169282 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:20.169260 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" Nov 01 16:36:20.189009 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:20.188979 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6"] Nov 01 16:36:20.245970 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:20.245934 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e26e65-03fd-4fad-99dd-a4d334b883e3-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-65xj6\" (UID: \"97e26e65-03fd-4fad-99dd-a4d334b883e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" Nov 01 16:36:20.245970 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:20.245968 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97e26e65-03fd-4fad-99dd-a4d334b883e3-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-65xj6\" (UID: \"97e26e65-03fd-4fad-99dd-a4d334b883e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" Nov 01 16:36:20.246193 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:20.245989 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97e26e65-03fd-4fad-99dd-a4d334b883e3-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-65xj6\" (UID: \"97e26e65-03fd-4fad-99dd-a4d334b883e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" Nov 01 16:36:20.246193 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:20.246013 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg48d\" (UniqueName: \"kubernetes.io/projected/97e26e65-03fd-4fad-99dd-a4d334b883e3-kube-api-access-gg48d\") pod \"maas-default-gateway-openshift-default-7f5667f86c-65xj6\" (UID: \"97e26e65-03fd-4fad-99dd-a4d334b883e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" Nov 01 16:36:20.246193 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:20.246154 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97e26e65-03fd-4fad-99dd-a4d334b883e3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-65xj6\" (UID: \"97e26e65-03fd-4fad-99dd-a4d334b883e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" Nov 01 16:36:20.246193 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:20.246187 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97e26e65-03fd-4fad-99dd-a4d334b883e3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-65xj6\" (UID: \"97e26e65-03fd-4fad-99dd-a4d334b883e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" Nov 01 16:36:20.246340 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:20.246218 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e26e65-03fd-4fad-99dd-a4d334b883e3-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-65xj6\" (UID: \"97e26e65-03fd-4fad-99dd-a4d334b883e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" Nov 01 16:36:20.246340 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:20.246256 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97e26e65-03fd-4fad-99dd-a4d334b883e3-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-65xj6\" (UID: \"97e26e65-03fd-4fad-99dd-a4d334b883e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" Nov 01 16:36:20.246413 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:20.246372 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97e26e65-03fd-4fad-99dd-a4d334b883e3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-65xj6\" (UID: \"97e26e65-03fd-4fad-99dd-a4d334b883e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" Nov 01 16:36:20.346806 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:20.346779 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97e26e65-03fd-4fad-99dd-a4d334b883e3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-65xj6\" (UID: \"97e26e65-03fd-4fad-99dd-a4d334b883e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" Nov 01 16:36:20.346953 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:20.346813 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e26e65-03fd-4fad-99dd-a4d334b883e3-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-65xj6\" (UID: \"97e26e65-03fd-4fad-99dd-a4d334b883e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" Nov 01 16:36:20.346953 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:20.346829 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97e26e65-03fd-4fad-99dd-a4d334b883e3-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-65xj6\" (UID: \"97e26e65-03fd-4fad-99dd-a4d334b883e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" Nov 01 16:36:20.346953 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:20.346848 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97e26e65-03fd-4fad-99dd-a4d334b883e3-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-65xj6\" (UID: \"97e26e65-03fd-4fad-99dd-a4d334b883e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" Nov 01 16:36:20.346953 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:20.346878 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg48d\" (UniqueName: \"kubernetes.io/projected/97e26e65-03fd-4fad-99dd-a4d334b883e3-kube-api-access-gg48d\") pod \"maas-default-gateway-openshift-default-7f5667f86c-65xj6\" (UID: \"97e26e65-03fd-4fad-99dd-a4d334b883e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" Nov 01 16:36:20.346953 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:20.346916 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97e26e65-03fd-4fad-99dd-a4d334b883e3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-65xj6\" (UID: \"97e26e65-03fd-4fad-99dd-a4d334b883e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" Nov 01 16:36:20.346953 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:20.346936 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97e26e65-03fd-4fad-99dd-a4d334b883e3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-65xj6\" (UID: \"97e26e65-03fd-4fad-99dd-a4d334b883e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" Nov 01 16:36:20.347380 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:20.346977 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e26e65-03fd-4fad-99dd-a4d334b883e3-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-65xj6\" (UID: \"97e26e65-03fd-4fad-99dd-a4d334b883e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" Nov 01 16:36:20.347380 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:20.347001 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97e26e65-03fd-4fad-99dd-a4d334b883e3-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-65xj6\" (UID: \"97e26e65-03fd-4fad-99dd-a4d334b883e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" Nov 01 16:36:20.347380 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:20.347268 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e26e65-03fd-4fad-99dd-a4d334b883e3-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-65xj6\" (UID: \"97e26e65-03fd-4fad-99dd-a4d334b883e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" Nov 01 16:36:20.347380 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:20.347371 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97e26e65-03fd-4fad-99dd-a4d334b883e3-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-65xj6\" (UID: \"97e26e65-03fd-4fad-99dd-a4d334b883e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" Nov 01 16:36:20.347604 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:20.347426 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e26e65-03fd-4fad-99dd-a4d334b883e3-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-65xj6\" (UID: \"97e26e65-03fd-4fad-99dd-a4d334b883e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" Nov 01 16:36:20.347604 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:20.347487 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97e26e65-03fd-4fad-99dd-a4d334b883e3-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-65xj6\" (UID: \"97e26e65-03fd-4fad-99dd-a4d334b883e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" Nov 01 16:36:20.347753 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:20.347733 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97e26e65-03fd-4fad-99dd-a4d334b883e3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-65xj6\" (UID: \"97e26e65-03fd-4fad-99dd-a4d334b883e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" Nov 01 16:36:20.349171 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:20.349152 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97e26e65-03fd-4fad-99dd-a4d334b883e3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-65xj6\" (UID: \"97e26e65-03fd-4fad-99dd-a4d334b883e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" Nov 01 16:36:20.349441 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:20.349425 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97e26e65-03fd-4fad-99dd-a4d334b883e3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-65xj6\" (UID: \"97e26e65-03fd-4fad-99dd-a4d334b883e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" Nov 01 16:36:20.365724 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:20.365691 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg48d\" (UniqueName: \"kubernetes.io/projected/97e26e65-03fd-4fad-99dd-a4d334b883e3-kube-api-access-gg48d\") pod \"maas-default-gateway-openshift-default-7f5667f86c-65xj6\" (UID: \"97e26e65-03fd-4fad-99dd-a4d334b883e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" Nov 01 16:36:20.365965 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:20.365945 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97e26e65-03fd-4fad-99dd-a4d334b883e3-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-65xj6\" (UID: \"97e26e65-03fd-4fad-99dd-a4d334b883e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" Nov 01 16:36:20.490198 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:20.490115 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" Nov 01 16:36:20.766220 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:20.766193 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6"] Nov 01 16:36:20.768575 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:36:20.768551 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97e26e65_03fd_4fad_99dd_a4d334b883e3.slice/crio-e1eba09ab844c3e451fc0d5e6f57a77f6745569fe96ba2d1a888e9b655d6c904 WatchSource:0}: Error finding container e1eba09ab844c3e451fc0d5e6f57a77f6745569fe96ba2d1a888e9b655d6c904: Status 404 returned error can't find the container with id e1eba09ab844c3e451fc0d5e6f57a77f6745569fe96ba2d1a888e9b655d6c904 Nov 01 16:36:20.770652 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:20.770619 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:36:20.770731 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:20.770689 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:36:20.770731 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:20.770716 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:36:21.023564 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:21.023472 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" event={"ID":"97e26e65-03fd-4fad-99dd-a4d334b883e3","Type":"ContainerStarted","Data":"f306eb4b69a7ec6e815ae99efe43564798f9a2b3dd00a645144e8571faa111f7"} Nov 01 16:36:21.023564 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:21.023541 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" event={"ID":"97e26e65-03fd-4fad-99dd-a4d334b883e3","Type":"ContainerStarted","Data":"e1eba09ab844c3e451fc0d5e6f57a77f6745569fe96ba2d1a888e9b655d6c904"} Nov 01 16:36:21.044198 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:21.044144 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" podStartSLOduration=1.044128496 podStartE2EDuration="1.044128496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:36:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:36:21.042284171 +0000 UTC m=+2914.374021821" watchObservedRunningTime="2026-11-01 16:36:21.044128496 +0000 UTC m=+2914.375866146" Nov 01 16:36:21.497042 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:21.492869 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" Nov 01 16:36:21.505116 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:21.502266 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-65xj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" start-of-body= Nov 01 16:36:21.505116 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:21.502365 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" podUID="97e26e65-03fd-4fad-99dd-a4d334b883e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" Nov 01 16:36:21.990228 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:21.989260 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-646677ff99-r8pcz" Nov 01 16:36:22.495863 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:22.495416 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-65xj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" start-of-body= Nov 01 16:36:22.495863 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:22.495475 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" podUID="97e26e65-03fd-4fad-99dd-a4d334b883e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" Nov 01 16:36:23.502070 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:23.501054 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-65xj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" start-of-body= Nov 01 16:36:23.502070 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:23.501112 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" podUID="97e26e65-03fd-4fad-99dd-a4d334b883e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" Nov 01 16:36:24.500041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:24.495074 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-65xj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" start-of-body= Nov 01 16:36:24.500041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:24.495131 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" podUID="97e26e65-03fd-4fad-99dd-a4d334b883e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" Nov 01 16:36:25.503241 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:25.503189 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-65xj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" start-of-body= Nov 01 16:36:25.503700 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:25.503253 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" podUID="97e26e65-03fd-4fad-99dd-a4d334b883e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" Nov 01 16:36:26.496484 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:26.496443 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-65xj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" start-of-body= Nov 01 16:36:26.496840 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:26.496806 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" podUID="97e26e65-03fd-4fad-99dd-a4d334b883e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" Nov 01 16:36:27.491339 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:27.491304 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-65xj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" start-of-body= Nov 01 16:36:27.491828 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:27.491365 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" podUID="97e26e65-03fd-4fad-99dd-a4d334b883e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" Nov 01 16:36:29.491091 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:29.491045 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-65xj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:36:29.491604 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:29.491122 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" podUID="97e26e65-03fd-4fad-99dd-a4d334b883e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:36:30.532202 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:30.532159 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-65xj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:36:30.532659 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:30.532229 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" podUID="97e26e65-03fd-4fad-99dd-a4d334b883e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:36:31.448936 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:31.448900 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" Nov 01 16:36:31.449283 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:31.449257 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" Nov 01 16:36:31.449973 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:31.449956 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" Nov 01 16:36:38.868486 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:38.868444 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-646677ff99-r8pcz"] Nov 01 16:36:38.869433 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:38.869398 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-646677ff99-r8pcz" podUID="cd02427d-9f19-445c-8201-d3523ae9a81e" containerName="manager" containerID="cri-o://d31127fbebc1e6494b9d99befb9403b52db1e6707525815ae11e7725d125fcfb" gracePeriod=30 Nov 01 16:36:39.108423 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:39.108393 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-646677ff99-r8pcz" Nov 01 16:36:39.113401 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:39.113378 4363 generic.go:356] "Generic (PLEG): container finished" podID="cd02427d-9f19-445c-8201-d3523ae9a81e" containerID="d31127fbebc1e6494b9d99befb9403b52db1e6707525815ae11e7725d125fcfb" exitCode=0 Nov 01 16:36:39.113507 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:39.113419 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-646677ff99-r8pcz" Nov 01 16:36:39.113507 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:39.113445 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-646677ff99-r8pcz" event={"ID":"cd02427d-9f19-445c-8201-d3523ae9a81e","Type":"ContainerDied","Data":"d31127fbebc1e6494b9d99befb9403b52db1e6707525815ae11e7725d125fcfb"} Nov 01 16:36:39.113507 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:39.113505 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-646677ff99-r8pcz" event={"ID":"cd02427d-9f19-445c-8201-d3523ae9a81e","Type":"ContainerDied","Data":"6a480ce5c0a9530369b2412d315a1c016fa11353e1db3174e37c8f8f91eb6890"} Nov 01 16:36:39.113619 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:39.113528 4363 scope.go:122] "RemoveContainer" containerID="d31127fbebc1e6494b9d99befb9403b52db1e6707525815ae11e7725d125fcfb" Nov 01 16:36:39.121979 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:39.121961 4363 scope.go:122] "RemoveContainer" containerID="d31127fbebc1e6494b9d99befb9403b52db1e6707525815ae11e7725d125fcfb" Nov 01 16:36:39.122244 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:36:39.122228 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d31127fbebc1e6494b9d99befb9403b52db1e6707525815ae11e7725d125fcfb\": container with ID starting with d31127fbebc1e6494b9d99befb9403b52db1e6707525815ae11e7725d125fcfb not found: ID does not exist" containerID="d31127fbebc1e6494b9d99befb9403b52db1e6707525815ae11e7725d125fcfb" Nov 01 16:36:39.122298 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:39.122253 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d31127fbebc1e6494b9d99befb9403b52db1e6707525815ae11e7725d125fcfb"} err="failed to get container status \"d31127fbebc1e6494b9d99befb9403b52db1e6707525815ae11e7725d125fcfb\": rpc error: code = NotFound desc = could not find container \"d31127fbebc1e6494b9d99befb9403b52db1e6707525815ae11e7725d125fcfb\": container with ID starting with d31127fbebc1e6494b9d99befb9403b52db1e6707525815ae11e7725d125fcfb not found: ID does not exist" Nov 01 16:36:39.257211 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:39.257181 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd02427d-9f19-445c-8201-d3523ae9a81e-kube-api-access-rpx2m\" (UniqueName: \"kubernetes.io/projected/cd02427d-9f19-445c-8201-d3523ae9a81e-kube-api-access-rpx2m\") pod \"cd02427d-9f19-445c-8201-d3523ae9a81e\" (UID: \"cd02427d-9f19-445c-8201-d3523ae9a81e\") " Nov 01 16:36:39.257386 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:39.257282 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd02427d-9f19-445c-8201-d3523ae9a81e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd02427d-9f19-445c-8201-d3523ae9a81e-metrics-certs\") pod \"cd02427d-9f19-445c-8201-d3523ae9a81e\" (UID: \"cd02427d-9f19-445c-8201-d3523ae9a81e\") " Nov 01 16:36:39.257386 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:39.257327 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd02427d-9f19-445c-8201-d3523ae9a81e-webhook-cert\" (UniqueName: \"kubernetes.io/secret/cd02427d-9f19-445c-8201-d3523ae9a81e-webhook-cert\") pod \"cd02427d-9f19-445c-8201-d3523ae9a81e\" (UID: \"cd02427d-9f19-445c-8201-d3523ae9a81e\") " Nov 01 16:36:39.259242 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:39.259211 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd02427d-9f19-445c-8201-d3523ae9a81e-kube-api-access-rpx2m" pod "cd02427d-9f19-445c-8201-d3523ae9a81e" (UID: "cd02427d-9f19-445c-8201-d3523ae9a81e"). InnerVolumeSpecName "kube-api-access-rpx2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:36:39.259361 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:39.259238 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd02427d-9f19-445c-8201-d3523ae9a81e-webhook-cert" pod "cd02427d-9f19-445c-8201-d3523ae9a81e" (UID: "cd02427d-9f19-445c-8201-d3523ae9a81e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:36:39.259361 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:39.259274 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd02427d-9f19-445c-8201-d3523ae9a81e-metrics-certs" pod "cd02427d-9f19-445c-8201-d3523ae9a81e" (UID: "cd02427d-9f19-445c-8201-d3523ae9a81e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:36:39.358303 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:39.358273 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpx2m\" (UniqueName: \"kubernetes.io/projected/cd02427d-9f19-445c-8201-d3523ae9a81e-kube-api-access-rpx2m\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:36:39.358303 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:39.358299 4363 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd02427d-9f19-445c-8201-d3523ae9a81e-metrics-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:36:39.358478 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:39.358312 4363 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cd02427d-9f19-445c-8201-d3523ae9a81e-webhook-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:36:39.446479 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:39.446450 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-646677ff99-r8pcz"] Nov 01 16:36:39.450920 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:39.450898 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-646677ff99-r8pcz"] Nov 01 16:36:40.671044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:40.670987 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-646677ff99-v7mvq"] Nov 01 16:36:40.671593 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:40.671573 4363 cpu_manager.go:414] "containerMap: removing container" podUID="cd02427d-9f19-445c-8201-d3523ae9a81e" containerName="manager" Nov 01 16:36:40.671702 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:40.671597 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd02427d-9f19-445c-8201-d3523ae9a81e" containerName="manager" Nov 01 16:36:40.671761 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:40.671723 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd02427d-9f19-445c-8201-d3523ae9a81e" containerName="manager" Nov 01 16:36:40.676074 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:40.676051 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-646677ff99-v7mvq" Nov 01 16:36:40.684246 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:40.684208 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-646677ff99-v7mvq"] Nov 01 16:36:40.685943 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:40.684897 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 01 16:36:40.685943 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:40.684916 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-kqkln\"" Nov 01 16:36:40.685943 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:40.684973 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 01 16:36:40.685943 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:40.685086 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 01 16:36:40.771068 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:40.771035 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5g6r\" (UniqueName: \"kubernetes.io/projected/372f3c14-1225-451c-ae57-e0b295b31364-kube-api-access-d5g6r\") pod \"maas-controller-646677ff99-v7mvq\" (UID: \"372f3c14-1225-451c-ae57-e0b295b31364\") " pod="opendatahub/maas-controller-646677ff99-v7mvq" Nov 01 16:36:40.771227 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:40.771130 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/372f3c14-1225-451c-ae57-e0b295b31364-webhook-cert\") pod \"maas-controller-646677ff99-v7mvq\" (UID: \"372f3c14-1225-451c-ae57-e0b295b31364\") " pod="opendatahub/maas-controller-646677ff99-v7mvq" Nov 01 16:36:40.771227 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:40.771201 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/372f3c14-1225-451c-ae57-e0b295b31364-metrics-certs\") pod \"maas-controller-646677ff99-v7mvq\" (UID: \"372f3c14-1225-451c-ae57-e0b295b31364\") " pod="opendatahub/maas-controller-646677ff99-v7mvq" Nov 01 16:36:40.795349 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:40.795323 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd02427d-9f19-445c-8201-d3523ae9a81e" path="/var/lib/kubelet/pods/cd02427d-9f19-445c-8201-d3523ae9a81e/volumes" Nov 01 16:36:40.871833 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:40.871800 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5g6r\" (UniqueName: \"kubernetes.io/projected/372f3c14-1225-451c-ae57-e0b295b31364-kube-api-access-d5g6r\") pod \"maas-controller-646677ff99-v7mvq\" (UID: \"372f3c14-1225-451c-ae57-e0b295b31364\") " pod="opendatahub/maas-controller-646677ff99-v7mvq" Nov 01 16:36:40.871980 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:40.871858 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/372f3c14-1225-451c-ae57-e0b295b31364-webhook-cert\") pod \"maas-controller-646677ff99-v7mvq\" (UID: \"372f3c14-1225-451c-ae57-e0b295b31364\") " pod="opendatahub/maas-controller-646677ff99-v7mvq" Nov 01 16:36:40.871980 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:40.871896 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/372f3c14-1225-451c-ae57-e0b295b31364-metrics-certs\") pod \"maas-controller-646677ff99-v7mvq\" (UID: \"372f3c14-1225-451c-ae57-e0b295b31364\") " pod="opendatahub/maas-controller-646677ff99-v7mvq" Nov 01 16:36:40.874558 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:40.874530 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/372f3c14-1225-451c-ae57-e0b295b31364-webhook-cert\") pod \"maas-controller-646677ff99-v7mvq\" (UID: \"372f3c14-1225-451c-ae57-e0b295b31364\") " pod="opendatahub/maas-controller-646677ff99-v7mvq" Nov 01 16:36:40.874673 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:40.874530 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/372f3c14-1225-451c-ae57-e0b295b31364-metrics-certs\") pod \"maas-controller-646677ff99-v7mvq\" (UID: \"372f3c14-1225-451c-ae57-e0b295b31364\") " pod="opendatahub/maas-controller-646677ff99-v7mvq" Nov 01 16:36:40.879409 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:40.879381 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5g6r\" (UniqueName: \"kubernetes.io/projected/372f3c14-1225-451c-ae57-e0b295b31364-kube-api-access-d5g6r\") pod \"maas-controller-646677ff99-v7mvq\" (UID: \"372f3c14-1225-451c-ae57-e0b295b31364\") " pod="opendatahub/maas-controller-646677ff99-v7mvq" Nov 01 16:36:40.992880 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:40.992795 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-646677ff99-v7mvq" Nov 01 16:36:41.271527 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:41.271504 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-646677ff99-v7mvq"] Nov 01 16:36:41.273734 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:36:41.273708 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod372f3c14_1225_451c_ae57_e0b295b31364.slice/crio-42fb99fda743be43706f4c4f623cdc2c86cc61badbcae8d400908b092ff33267 WatchSource:0}: Error finding container 42fb99fda743be43706f4c4f623cdc2c86cc61badbcae8d400908b092ff33267: Status 404 returned error can't find the container with id 42fb99fda743be43706f4c4f623cdc2c86cc61badbcae8d400908b092ff33267 Nov 01 16:36:42.130519 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:42.130476 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-646677ff99-v7mvq" event={"ID":"372f3c14-1225-451c-ae57-e0b295b31364","Type":"ContainerStarted","Data":"ed2d5e7fb6a4d80c524b05ede0ec1de8381ff0b0dfe50f279abb2df2c4d934cf"} Nov 01 16:36:42.130888 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:42.130544 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-646677ff99-v7mvq" Nov 01 16:36:42.130888 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:42.130561 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-646677ff99-v7mvq" event={"ID":"372f3c14-1225-451c-ae57-e0b295b31364","Type":"ContainerStarted","Data":"42fb99fda743be43706f4c4f623cdc2c86cc61badbcae8d400908b092ff33267"} Nov 01 16:36:42.148617 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:42.148523 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-646677ff99-v7mvq" podStartSLOduration=1.540003207 podStartE2EDuration="2.148507199s" totalImagesPullingTime="608.503992ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:36:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:36:42.147608068 +0000 UTC m=+2935.479345723" watchObservedRunningTime="2026-11-01 16:36:42.148507199 +0000 UTC m=+2935.480244851" Nov 01 16:36:48.307577 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:36:48.307538 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:36:51.894240 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:51.894210 4363 scope.go:122] "RemoveContainer" containerID="c5559907747a696a36e741066171b1c3ee79328c4c0ee47ab89fec29e0cb0ca9" Nov 01 16:36:53.137937 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:36:53.137908 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-646677ff99-v7mvq" Nov 01 16:37:18.307417 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:37:18.307381 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:37:22.028011 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:37:22.027971 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8b7b001-lkbmv"] Nov 01 16:37:22.029933 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:37:22.029907 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8b7b001-lkbmv"] Nov 01 16:37:22.795429 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:37:22.795397 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf475632-fcc1-4d2d-a98b-65464c14f9b7" path="/var/lib/kubelet/pods/bf475632-fcc1-4d2d-a98b-65464c14f9b7/volumes" Nov 01 16:37:47.987177 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:37:47.987144 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtwhw_8254aafa-dd5b-4150-87e4-d165dab6bcb5/ovn-acl-logging/0.log" Nov 01 16:37:47.999423 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:37:47.999395 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtwhw_8254aafa-dd5b-4150-87e4-d165dab6bcb5/ovn-acl-logging/0.log" Nov 01 16:37:48.308263 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:37:48.308185 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:37:51.925499 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:37:51.925463 4363 scope.go:122] "RemoveContainer" containerID="b3593911a0b22644699a9af8e38d6f655dc34a0915679e91809926a5a6cc4102" Nov 01 16:37:51.940514 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:37:51.940478 4363 scope.go:122] "RemoveContainer" containerID="61022769ba07beb383d453bb2975b8e961d5508c7b62aa20ba254df1b3b270f5" Nov 01 16:38:18.308795 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:38:18.308700 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:38:35.326855 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.326818 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm"] Nov 01 16:38:35.331164 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.331137 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm" Nov 01 16:38:35.348684 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.348656 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n"] Nov 01 16:38:35.353299 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.353272 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n" Nov 01 16:38:35.354371 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.354345 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm"] Nov 01 16:38:35.367565 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.367537 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n"] Nov 01 16:38:35.467148 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.467109 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38489f4e-73ae-416a-a240-3e4942ffea4a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hmxgm\" (UID: \"38489f4e-73ae-416a-a240-3e4942ffea4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm" Nov 01 16:38:35.467148 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.467150 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0108f2e6-1b6d-4c70-a898-35c26d039d2a-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lfl9n\" (UID: \"0108f2e6-1b6d-4c70-a898-35c26d039d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n" Nov 01 16:38:35.467377 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.467175 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0108f2e6-1b6d-4c70-a898-35c26d039d2a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lfl9n\" (UID: \"0108f2e6-1b6d-4c70-a898-35c26d039d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n" Nov 01 16:38:35.467377 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.467273 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38489f4e-73ae-416a-a240-3e4942ffea4a-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hmxgm\" (UID: \"38489f4e-73ae-416a-a240-3e4942ffea4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm" Nov 01 16:38:35.467377 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.467300 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0108f2e6-1b6d-4c70-a898-35c26d039d2a-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lfl9n\" (UID: \"0108f2e6-1b6d-4c70-a898-35c26d039d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n" Nov 01 16:38:35.467377 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.467316 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38489f4e-73ae-416a-a240-3e4942ffea4a-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hmxgm\" (UID: \"38489f4e-73ae-416a-a240-3e4942ffea4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm" Nov 01 16:38:35.467377 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.467333 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0108f2e6-1b6d-4c70-a898-35c26d039d2a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lfl9n\" (UID: \"0108f2e6-1b6d-4c70-a898-35c26d039d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n" Nov 01 16:38:35.467550 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.467463 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0108f2e6-1b6d-4c70-a898-35c26d039d2a-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lfl9n\" (UID: \"0108f2e6-1b6d-4c70-a898-35c26d039d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n" Nov 01 16:38:35.467550 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.467488 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sqqd\" (UniqueName: \"kubernetes.io/projected/0108f2e6-1b6d-4c70-a898-35c26d039d2a-kube-api-access-6sqqd\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lfl9n\" (UID: \"0108f2e6-1b6d-4c70-a898-35c26d039d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n" Nov 01 16:38:35.467550 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.467509 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38489f4e-73ae-416a-a240-3e4942ffea4a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hmxgm\" (UID: \"38489f4e-73ae-416a-a240-3e4942ffea4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm" Nov 01 16:38:35.467550 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.467533 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0108f2e6-1b6d-4c70-a898-35c26d039d2a-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lfl9n\" (UID: \"0108f2e6-1b6d-4c70-a898-35c26d039d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n" Nov 01 16:38:35.467692 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.467597 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38489f4e-73ae-416a-a240-3e4942ffea4a-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hmxgm\" (UID: \"38489f4e-73ae-416a-a240-3e4942ffea4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm" Nov 01 16:38:35.467692 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.467630 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0108f2e6-1b6d-4c70-a898-35c26d039d2a-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lfl9n\" (UID: \"0108f2e6-1b6d-4c70-a898-35c26d039d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n" Nov 01 16:38:35.467692 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.467653 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0108f2e6-1b6d-4c70-a898-35c26d039d2a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lfl9n\" (UID: \"0108f2e6-1b6d-4c70-a898-35c26d039d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n" Nov 01 16:38:35.467692 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.467687 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btd7q\" (UniqueName: \"kubernetes.io/projected/38489f4e-73ae-416a-a240-3e4942ffea4a-kube-api-access-btd7q\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hmxgm\" (UID: \"38489f4e-73ae-416a-a240-3e4942ffea4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm" Nov 01 16:38:35.467838 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.467726 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38489f4e-73ae-416a-a240-3e4942ffea4a-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hmxgm\" (UID: \"38489f4e-73ae-416a-a240-3e4942ffea4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm" Nov 01 16:38:35.467838 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.467784 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38489f4e-73ae-416a-a240-3e4942ffea4a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hmxgm\" (UID: \"38489f4e-73ae-416a-a240-3e4942ffea4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm" Nov 01 16:38:35.467838 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.467814 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38489f4e-73ae-416a-a240-3e4942ffea4a-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hmxgm\" (UID: \"38489f4e-73ae-416a-a240-3e4942ffea4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm" Nov 01 16:38:35.569144 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.569101 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38489f4e-73ae-416a-a240-3e4942ffea4a-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hmxgm\" (UID: \"38489f4e-73ae-416a-a240-3e4942ffea4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm" Nov 01 16:38:35.569354 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.569149 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38489f4e-73ae-416a-a240-3e4942ffea4a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hmxgm\" (UID: \"38489f4e-73ae-416a-a240-3e4942ffea4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm" Nov 01 16:38:35.569354 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.569174 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0108f2e6-1b6d-4c70-a898-35c26d039d2a-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lfl9n\" (UID: \"0108f2e6-1b6d-4c70-a898-35c26d039d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n" Nov 01 16:38:35.569354 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.569208 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0108f2e6-1b6d-4c70-a898-35c26d039d2a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lfl9n\" (UID: \"0108f2e6-1b6d-4c70-a898-35c26d039d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n" Nov 01 16:38:35.569354 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.569237 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38489f4e-73ae-416a-a240-3e4942ffea4a-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hmxgm\" (UID: \"38489f4e-73ae-416a-a240-3e4942ffea4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm" Nov 01 16:38:35.569354 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.569267 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0108f2e6-1b6d-4c70-a898-35c26d039d2a-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lfl9n\" (UID: \"0108f2e6-1b6d-4c70-a898-35c26d039d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n" Nov 01 16:38:35.569354 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.569289 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38489f4e-73ae-416a-a240-3e4942ffea4a-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hmxgm\" (UID: \"38489f4e-73ae-416a-a240-3e4942ffea4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm" Nov 01 16:38:35.569354 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.569317 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0108f2e6-1b6d-4c70-a898-35c26d039d2a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lfl9n\" (UID: \"0108f2e6-1b6d-4c70-a898-35c26d039d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n" Nov 01 16:38:35.569751 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.569362 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0108f2e6-1b6d-4c70-a898-35c26d039d2a-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lfl9n\" (UID: \"0108f2e6-1b6d-4c70-a898-35c26d039d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n" Nov 01 16:38:35.569751 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.569396 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sqqd\" (UniqueName: \"kubernetes.io/projected/0108f2e6-1b6d-4c70-a898-35c26d039d2a-kube-api-access-6sqqd\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lfl9n\" (UID: \"0108f2e6-1b6d-4c70-a898-35c26d039d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n" Nov 01 16:38:35.569751 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.569431 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38489f4e-73ae-416a-a240-3e4942ffea4a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hmxgm\" (UID: \"38489f4e-73ae-416a-a240-3e4942ffea4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm" Nov 01 16:38:35.569751 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.569463 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0108f2e6-1b6d-4c70-a898-35c26d039d2a-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lfl9n\" (UID: \"0108f2e6-1b6d-4c70-a898-35c26d039d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n" Nov 01 16:38:35.569751 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.569498 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38489f4e-73ae-416a-a240-3e4942ffea4a-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hmxgm\" (UID: \"38489f4e-73ae-416a-a240-3e4942ffea4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm" Nov 01 16:38:35.569751 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.569532 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0108f2e6-1b6d-4c70-a898-35c26d039d2a-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lfl9n\" (UID: \"0108f2e6-1b6d-4c70-a898-35c26d039d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n" Nov 01 16:38:35.569751 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.569566 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0108f2e6-1b6d-4c70-a898-35c26d039d2a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lfl9n\" (UID: \"0108f2e6-1b6d-4c70-a898-35c26d039d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n" Nov 01 16:38:35.569751 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.569594 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38489f4e-73ae-416a-a240-3e4942ffea4a-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hmxgm\" (UID: \"38489f4e-73ae-416a-a240-3e4942ffea4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm" Nov 01 16:38:35.569751 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.569648 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38489f4e-73ae-416a-a240-3e4942ffea4a-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hmxgm\" (UID: \"38489f4e-73ae-416a-a240-3e4942ffea4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm" Nov 01 16:38:35.569751 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.569662 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0108f2e6-1b6d-4c70-a898-35c26d039d2a-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lfl9n\" (UID: \"0108f2e6-1b6d-4c70-a898-35c26d039d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n" Nov 01 16:38:35.569751 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.569599 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btd7q\" (UniqueName: \"kubernetes.io/projected/38489f4e-73ae-416a-a240-3e4942ffea4a-kube-api-access-btd7q\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hmxgm\" (UID: \"38489f4e-73ae-416a-a240-3e4942ffea4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm" Nov 01 16:38:35.569751 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.569701 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38489f4e-73ae-416a-a240-3e4942ffea4a-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hmxgm\" (UID: \"38489f4e-73ae-416a-a240-3e4942ffea4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm" Nov 01 16:38:35.569751 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.569728 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0108f2e6-1b6d-4c70-a898-35c26d039d2a-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lfl9n\" (UID: \"0108f2e6-1b6d-4c70-a898-35c26d039d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n" Nov 01 16:38:35.569751 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.569752 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38489f4e-73ae-416a-a240-3e4942ffea4a-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hmxgm\" (UID: \"38489f4e-73ae-416a-a240-3e4942ffea4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm" Nov 01 16:38:35.570619 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.569808 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38489f4e-73ae-416a-a240-3e4942ffea4a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hmxgm\" (UID: \"38489f4e-73ae-416a-a240-3e4942ffea4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm" Nov 01 16:38:35.570619 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.569925 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0108f2e6-1b6d-4c70-a898-35c26d039d2a-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lfl9n\" (UID: \"0108f2e6-1b6d-4c70-a898-35c26d039d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n" Nov 01 16:38:35.570619 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.569982 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0108f2e6-1b6d-4c70-a898-35c26d039d2a-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lfl9n\" (UID: \"0108f2e6-1b6d-4c70-a898-35c26d039d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n" Nov 01 16:38:35.570619 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.570100 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0108f2e6-1b6d-4c70-a898-35c26d039d2a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lfl9n\" (UID: \"0108f2e6-1b6d-4c70-a898-35c26d039d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n" Nov 01 16:38:35.570619 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.570246 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38489f4e-73ae-416a-a240-3e4942ffea4a-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hmxgm\" (UID: \"38489f4e-73ae-416a-a240-3e4942ffea4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm" Nov 01 16:38:35.570823 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.570761 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38489f4e-73ae-416a-a240-3e4942ffea4a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hmxgm\" (UID: \"38489f4e-73ae-416a-a240-3e4942ffea4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm" Nov 01 16:38:35.572245 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.572220 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38489f4e-73ae-416a-a240-3e4942ffea4a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hmxgm\" (UID: \"38489f4e-73ae-416a-a240-3e4942ffea4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm" Nov 01 16:38:35.572366 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.572320 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0108f2e6-1b6d-4c70-a898-35c26d039d2a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lfl9n\" (UID: \"0108f2e6-1b6d-4c70-a898-35c26d039d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n" Nov 01 16:38:35.572583 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.572561 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0108f2e6-1b6d-4c70-a898-35c26d039d2a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lfl9n\" (UID: \"0108f2e6-1b6d-4c70-a898-35c26d039d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n" Nov 01 16:38:35.572663 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.572646 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38489f4e-73ae-416a-a240-3e4942ffea4a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hmxgm\" (UID: \"38489f4e-73ae-416a-a240-3e4942ffea4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm" Nov 01 16:38:35.578190 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.578132 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0108f2e6-1b6d-4c70-a898-35c26d039d2a-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lfl9n\" (UID: \"0108f2e6-1b6d-4c70-a898-35c26d039d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n" Nov 01 16:38:35.578390 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.578372 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38489f4e-73ae-416a-a240-3e4942ffea4a-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hmxgm\" (UID: \"38489f4e-73ae-416a-a240-3e4942ffea4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm" Nov 01 16:38:35.578544 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.578525 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btd7q\" (UniqueName: \"kubernetes.io/projected/38489f4e-73ae-416a-a240-3e4942ffea4a-kube-api-access-btd7q\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hmxgm\" (UID: \"38489f4e-73ae-416a-a240-3e4942ffea4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm" Nov 01 16:38:35.578633 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.578616 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sqqd\" (UniqueName: \"kubernetes.io/projected/0108f2e6-1b6d-4c70-a898-35c26d039d2a-kube-api-access-6sqqd\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lfl9n\" (UID: \"0108f2e6-1b6d-4c70-a898-35c26d039d2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n" Nov 01 16:38:35.646116 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.646084 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm" Nov 01 16:38:35.669208 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.669176 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n" Nov 01 16:38:35.937694 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.937582 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm"] Nov 01 16:38:35.940231 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:38:35.940206 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38489f4e_73ae_416a_a240_3e4942ffea4a.slice/crio-27f09b87f96feca90e6b1bbc7c4a2494781ca3a35aa96a93651c4cd37ec339f7 WatchSource:0}: Error finding container 27f09b87f96feca90e6b1bbc7c4a2494781ca3a35aa96a93651c4cd37ec339f7: Status 404 returned error can't find the container with id 27f09b87f96feca90e6b1bbc7c4a2494781ca3a35aa96a93651c4cd37ec339f7 Nov 01 16:38:35.942593 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.942564 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:38:35.942683 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.942629 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:38:35.942683 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.942662 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:38:35.962954 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.962928 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n"] Nov 01 16:38:35.973154 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:38:35.973126 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0108f2e6_1b6d_4c70_a898_35c26d039d2a.slice/crio-ac087896168186f0d561fc19d58c65907d4e9f9a078739b20bbffb1a5f5182e3 WatchSource:0}: Error finding container ac087896168186f0d561fc19d58c65907d4e9f9a078739b20bbffb1a5f5182e3: Status 404 returned error can't find the container with id ac087896168186f0d561fc19d58c65907d4e9f9a078739b20bbffb1a5f5182e3 Nov 01 16:38:35.975485 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.975453 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:38:35.975590 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.975525 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:38:35.975590 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:35.975568 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:38:36.673637 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:36.673598 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm" event={"ID":"38489f4e-73ae-416a-a240-3e4942ffea4a","Type":"ContainerStarted","Data":"41e5e606ab4558cd3b502ee7d2bf2563efc3b955874f41b8363e0c5f171b6c18"} Nov 01 16:38:36.674142 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:36.673678 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm" event={"ID":"38489f4e-73ae-416a-a240-3e4942ffea4a","Type":"ContainerStarted","Data":"27f09b87f96feca90e6b1bbc7c4a2494781ca3a35aa96a93651c4cd37ec339f7"} Nov 01 16:38:36.675073 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:36.675045 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n" event={"ID":"0108f2e6-1b6d-4c70-a898-35c26d039d2a","Type":"ContainerStarted","Data":"9217c69fa181d89a7f483ace8939e8945a97de839944331ad22c1b7e6830659f"} Nov 01 16:38:36.675174 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:36.675093 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n" event={"ID":"0108f2e6-1b6d-4c70-a898-35c26d039d2a","Type":"ContainerStarted","Data":"ac087896168186f0d561fc19d58c65907d4e9f9a078739b20bbffb1a5f5182e3"} Nov 01 16:38:36.697039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:36.694843 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm" podStartSLOduration=1.694825343 podStartE2EDuration="1.694825343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:38:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:38:36.694479054 +0000 UTC m=+3050.026216706" watchObservedRunningTime="2026-11-01 16:38:36.694825343 +0000 UTC m=+3050.026562990" Nov 01 16:38:36.725040 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:36.724454 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n" podStartSLOduration=1.72443295 podStartE2EDuration="1.72443295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:38:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:38:36.723533709 +0000 UTC m=+3050.055271359" watchObservedRunningTime="2026-11-01 16:38:36.72443295 +0000 UTC m=+3050.056170601" Nov 01 16:38:37.646690 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:37.646663 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm" Nov 01 16:38:37.647782 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:37.647757 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hmxgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body= Nov 01 16:38:37.647931 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:37.647809 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm" podUID="38489f4e-73ae-416a-a240-3e4942ffea4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" Nov 01 16:38:37.669846 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:37.669787 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n" Nov 01 16:38:37.670764 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:37.670744 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lfl9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" start-of-body= Nov 01 16:38:37.670928 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:37.670802 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n" podUID="0108f2e6-1b6d-4c70-a898-35c26d039d2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" Nov 01 16:38:38.646445 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:38.646410 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hmxgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body= Nov 01 16:38:38.646852 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:38.646461 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm" podUID="38489f4e-73ae-416a-a240-3e4942ffea4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" Nov 01 16:38:38.670150 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:38.670097 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lfl9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" start-of-body= Nov 01 16:38:38.670297 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:38.670172 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n" podUID="0108f2e6-1b6d-4c70-a898-35c26d039d2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" Nov 01 16:38:39.646793 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:39.646759 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hmxgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body= Nov 01 16:38:39.647222 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:39.646815 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm" podUID="38489f4e-73ae-416a-a240-3e4942ffea4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" Nov 01 16:38:39.669815 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:39.669789 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lfl9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" start-of-body= Nov 01 16:38:39.669971 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:39.669834 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n" podUID="0108f2e6-1b6d-4c70-a898-35c26d039d2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" Nov 01 16:38:40.647272 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:40.647236 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hmxgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body= Nov 01 16:38:40.647686 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:40.647286 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm" podUID="38489f4e-73ae-416a-a240-3e4942ffea4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" Nov 01 16:38:40.670174 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:40.670150 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lfl9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" start-of-body= Nov 01 16:38:40.670294 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:40.670219 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n" podUID="0108f2e6-1b6d-4c70-a898-35c26d039d2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" Nov 01 16:38:41.646726 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:41.646686 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hmxgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body= Nov 01 16:38:41.646930 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:41.646744 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm" podUID="38489f4e-73ae-416a-a240-3e4942ffea4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" Nov 01 16:38:41.669666 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:41.669639 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lfl9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" start-of-body= Nov 01 16:38:41.669991 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:41.669689 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n" podUID="0108f2e6-1b6d-4c70-a898-35c26d039d2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" Nov 01 16:38:42.647078 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:42.647039 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hmxgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body= Nov 01 16:38:42.647274 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:42.647119 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm" podUID="38489f4e-73ae-416a-a240-3e4942ffea4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" Nov 01 16:38:42.669815 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:42.669776 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lfl9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" start-of-body= Nov 01 16:38:42.670212 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:42.669827 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n" podUID="0108f2e6-1b6d-4c70-a898-35c26d039d2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" Nov 01 16:38:44.248110 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:44.248070 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm" Nov 01 16:38:44.248906 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:44.248883 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm" Nov 01 16:38:44.250195 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:44.250169 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm" Nov 01 16:38:44.282600 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:44.282574 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n" Nov 01 16:38:44.283244 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:44.283198 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n" Nov 01 16:38:44.284480 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:44.284458 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lfl9n" Nov 01 16:38:48.356917 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:38:48.356880 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:38:51.976044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:51.966851 4363 scope.go:122] "RemoveContainer" containerID="55f86b87d59b58a53b74749d7d0aaca3981acd6c125b9a17d6f01c0f08813daa" Nov 01 16:38:52.049328 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:52.049299 4363 scope.go:122] "RemoveContainer" containerID="c83d66f9e08f683a80df6a991d434b0e18ae5dc6f7d1fd6370ad1ae2976024d0" Nov 01 16:38:52.075134 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:52.075093 4363 scope.go:122] "RemoveContainer" containerID="3f32c601b7820c9f7ce4f2745fcb8f519a0c7c0e0c8874ea66d25a0147f0c7df" Nov 01 16:38:52.132574 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:38:52.132430 4363 scope.go:122] "RemoveContainer" containerID="39ee3493f648383e85b8dc746b119e6cee1179fd6c64aa444d4dde884ab9da59" Nov 01 16:39:02.497041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:39:02.496165 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-65xj6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:39:02.497041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:39:02.496254 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-65xj6" podUID="97e26e65-03fd-4fad-99dd-a4d334b883e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:39:18.307681 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:39:18.307647 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:39:48.309511 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:39:48.309473 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:39:59.272143 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:39:59.272101 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8s26k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:39:59.272880 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:39:59.272842 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k" podUID="e0d02b9e-d3ce-4b70-9773-301792717809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:40:00.280515 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:40:00.280476 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hmxgm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:40:00.281258 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:40:00.281212 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hmxgm" podUID="38489f4e-73ae-416a-a240-3e4942ffea4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:40:18.477414 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:40:18.477370 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:40:48.306963 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:40:48.306923 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:41:18.308150 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:41:18.308065 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:41:48.424342 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:41:48.424304 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:42:14.270519 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:42:14.270480 4363 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8s26k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:42:14.271320 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:42:14.271271 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8s26k" podUID="e0d02b9e-d3ce-4b70-9773-301792717809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:42:18.309613 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:42:18.309573 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:42:25.123478 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:42:25.123438 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee2b3d-59d67jgrpx"] Nov 01 16:42:25.127367 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:42:25.127345 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee2b3d-59d67jgrpx" Nov 01 16:42:25.136747 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:42:25.136719 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-4e6474d0-dockercfg-tfskr\"" Nov 01 16:42:25.137462 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:42:25.137440 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-ee2b3d\"" Nov 01 16:42:25.143629 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:42:25.143604 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee2b3d-59d67jgrpx"] Nov 01 16:42:25.197167 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:42:25.197138 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/972521eb-7187-4d9c-9cc2-cfdead9d1ae3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ee2b3d-59d67jgrpx\" (UID: \"972521eb-7187-4d9c-9cc2-cfdead9d1ae3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee2b3d-59d67jgrpx" Nov 01 16:42:25.197328 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:42:25.197261 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mmsp\" (UniqueName: \"kubernetes.io/projected/972521eb-7187-4d9c-9cc2-cfdead9d1ae3-kube-api-access-5mmsp\") pod \"payload-pre-processing-e2e-worker-single-main-ee2b3d-59d67jgrpx\" (UID: \"972521eb-7187-4d9c-9cc2-cfdead9d1ae3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee2b3d-59d67jgrpx" Nov 01 16:42:25.213249 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:42:25.213220 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ee2b3d-58c9fb6f7rkpzn"] Nov 01 16:42:25.216736 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:42:25.216721 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee2b3d-58c9fb6f7rkpzn" Nov 01 16:42:25.240132 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:42:25.240109 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ee2b3d-58c9fb6f7rkpzn"] Nov 01 16:42:25.304522 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:42:25.300684 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mmsp\" (UniqueName: \"kubernetes.io/projected/972521eb-7187-4d9c-9cc2-cfdead9d1ae3-kube-api-access-5mmsp\") pod \"payload-pre-processing-e2e-worker-single-main-ee2b3d-59d67jgrpx\" (UID: \"972521eb-7187-4d9c-9cc2-cfdead9d1ae3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee2b3d-59d67jgrpx" Nov 01 16:42:25.304522 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:42:25.300745 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdgpc\" (UniqueName: \"kubernetes.io/projected/8a13de4c-91e5-48c2-9f5e-fb48b93d4869-kube-api-access-kdgpc\") pod \"payload-processing-e2e-worker-single-main-ee2b3d-58c9fb6f7rkpzn\" (UID: \"8a13de4c-91e5-48c2-9f5e-fb48b93d4869\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee2b3d-58c9fb6f7rkpzn" Nov 01 16:42:25.304522 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:42:25.300789 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a13de4c-91e5-48c2-9f5e-fb48b93d4869-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ee2b3d-58c9fb6f7rkpzn\" (UID: \"8a13de4c-91e5-48c2-9f5e-fb48b93d4869\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee2b3d-58c9fb6f7rkpzn" Nov 01 16:42:25.304522 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:42:25.300885 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/972521eb-7187-4d9c-9cc2-cfdead9d1ae3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ee2b3d-59d67jgrpx\" (UID: \"972521eb-7187-4d9c-9cc2-cfdead9d1ae3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee2b3d-59d67jgrpx" Nov 01 16:42:25.304522 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:42:25.301914 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/972521eb-7187-4d9c-9cc2-cfdead9d1ae3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ee2b3d-59d67jgrpx\" (UID: \"972521eb-7187-4d9c-9cc2-cfdead9d1ae3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee2b3d-59d67jgrpx" Nov 01 16:42:25.320118 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:42:25.320077 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mmsp\" (UniqueName: \"kubernetes.io/projected/972521eb-7187-4d9c-9cc2-cfdead9d1ae3-kube-api-access-5mmsp\") pod \"payload-pre-processing-e2e-worker-single-main-ee2b3d-59d67jgrpx\" (UID: \"972521eb-7187-4d9c-9cc2-cfdead9d1ae3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee2b3d-59d67jgrpx" Nov 01 16:42:25.402093 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:42:25.401987 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdgpc\" (UniqueName: \"kubernetes.io/projected/8a13de4c-91e5-48c2-9f5e-fb48b93d4869-kube-api-access-kdgpc\") pod \"payload-processing-e2e-worker-single-main-ee2b3d-58c9fb6f7rkpzn\" (UID: \"8a13de4c-91e5-48c2-9f5e-fb48b93d4869\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee2b3d-58c9fb6f7rkpzn" Nov 01 16:42:25.402093 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:42:25.402037 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a13de4c-91e5-48c2-9f5e-fb48b93d4869-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ee2b3d-58c9fb6f7rkpzn\" (UID: \"8a13de4c-91e5-48c2-9f5e-fb48b93d4869\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee2b3d-58c9fb6f7rkpzn" Nov 01 16:42:25.402846 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:42:25.402821 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a13de4c-91e5-48c2-9f5e-fb48b93d4869-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ee2b3d-58c9fb6f7rkpzn\" (UID: \"8a13de4c-91e5-48c2-9f5e-fb48b93d4869\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee2b3d-58c9fb6f7rkpzn" Nov 01 16:42:25.410612 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:42:25.410587 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdgpc\" (UniqueName: \"kubernetes.io/projected/8a13de4c-91e5-48c2-9f5e-fb48b93d4869-kube-api-access-kdgpc\") pod \"payload-processing-e2e-worker-single-main-ee2b3d-58c9fb6f7rkpzn\" (UID: \"8a13de4c-91e5-48c2-9f5e-fb48b93d4869\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee2b3d-58c9fb6f7rkpzn" Nov 01 16:42:25.445443 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:42:25.445419 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee2b3d-59d67jgrpx" Nov 01 16:42:25.529436 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:42:25.529403 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee2b3d-58c9fb6f7rkpzn" Nov 01 16:42:25.728033 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:42:25.723760 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee2b3d-59d67jgrpx"] Nov 01 16:42:25.743186 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:42:25.736388 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod972521eb_7187_4d9c_9cc2_cfdead9d1ae3.slice/crio-f5783ffd2c21a6f088589c0f22d6e34c1f57c9ee29ecc4008ddd5b246bac0623 WatchSource:0}: Error finding container f5783ffd2c21a6f088589c0f22d6e34c1f57c9ee29ecc4008ddd5b246bac0623: Status 404 returned error can't find the container with id f5783ffd2c21a6f088589c0f22d6e34c1f57c9ee29ecc4008ddd5b246bac0623 Nov 01 16:42:25.809485 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:42:25.809337 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ee2b3d-58c9fb6f7rkpzn"] Nov 01 16:42:25.812063 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:42:25.812032 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a13de4c_91e5_48c2_9f5e_fb48b93d4869.slice/crio-31d407a4dd4a0ecca49a7ca6daad50940b3ced2fd717b7a494149e588bb0e95c WatchSource:0}: Error finding container 31d407a4dd4a0ecca49a7ca6daad50940b3ced2fd717b7a494149e588bb0e95c: Status 404 returned error can't find the container with id 31d407a4dd4a0ecca49a7ca6daad50940b3ced2fd717b7a494149e588bb0e95c Nov 01 16:42:26.337738 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:42:26.337702 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee2b3d-58c9fb6f7rkpzn" event={"ID":"8a13de4c-91e5-48c2-9f5e-fb48b93d4869","Type":"ContainerStarted","Data":"09967322d3b770065a24d1ecdec1ab134df85be69aeea7450f842d3495c59c89"} Nov 01 16:42:26.338262 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:42:26.337757 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee2b3d-58c9fb6f7rkpzn" event={"ID":"8a13de4c-91e5-48c2-9f5e-fb48b93d4869","Type":"ContainerStarted","Data":"31d407a4dd4a0ecca49a7ca6daad50940b3ced2fd717b7a494149e588bb0e95c"} Nov 01 16:42:26.338262 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:42:26.338094 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee2b3d-58c9fb6f7rkpzn" Nov 01 16:42:26.339436 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:42:26.339406 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee2b3d-59d67jgrpx" event={"ID":"972521eb-7187-4d9c-9cc2-cfdead9d1ae3","Type":"ContainerStarted","Data":"4a1b8af5bdbb10a956331337a28b9bcdf885779ac42a3e3b101549ce488e4a83"} Nov 01 16:42:26.339570 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:42:26.339457 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee2b3d-59d67jgrpx" event={"ID":"972521eb-7187-4d9c-9cc2-cfdead9d1ae3","Type":"ContainerStarted","Data":"f5783ffd2c21a6f088589c0f22d6e34c1f57c9ee29ecc4008ddd5b246bac0623"} Nov 01 16:42:26.339570 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:42:26.339521 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee2b3d-59d67jgrpx" Nov 01 16:42:26.356186 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:42:26.356136 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee2b3d-58c9fb6f7rkpzn" podStartSLOduration=1.356119949 podStartE2EDuration="1.356119949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:42:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:42:26.353291062 +0000 UTC m=+3279.685028713" watchObservedRunningTime="2026-11-01 16:42:26.356119949 +0000 UTC m=+3279.687857599" Nov 01 16:42:26.379380 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:42:26.379339 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee2b3d-59d67jgrpx" podStartSLOduration=1.379326031 podStartE2EDuration="1.379326031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:42:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:42:26.377739493 +0000 UTC m=+3279.709477163" watchObservedRunningTime="2026-11-01 16:42:26.379326031 +0000 UTC m=+3279.711063682" Nov 01 16:42:37.357345 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:42:37.357268 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee2b3d-58c9fb6f7rkpzn" Nov 01 16:42:37.357345 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:42:37.357335 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee2b3d-59d67jgrpx" Nov 01 16:42:48.020625 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:42:48.020594 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtwhw_8254aafa-dd5b-4150-87e4-d165dab6bcb5/ovn-acl-logging/0.log" Nov 01 16:42:48.029164 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:42:48.029141 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtwhw_8254aafa-dd5b-4150-87e4-d165dab6bcb5/ovn-acl-logging/0.log" Nov 01 16:42:48.310982 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:42:48.310898 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:43:18.308962 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:43:18.308916 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:43:31.974708 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:43:31.974667 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ee2b3d-58c9fb6f7rkpzn"] Nov 01 16:43:32.012903 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:43:32.012871 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee2b3d-59d67jgrpx"] Nov 01 16:43:36.975202 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:43:36.975148 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee2b3d-58c9fb6f7rkpzn" podUID="8a13de4c-91e5-48c2-9f5e-fb48b93d4869" containerName="payload-processing" containerID="cri-o://09967322d3b770065a24d1ecdec1ab134df85be69aeea7450f842d3495c59c89" gracePeriod=25 Nov 01 16:43:37.013379 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:43:37.013326 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee2b3d-59d67jgrpx" podUID="972521eb-7187-4d9c-9cc2-cfdead9d1ae3" containerName="payload-pre-processing" containerID="cri-o://4a1b8af5bdbb10a956331337a28b9bcdf885779ac42a3e3b101549ce488e4a83" gracePeriod=25 Nov 01 16:43:37.355995 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:43:37.355949 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee2b3d-58c9fb6f7rkpzn" podUID="8a13de4c-91e5-48c2-9f5e-fb48b93d4869" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:43:37.355995 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:43:37.355989 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee2b3d-59d67jgrpx" podUID="972521eb-7187-4d9c-9cc2-cfdead9d1ae3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:43:47.356198 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:43:47.356155 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee2b3d-59d67jgrpx" podUID="972521eb-7187-4d9c-9cc2-cfdead9d1ae3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:43:47.356599 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:43:47.356163 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee2b3d-58c9fb6f7rkpzn" podUID="8a13de4c-91e5-48c2-9f5e-fb48b93d4869" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:43:48.308362 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:43:48.308326 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:43:57.273567 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:43:57.273537 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee2b3d-58c9fb6f7rkpzn" Nov 01 16:43:57.315243 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:43:57.315205 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee2b3d-59d67jgrpx" Nov 01 16:43:57.380855 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:43:57.380820 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/972521eb-7187-4d9c-9cc2-cfdead9d1ae3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/972521eb-7187-4d9c-9cc2-cfdead9d1ae3-plugins-config-volume\") pod \"972521eb-7187-4d9c-9cc2-cfdead9d1ae3\" (UID: \"972521eb-7187-4d9c-9cc2-cfdead9d1ae3\") " Nov 01 16:43:57.381057 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:43:57.380908 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a13de4c-91e5-48c2-9f5e-fb48b93d4869-kube-api-access-kdgpc\" (UniqueName: \"kubernetes.io/projected/8a13de4c-91e5-48c2-9f5e-fb48b93d4869-kube-api-access-kdgpc\") pod \"8a13de4c-91e5-48c2-9f5e-fb48b93d4869\" (UID: \"8a13de4c-91e5-48c2-9f5e-fb48b93d4869\") " Nov 01 16:43:57.381057 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:43:57.380939 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a13de4c-91e5-48c2-9f5e-fb48b93d4869-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a13de4c-91e5-48c2-9f5e-fb48b93d4869-plugins-config-volume\") pod \"8a13de4c-91e5-48c2-9f5e-fb48b93d4869\" (UID: \"8a13de4c-91e5-48c2-9f5e-fb48b93d4869\") " Nov 01 16:43:57.381057 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:43:57.381041 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/972521eb-7187-4d9c-9cc2-cfdead9d1ae3-kube-api-access-5mmsp\" (UniqueName: \"kubernetes.io/projected/972521eb-7187-4d9c-9cc2-cfdead9d1ae3-kube-api-access-5mmsp\") pod \"972521eb-7187-4d9c-9cc2-cfdead9d1ae3\" (UID: \"972521eb-7187-4d9c-9cc2-cfdead9d1ae3\") " Nov 01 16:43:57.381290 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:43:57.381263 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/972521eb-7187-4d9c-9cc2-cfdead9d1ae3-plugins-config-volume" pod "972521eb-7187-4d9c-9cc2-cfdead9d1ae3" (UID: "972521eb-7187-4d9c-9cc2-cfdead9d1ae3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:43:57.381360 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:43:57.381347 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a13de4c-91e5-48c2-9f5e-fb48b93d4869-plugins-config-volume" pod "8a13de4c-91e5-48c2-9f5e-fb48b93d4869" (UID: "8a13de4c-91e5-48c2-9f5e-fb48b93d4869"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:43:57.383119 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:43:57.383093 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/972521eb-7187-4d9c-9cc2-cfdead9d1ae3-kube-api-access-5mmsp" pod "972521eb-7187-4d9c-9cc2-cfdead9d1ae3" (UID: "972521eb-7187-4d9c-9cc2-cfdead9d1ae3"). InnerVolumeSpecName "kube-api-access-5mmsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:43:57.383264 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:43:57.383126 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a13de4c-91e5-48c2-9f5e-fb48b93d4869-kube-api-access-kdgpc" pod "8a13de4c-91e5-48c2-9f5e-fb48b93d4869" (UID: "8a13de4c-91e5-48c2-9f5e-fb48b93d4869"). InnerVolumeSpecName "kube-api-access-kdgpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:43:57.482543 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:43:57.482517 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/972521eb-7187-4d9c-9cc2-cfdead9d1ae3-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:43:57.482543 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:43:57.482542 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdgpc\" (UniqueName: \"kubernetes.io/projected/8a13de4c-91e5-48c2-9f5e-fb48b93d4869-kube-api-access-kdgpc\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:43:57.482731 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:43:57.482551 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a13de4c-91e5-48c2-9f5e-fb48b93d4869-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:43:57.482731 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:43:57.482560 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mmsp\" (UniqueName: \"kubernetes.io/projected/972521eb-7187-4d9c-9cc2-cfdead9d1ae3-kube-api-access-5mmsp\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:43:57.834809 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:43:57.834773 4363 generic.go:356] "Generic (PLEG): container finished" podID="8a13de4c-91e5-48c2-9f5e-fb48b93d4869" containerID="09967322d3b770065a24d1ecdec1ab134df85be69aeea7450f842d3495c59c89" exitCode=0 Nov 01 16:43:57.835005 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:43:57.834821 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee2b3d-58c9fb6f7rkpzn" Nov 01 16:43:57.835005 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:43:57.834868 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee2b3d-58c9fb6f7rkpzn" event={"ID":"8a13de4c-91e5-48c2-9f5e-fb48b93d4869","Type":"ContainerDied","Data":"09967322d3b770065a24d1ecdec1ab134df85be69aeea7450f842d3495c59c89"} Nov 01 16:43:57.835005 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:43:57.834937 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee2b3d-58c9fb6f7rkpzn" event={"ID":"8a13de4c-91e5-48c2-9f5e-fb48b93d4869","Type":"ContainerDied","Data":"31d407a4dd4a0ecca49a7ca6daad50940b3ced2fd717b7a494149e588bb0e95c"} Nov 01 16:43:57.835005 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:43:57.834968 4363 scope.go:122] "RemoveContainer" containerID="09967322d3b770065a24d1ecdec1ab134df85be69aeea7450f842d3495c59c89" Nov 01 16:43:57.836293 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:43:57.836268 4363 generic.go:356] "Generic (PLEG): container finished" podID="972521eb-7187-4d9c-9cc2-cfdead9d1ae3" containerID="4a1b8af5bdbb10a956331337a28b9bcdf885779ac42a3e3b101549ce488e4a83" exitCode=0 Nov 01 16:43:57.836408 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:43:57.836321 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee2b3d-59d67jgrpx" Nov 01 16:43:57.836408 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:43:57.836320 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee2b3d-59d67jgrpx" event={"ID":"972521eb-7187-4d9c-9cc2-cfdead9d1ae3","Type":"ContainerDied","Data":"4a1b8af5bdbb10a956331337a28b9bcdf885779ac42a3e3b101549ce488e4a83"} Nov 01 16:43:57.836525 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:43:57.836454 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee2b3d-59d67jgrpx" event={"ID":"972521eb-7187-4d9c-9cc2-cfdead9d1ae3","Type":"ContainerDied","Data":"f5783ffd2c21a6f088589c0f22d6e34c1f57c9ee29ecc4008ddd5b246bac0623"} Nov 01 16:43:57.844584 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:43:57.844553 4363 scope.go:122] "RemoveContainer" containerID="09967322d3b770065a24d1ecdec1ab134df85be69aeea7450f842d3495c59c89" Nov 01 16:43:57.844891 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:43:57.844862 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09967322d3b770065a24d1ecdec1ab134df85be69aeea7450f842d3495c59c89\": container with ID starting with 09967322d3b770065a24d1ecdec1ab134df85be69aeea7450f842d3495c59c89 not found: ID does not exist" containerID="09967322d3b770065a24d1ecdec1ab134df85be69aeea7450f842d3495c59c89" Nov 01 16:43:57.844973 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:43:57.844903 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09967322d3b770065a24d1ecdec1ab134df85be69aeea7450f842d3495c59c89"} err="failed to get container status \"09967322d3b770065a24d1ecdec1ab134df85be69aeea7450f842d3495c59c89\": rpc error: code = NotFound desc = could not find container \"09967322d3b770065a24d1ecdec1ab134df85be69aeea7450f842d3495c59c89\": container with ID starting with 09967322d3b770065a24d1ecdec1ab134df85be69aeea7450f842d3495c59c89 not found: ID does not exist" Nov 01 16:43:57.844973 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:43:57.844927 4363 scope.go:122] "RemoveContainer" containerID="4a1b8af5bdbb10a956331337a28b9bcdf885779ac42a3e3b101549ce488e4a83" Nov 01 16:43:57.853257 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:43:57.853241 4363 scope.go:122] "RemoveContainer" containerID="4a1b8af5bdbb10a956331337a28b9bcdf885779ac42a3e3b101549ce488e4a83" Nov 01 16:43:57.853552 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:43:57.853526 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a1b8af5bdbb10a956331337a28b9bcdf885779ac42a3e3b101549ce488e4a83\": container with ID starting with 4a1b8af5bdbb10a956331337a28b9bcdf885779ac42a3e3b101549ce488e4a83 not found: ID does not exist" containerID="4a1b8af5bdbb10a956331337a28b9bcdf885779ac42a3e3b101549ce488e4a83" Nov 01 16:43:57.853682 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:43:57.853554 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a1b8af5bdbb10a956331337a28b9bcdf885779ac42a3e3b101549ce488e4a83"} err="failed to get container status \"4a1b8af5bdbb10a956331337a28b9bcdf885779ac42a3e3b101549ce488e4a83\": rpc error: code = NotFound desc = could not find container \"4a1b8af5bdbb10a956331337a28b9bcdf885779ac42a3e3b101549ce488e4a83\": container with ID starting with 4a1b8af5bdbb10a956331337a28b9bcdf885779ac42a3e3b101549ce488e4a83 not found: ID does not exist" Nov 01 16:43:57.860623 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:43:57.860593 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ee2b3d-58c9fb6f7rkpzn"] Nov 01 16:43:57.863000 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:43:57.862976 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ee2b3d-58c9fb6f7rkpzn"] Nov 01 16:43:57.872697 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:43:57.872676 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee2b3d-59d67jgrpx"] Nov 01 16:43:57.876554 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:43:57.876533 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee2b3d-59d67jgrpx"] Nov 01 16:43:58.795307 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:43:58.795270 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a13de4c-91e5-48c2-9f5e-fb48b93d4869" path="/var/lib/kubelet/pods/8a13de4c-91e5-48c2-9f5e-fb48b93d4869/volumes" Nov 01 16:43:58.795817 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:43:58.795675 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="972521eb-7187-4d9c-9cc2-cfdead9d1ae3" path="/var/lib/kubelet/pods/972521eb-7187-4d9c-9cc2-cfdead9d1ae3/volumes" Nov 01 16:44:02.453324 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:44:02.453252 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v8vcm"] Nov 01 16:44:02.454078 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:44:02.454038 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v8vcm" podUID="5a739d41-eca2-41b0-8758-d8236e0eccd6" containerName="manager" containerID="cri-o://d4039a5ba251de67806e5dc589592eecc9e830cb9b0c2d32416441847aa8ae72" gracePeriod=10 Nov 01 16:44:02.704822 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:44:02.704769 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v8vcm" Nov 01 16:44:02.836766 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:44:02.836737 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a739d41-eca2-41b0-8758-d8236e0eccd6-kube-api-access-c7dqx\" (UniqueName: \"kubernetes.io/projected/5a739d41-eca2-41b0-8758-d8236e0eccd6-kube-api-access-c7dqx\") pod \"5a739d41-eca2-41b0-8758-d8236e0eccd6\" (UID: \"5a739d41-eca2-41b0-8758-d8236e0eccd6\") " Nov 01 16:44:02.836934 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:44:02.836774 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a739d41-eca2-41b0-8758-d8236e0eccd6-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5a739d41-eca2-41b0-8758-d8236e0eccd6-extensions-socket-volume\") pod \"5a739d41-eca2-41b0-8758-d8236e0eccd6\" (UID: \"5a739d41-eca2-41b0-8758-d8236e0eccd6\") " Nov 01 16:44:02.837280 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:44:02.837253 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a739d41-eca2-41b0-8758-d8236e0eccd6-extensions-socket-volume" pod "5a739d41-eca2-41b0-8758-d8236e0eccd6" (UID: "5a739d41-eca2-41b0-8758-d8236e0eccd6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:44:02.838909 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:44:02.838883 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a739d41-eca2-41b0-8758-d8236e0eccd6-kube-api-access-c7dqx" pod "5a739d41-eca2-41b0-8758-d8236e0eccd6" (UID: "5a739d41-eca2-41b0-8758-d8236e0eccd6"). InnerVolumeSpecName "kube-api-access-c7dqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:44:02.862234 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:44:02.862205 4363 generic.go:356] "Generic (PLEG): container finished" podID="5a739d41-eca2-41b0-8758-d8236e0eccd6" containerID="d4039a5ba251de67806e5dc589592eecc9e830cb9b0c2d32416441847aa8ae72" exitCode=0 Nov 01 16:44:02.862381 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:44:02.862249 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v8vcm" Nov 01 16:44:02.862381 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:44:02.862278 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v8vcm" event={"ID":"5a739d41-eca2-41b0-8758-d8236e0eccd6","Type":"ContainerDied","Data":"d4039a5ba251de67806e5dc589592eecc9e830cb9b0c2d32416441847aa8ae72"} Nov 01 16:44:02.862381 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:44:02.862330 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v8vcm" event={"ID":"5a739d41-eca2-41b0-8758-d8236e0eccd6","Type":"ContainerDied","Data":"2dbe4c678277fc6fe5a432734b97e6b4f2fb990bf4284f6b71433014047cd490"} Nov 01 16:44:02.862381 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:44:02.862358 4363 scope.go:122] "RemoveContainer" containerID="d4039a5ba251de67806e5dc589592eecc9e830cb9b0c2d32416441847aa8ae72" Nov 01 16:44:02.912910 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:44:02.912883 4363 scope.go:122] "RemoveContainer" containerID="d4039a5ba251de67806e5dc589592eecc9e830cb9b0c2d32416441847aa8ae72" Nov 01 16:44:02.913324 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:44:02.913301 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4039a5ba251de67806e5dc589592eecc9e830cb9b0c2d32416441847aa8ae72\": container with ID starting with d4039a5ba251de67806e5dc589592eecc9e830cb9b0c2d32416441847aa8ae72 not found: ID does not exist" containerID="d4039a5ba251de67806e5dc589592eecc9e830cb9b0c2d32416441847aa8ae72" Nov 01 16:44:02.913419 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:44:02.913336 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4039a5ba251de67806e5dc589592eecc9e830cb9b0c2d32416441847aa8ae72"} err="failed to get container status \"d4039a5ba251de67806e5dc589592eecc9e830cb9b0c2d32416441847aa8ae72\": rpc error: code = NotFound desc = could not find container \"d4039a5ba251de67806e5dc589592eecc9e830cb9b0c2d32416441847aa8ae72\": container with ID starting with d4039a5ba251de67806e5dc589592eecc9e830cb9b0c2d32416441847aa8ae72 not found: ID does not exist" Nov 01 16:44:02.917922 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:44:02.917893 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v8vcm"] Nov 01 16:44:02.921402 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:44:02.921381 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v8vcm"] Nov 01 16:44:02.938704 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:44:02.938680 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7dqx\" (UniqueName: \"kubernetes.io/projected/5a739d41-eca2-41b0-8758-d8236e0eccd6-kube-api-access-c7dqx\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:44:02.938704 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:44:02.938705 4363 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5a739d41-eca2-41b0-8758-d8236e0eccd6-extensions-socket-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:44:04.795713 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:44:04.795669 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a739d41-eca2-41b0-8758-d8236e0eccd6" path="/var/lib/kubelet/pods/5a739d41-eca2-41b0-8758-d8236e0eccd6/volumes" Nov 01 16:44:18.307958 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:44:18.307920 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:44:48.307099 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:44:48.307060 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:45:03.518744 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:45:03.518705 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ztr6x"] Nov 01 16:45:03.519442 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:45:03.519408 4363 cpu_manager.go:414] "containerMap: removing container" podUID="5a739d41-eca2-41b0-8758-d8236e0eccd6" containerName="manager" Nov 01 16:45:03.519442 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:45:03.519439 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a739d41-eca2-41b0-8758-d8236e0eccd6" containerName="manager" Nov 01 16:45:03.519602 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:45:03.519472 4363 cpu_manager.go:414] "containerMap: removing container" podUID="972521eb-7187-4d9c-9cc2-cfdead9d1ae3" containerName="payload-pre-processing" Nov 01 16:45:03.519602 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:45:03.519480 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="972521eb-7187-4d9c-9cc2-cfdead9d1ae3" containerName="payload-pre-processing" Nov 01 16:45:03.519602 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:45:03.519491 4363 cpu_manager.go:414] "containerMap: removing container" podUID="8a13de4c-91e5-48c2-9f5e-fb48b93d4869" containerName="payload-processing" Nov 01 16:45:03.519602 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:45:03.519499 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a13de4c-91e5-48c2-9f5e-fb48b93d4869" containerName="payload-processing" Nov 01 16:45:03.519735 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:45:03.519611 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="972521eb-7187-4d9c-9cc2-cfdead9d1ae3" containerName="payload-pre-processing" Nov 01 16:45:03.519735 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:45:03.519622 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a13de4c-91e5-48c2-9f5e-fb48b93d4869" containerName="payload-processing" Nov 01 16:45:03.519735 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:45:03.519642 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a739d41-eca2-41b0-8758-d8236e0eccd6" containerName="manager" Nov 01 16:45:03.527242 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:45:03.527219 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ztr6x"] Nov 01 16:45:03.527378 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:45:03.527321 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ztr6x" Nov 01 16:45:03.535052 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:45:03.533693 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 01 16:45:03.535052 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:45:03.533743 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-r57c6\"" Nov 01 16:45:03.535052 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:45:03.533950 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 01 16:45:03.610751 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:45:03.610710 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cfe104c9-1ec9-4536-a292-5fbe4bfb849c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-ztr6x\" (UID: \"cfe104c9-1ec9-4536-a292-5fbe4bfb849c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ztr6x" Nov 01 16:45:03.610956 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:45:03.610838 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swgxj\" (UniqueName: \"kubernetes.io/projected/cfe104c9-1ec9-4536-a292-5fbe4bfb849c-kube-api-access-swgxj\") pod \"kuadrant-operator-controller-manager-678d87766f-ztr6x\" (UID: \"cfe104c9-1ec9-4536-a292-5fbe4bfb849c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ztr6x" Nov 01 16:45:03.711721 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:45:03.711686 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swgxj\" (UniqueName: \"kubernetes.io/projected/cfe104c9-1ec9-4536-a292-5fbe4bfb849c-kube-api-access-swgxj\") pod \"kuadrant-operator-controller-manager-678d87766f-ztr6x\" (UID: \"cfe104c9-1ec9-4536-a292-5fbe4bfb849c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ztr6x" Nov 01 16:45:03.711892 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:45:03.711788 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cfe104c9-1ec9-4536-a292-5fbe4bfb849c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-ztr6x\" (UID: \"cfe104c9-1ec9-4536-a292-5fbe4bfb849c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ztr6x" Nov 01 16:45:03.712183 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:45:03.712166 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cfe104c9-1ec9-4536-a292-5fbe4bfb849c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-ztr6x\" (UID: \"cfe104c9-1ec9-4536-a292-5fbe4bfb849c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ztr6x" Nov 01 16:45:03.719702 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:45:03.719680 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swgxj\" (UniqueName: \"kubernetes.io/projected/cfe104c9-1ec9-4536-a292-5fbe4bfb849c-kube-api-access-swgxj\") pod \"kuadrant-operator-controller-manager-678d87766f-ztr6x\" (UID: \"cfe104c9-1ec9-4536-a292-5fbe4bfb849c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ztr6x" Nov 01 16:45:03.843167 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:45:03.843140 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ztr6x" Nov 01 16:45:04.126198 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:45:04.126170 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ztr6x"] Nov 01 16:45:04.128963 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:45:04.128938 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfe104c9_1ec9_4536_a292_5fbe4bfb849c.slice/crio-f40f2b9a1352c58e9a940cde6d5e75ffe0e19833a45bd3878271857df32cf050 WatchSource:0}: Error finding container f40f2b9a1352c58e9a940cde6d5e75ffe0e19833a45bd3878271857df32cf050: Status 404 returned error can't find the container with id f40f2b9a1352c58e9a940cde6d5e75ffe0e19833a45bd3878271857df32cf050 Nov 01 16:45:04.137860 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:45:04.137819 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ztr6x" event={"ID":"cfe104c9-1ec9-4536-a292-5fbe4bfb849c","Type":"ContainerStarted","Data":"f40f2b9a1352c58e9a940cde6d5e75ffe0e19833a45bd3878271857df32cf050"} Nov 01 16:45:05.144105 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:45:05.144066 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ztr6x" event={"ID":"cfe104c9-1ec9-4536-a292-5fbe4bfb849c","Type":"ContainerStarted","Data":"bc0228000c59d61b44cee2899519edb0330565e8ddcc1a6e17c723ee8affa9ec"} Nov 01 16:45:05.144542 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:45:05.144165 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ztr6x" Nov 01 16:45:05.181038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:45:05.180964 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ztr6x" podStartSLOduration=2.180945728 podStartE2EDuration="2.180945728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:45:05.179808434 +0000 UTC m=+3438.511546084" watchObservedRunningTime="2026-11-01 16:45:05.180945728 +0000 UTC m=+3438.512683380" Nov 01 16:45:16.150918 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:45:16.150890 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ztr6x" Nov 01 16:45:18.308826 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:45:18.308791 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:45:48.308543 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:45:48.308507 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:46:01.420255 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:46:01.420216 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-230d0a-5b7bc975flwttp"] Nov 01 16:46:01.423935 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:46:01.423918 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-230d0a-5b7bc975flwttp" Nov 01 16:46:01.429405 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:46:01.429378 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-230d0a\"" Nov 01 16:46:01.430879 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:46:01.430743 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-785e6019-dockercfg-ckcnl\"" Nov 01 16:46:01.434600 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:46:01.434579 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-230d0a-5b7bc975flwttp"] Nov 01 16:46:01.557374 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:46:01.557338 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvrmr\" (UniqueName: \"kubernetes.io/projected/73c8d1ad-5954-4dc2-9809-01a61299dc17-kube-api-access-tvrmr\") pod \"payload-processing-e2e-worker-single-main-230d0a-5b7bc975flwttp\" (UID: \"73c8d1ad-5954-4dc2-9809-01a61299dc17\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-230d0a-5b7bc975flwttp" Nov 01 16:46:01.557575 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:46:01.557410 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c8d1ad-5954-4dc2-9809-01a61299dc17-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-230d0a-5b7bc975flwttp\" (UID: \"73c8d1ad-5954-4dc2-9809-01a61299dc17\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-230d0a-5b7bc975flwttp" Nov 01 16:46:01.658876 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:46:01.658846 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvrmr\" (UniqueName: \"kubernetes.io/projected/73c8d1ad-5954-4dc2-9809-01a61299dc17-kube-api-access-tvrmr\") pod \"payload-processing-e2e-worker-single-main-230d0a-5b7bc975flwttp\" (UID: \"73c8d1ad-5954-4dc2-9809-01a61299dc17\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-230d0a-5b7bc975flwttp" Nov 01 16:46:01.659068 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:46:01.658894 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c8d1ad-5954-4dc2-9809-01a61299dc17-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-230d0a-5b7bc975flwttp\" (UID: \"73c8d1ad-5954-4dc2-9809-01a61299dc17\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-230d0a-5b7bc975flwttp" Nov 01 16:46:01.659619 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:46:01.659598 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c8d1ad-5954-4dc2-9809-01a61299dc17-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-230d0a-5b7bc975flwttp\" (UID: \"73c8d1ad-5954-4dc2-9809-01a61299dc17\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-230d0a-5b7bc975flwttp" Nov 01 16:46:01.666083 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:46:01.666061 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvrmr\" (UniqueName: \"kubernetes.io/projected/73c8d1ad-5954-4dc2-9809-01a61299dc17-kube-api-access-tvrmr\") pod \"payload-processing-e2e-worker-single-main-230d0a-5b7bc975flwttp\" (UID: \"73c8d1ad-5954-4dc2-9809-01a61299dc17\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-230d0a-5b7bc975flwttp" Nov 01 16:46:01.739192 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:46:01.739122 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-230d0a-5b7bc975flwttp" Nov 01 16:46:02.012453 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:46:02.012420 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-230d0a-5b7bc975flwttp"] Nov 01 16:46:02.014079 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:46:02.014049 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73c8d1ad_5954_4dc2_9809_01a61299dc17.slice/crio-d0abeda43f69c91f1f4ca41746efc7bd7de2f78154bb649300dde40cc9418dce WatchSource:0}: Error finding container d0abeda43f69c91f1f4ca41746efc7bd7de2f78154bb649300dde40cc9418dce: Status 404 returned error can't find the container with id d0abeda43f69c91f1f4ca41746efc7bd7de2f78154bb649300dde40cc9418dce Nov 01 16:46:02.488506 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:46:02.488469 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-230d0a-5b7bc975flwttp" event={"ID":"73c8d1ad-5954-4dc2-9809-01a61299dc17","Type":"ContainerStarted","Data":"e8a1b1caffb6ca6a49a2ee6f55bf735d0fa32b971c3033f05ffc78975e86d792"} Nov 01 16:46:02.488977 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:46:02.488524 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-230d0a-5b7bc975flwttp" event={"ID":"73c8d1ad-5954-4dc2-9809-01a61299dc17","Type":"ContainerStarted","Data":"d0abeda43f69c91f1f4ca41746efc7bd7de2f78154bb649300dde40cc9418dce"} Nov 01 16:46:02.488977 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:46:02.488557 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-230d0a-5b7bc975flwttp" Nov 01 16:46:02.506346 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:46:02.506287 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-230d0a-5b7bc975flwttp" podStartSLOduration=1.506269218 podStartE2EDuration="1.506269218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:46:02.5034566 +0000 UTC m=+3495.835194250" watchObservedRunningTime="2026-11-01 16:46:02.506269218 +0000 UTC m=+3495.838006870" Nov 01 16:46:13.497662 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:46:13.497632 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-230d0a-5b7bc975flwttp" Nov 01 16:46:18.308621 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:46:18.308576 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:46:48.312195 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:46:48.312154 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:47:08.164536 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:47:08.164502 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-230d0a-5b7bc975flwttp"] Nov 01 16:47:13.164842 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:47:13.164776 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-230d0a-5b7bc975flwttp" podUID="73c8d1ad-5954-4dc2-9809-01a61299dc17" containerName="payload-processing" containerID="cri-o://e8a1b1caffb6ca6a49a2ee6f55bf735d0fa32b971c3033f05ffc78975e86d792" gracePeriod=25 Nov 01 16:47:13.495001 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:47:13.494910 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-230d0a-5b7bc975flwttp" podUID="73c8d1ad-5954-4dc2-9809-01a61299dc17" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:47:18.310182 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:47:18.310139 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:47:23.496129 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:47:23.496090 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-230d0a-5b7bc975flwttp" podUID="73c8d1ad-5954-4dc2-9809-01a61299dc17" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:47:33.412953 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:47:33.412929 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-230d0a-5b7bc975flwttp" Nov 01 16:47:33.482211 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:47:33.482090 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73c8d1ad-5954-4dc2-9809-01a61299dc17-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c8d1ad-5954-4dc2-9809-01a61299dc17-plugins-config-volume\") pod \"73c8d1ad-5954-4dc2-9809-01a61299dc17\" (UID: \"73c8d1ad-5954-4dc2-9809-01a61299dc17\") " Nov 01 16:47:33.482354 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:47:33.482257 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73c8d1ad-5954-4dc2-9809-01a61299dc17-kube-api-access-tvrmr\" (UniqueName: \"kubernetes.io/projected/73c8d1ad-5954-4dc2-9809-01a61299dc17-kube-api-access-tvrmr\") pod \"73c8d1ad-5954-4dc2-9809-01a61299dc17\" (UID: \"73c8d1ad-5954-4dc2-9809-01a61299dc17\") " Nov 01 16:47:33.482422 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:47:33.482404 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73c8d1ad-5954-4dc2-9809-01a61299dc17-plugins-config-volume" pod "73c8d1ad-5954-4dc2-9809-01a61299dc17" (UID: "73c8d1ad-5954-4dc2-9809-01a61299dc17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:47:33.482598 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:47:33.482584 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c8d1ad-5954-4dc2-9809-01a61299dc17-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:47:33.484301 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:47:33.484277 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73c8d1ad-5954-4dc2-9809-01a61299dc17-kube-api-access-tvrmr" pod "73c8d1ad-5954-4dc2-9809-01a61299dc17" (UID: "73c8d1ad-5954-4dc2-9809-01a61299dc17"). InnerVolumeSpecName "kube-api-access-tvrmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:47:33.583530 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:47:33.583495 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvrmr\" (UniqueName: \"kubernetes.io/projected/73c8d1ad-5954-4dc2-9809-01a61299dc17-kube-api-access-tvrmr\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:47:33.998039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:47:33.997983 4363 generic.go:356] "Generic (PLEG): container finished" podID="73c8d1ad-5954-4dc2-9809-01a61299dc17" containerID="e8a1b1caffb6ca6a49a2ee6f55bf735d0fa32b971c3033f05ffc78975e86d792" exitCode=0 Nov 01 16:47:33.998234 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:47:33.998051 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-230d0a-5b7bc975flwttp" Nov 01 16:47:33.998234 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:47:33.998067 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-230d0a-5b7bc975flwttp" event={"ID":"73c8d1ad-5954-4dc2-9809-01a61299dc17","Type":"ContainerDied","Data":"e8a1b1caffb6ca6a49a2ee6f55bf735d0fa32b971c3033f05ffc78975e86d792"} Nov 01 16:47:33.998234 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:47:33.998123 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-230d0a-5b7bc975flwttp" event={"ID":"73c8d1ad-5954-4dc2-9809-01a61299dc17","Type":"ContainerDied","Data":"d0abeda43f69c91f1f4ca41746efc7bd7de2f78154bb649300dde40cc9418dce"} Nov 01 16:47:33.998234 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:47:33.998145 4363 scope.go:122] "RemoveContainer" containerID="e8a1b1caffb6ca6a49a2ee6f55bf735d0fa32b971c3033f05ffc78975e86d792" Nov 01 16:47:34.007567 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:47:34.007552 4363 scope.go:122] "RemoveContainer" containerID="e8a1b1caffb6ca6a49a2ee6f55bf735d0fa32b971c3033f05ffc78975e86d792" Nov 01 16:47:34.007838 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:47:34.007820 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8a1b1caffb6ca6a49a2ee6f55bf735d0fa32b971c3033f05ffc78975e86d792\": container with ID starting with e8a1b1caffb6ca6a49a2ee6f55bf735d0fa32b971c3033f05ffc78975e86d792 not found: ID does not exist" containerID="e8a1b1caffb6ca6a49a2ee6f55bf735d0fa32b971c3033f05ffc78975e86d792" Nov 01 16:47:34.007954 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:47:34.007847 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8a1b1caffb6ca6a49a2ee6f55bf735d0fa32b971c3033f05ffc78975e86d792"} err="failed to get container status \"e8a1b1caffb6ca6a49a2ee6f55bf735d0fa32b971c3033f05ffc78975e86d792\": rpc error: code = NotFound desc = could not find container \"e8a1b1caffb6ca6a49a2ee6f55bf735d0fa32b971c3033f05ffc78975e86d792\": container with ID starting with e8a1b1caffb6ca6a49a2ee6f55bf735d0fa32b971c3033f05ffc78975e86d792 not found: ID does not exist" Nov 01 16:47:34.017038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:47:34.016706 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-230d0a-5b7bc975flwttp"] Nov 01 16:47:34.018830 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:47:34.018811 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-230d0a-5b7bc975flwttp"] Nov 01 16:47:34.798477 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:47:34.798445 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73c8d1ad-5954-4dc2-9809-01a61299dc17" path="/var/lib/kubelet/pods/73c8d1ad-5954-4dc2-9809-01a61299dc17/volumes" Nov 01 16:47:48.051211 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:47:48.051186 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtwhw_8254aafa-dd5b-4150-87e4-d165dab6bcb5/ovn-acl-logging/0.log" Nov 01 16:47:48.065371 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:47:48.065348 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtwhw_8254aafa-dd5b-4150-87e4-d165dab6bcb5/ovn-acl-logging/0.log" Nov 01 16:47:48.308657 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:47:48.308562 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:48:18.446425 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:48:18.446377 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:48:46.231983 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:46.231945 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c"] Nov 01 16:48:46.232623 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:46.232460 4363 cpu_manager.go:414] "containerMap: removing container" podUID="73c8d1ad-5954-4dc2-9809-01a61299dc17" containerName="payload-processing" Nov 01 16:48:46.232623 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:46.232474 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73c8d1ad-5954-4dc2-9809-01a61299dc17" containerName="payload-processing" Nov 01 16:48:46.232623 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:46.232594 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="73c8d1ad-5954-4dc2-9809-01a61299dc17" containerName="payload-processing" Nov 01 16:48:46.236043 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:46.235981 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" Nov 01 16:48:46.253462 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:46.253432 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c"] Nov 01 16:48:46.311674 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:46.311641 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f9b881a4-openshift-default-dockercfg-gf4dc\"" Nov 01 16:48:46.381041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:46.380542 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1680feb3-7324-4c37-98f7-bc5104acd989-istiod-ca-cert\") pod \"e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c\" (UID: \"1680feb3-7324-4c37-98f7-bc5104acd989\") " pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" Nov 01 16:48:46.381041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:46.380645 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1680feb3-7324-4c37-98f7-bc5104acd989-istio-token\") pod \"e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c\" (UID: \"1680feb3-7324-4c37-98f7-bc5104acd989\") " pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" Nov 01 16:48:46.381041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:46.380695 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1680feb3-7324-4c37-98f7-bc5104acd989-credential-socket\") pod \"e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c\" (UID: \"1680feb3-7324-4c37-98f7-bc5104acd989\") " pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" Nov 01 16:48:46.381041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:46.380739 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1680feb3-7324-4c37-98f7-bc5104acd989-workload-socket\") pod \"e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c\" (UID: \"1680feb3-7324-4c37-98f7-bc5104acd989\") " pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" Nov 01 16:48:46.381041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:46.380789 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1680feb3-7324-4c37-98f7-bc5104acd989-workload-certs\") pod \"e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c\" (UID: \"1680feb3-7324-4c37-98f7-bc5104acd989\") " pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" Nov 01 16:48:46.381041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:46.380824 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1680feb3-7324-4c37-98f7-bc5104acd989-istio-envoy\") pod \"e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c\" (UID: \"1680feb3-7324-4c37-98f7-bc5104acd989\") " pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" Nov 01 16:48:46.381041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:46.380875 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1680feb3-7324-4c37-98f7-bc5104acd989-istio-data\") pod \"e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c\" (UID: \"1680feb3-7324-4c37-98f7-bc5104acd989\") " pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" Nov 01 16:48:46.381041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:46.380899 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1680feb3-7324-4c37-98f7-bc5104acd989-istio-podinfo\") pod \"e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c\" (UID: \"1680feb3-7324-4c37-98f7-bc5104acd989\") " pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" Nov 01 16:48:46.381041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:46.380927 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqhp4\" (UniqueName: \"kubernetes.io/projected/1680feb3-7324-4c37-98f7-bc5104acd989-kube-api-access-xqhp4\") pod \"e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c\" (UID: \"1680feb3-7324-4c37-98f7-bc5104acd989\") " pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" Nov 01 16:48:46.482254 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:46.482169 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1680feb3-7324-4c37-98f7-bc5104acd989-istio-data\") pod \"e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c\" (UID: \"1680feb3-7324-4c37-98f7-bc5104acd989\") " pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" Nov 01 16:48:46.482254 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:46.482200 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1680feb3-7324-4c37-98f7-bc5104acd989-istio-podinfo\") pod \"e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c\" (UID: \"1680feb3-7324-4c37-98f7-bc5104acd989\") " pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" Nov 01 16:48:46.482254 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:46.482222 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqhp4\" (UniqueName: \"kubernetes.io/projected/1680feb3-7324-4c37-98f7-bc5104acd989-kube-api-access-xqhp4\") pod \"e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c\" (UID: \"1680feb3-7324-4c37-98f7-bc5104acd989\") " pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" Nov 01 16:48:46.482254 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:46.482252 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1680feb3-7324-4c37-98f7-bc5104acd989-istiod-ca-cert\") pod \"e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c\" (UID: \"1680feb3-7324-4c37-98f7-bc5104acd989\") " pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" Nov 01 16:48:46.482624 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:46.482300 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1680feb3-7324-4c37-98f7-bc5104acd989-istio-token\") pod \"e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c\" (UID: \"1680feb3-7324-4c37-98f7-bc5104acd989\") " pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" Nov 01 16:48:46.482624 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:46.482332 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1680feb3-7324-4c37-98f7-bc5104acd989-credential-socket\") pod \"e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c\" (UID: \"1680feb3-7324-4c37-98f7-bc5104acd989\") " pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" Nov 01 16:48:46.482624 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:46.482375 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1680feb3-7324-4c37-98f7-bc5104acd989-workload-socket\") pod \"e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c\" (UID: \"1680feb3-7324-4c37-98f7-bc5104acd989\") " pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" Nov 01 16:48:46.482624 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:46.482426 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1680feb3-7324-4c37-98f7-bc5104acd989-workload-certs\") pod \"e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c\" (UID: \"1680feb3-7324-4c37-98f7-bc5104acd989\") " pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" Nov 01 16:48:46.482624 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:46.482461 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1680feb3-7324-4c37-98f7-bc5104acd989-istio-envoy\") pod \"e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c\" (UID: \"1680feb3-7324-4c37-98f7-bc5104acd989\") " pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" Nov 01 16:48:46.482925 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:46.482640 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1680feb3-7324-4c37-98f7-bc5104acd989-istio-data\") pod \"e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c\" (UID: \"1680feb3-7324-4c37-98f7-bc5104acd989\") " pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" Nov 01 16:48:46.482925 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:46.482884 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1680feb3-7324-4c37-98f7-bc5104acd989-credential-socket\") pod \"e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c\" (UID: \"1680feb3-7324-4c37-98f7-bc5104acd989\") " pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" Nov 01 16:48:46.482925 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:46.482895 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1680feb3-7324-4c37-98f7-bc5104acd989-workload-socket\") pod \"e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c\" (UID: \"1680feb3-7324-4c37-98f7-bc5104acd989\") " pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" Nov 01 16:48:46.483153 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:46.483035 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1680feb3-7324-4c37-98f7-bc5104acd989-workload-certs\") pod \"e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c\" (UID: \"1680feb3-7324-4c37-98f7-bc5104acd989\") " pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" Nov 01 16:48:46.483153 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:46.483102 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1680feb3-7324-4c37-98f7-bc5104acd989-istiod-ca-cert\") pod \"e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c\" (UID: \"1680feb3-7324-4c37-98f7-bc5104acd989\") " pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" Nov 01 16:48:46.484724 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:46.484695 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1680feb3-7324-4c37-98f7-bc5104acd989-istio-envoy\") pod \"e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c\" (UID: \"1680feb3-7324-4c37-98f7-bc5104acd989\") " pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" Nov 01 16:48:46.484923 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:46.484905 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1680feb3-7324-4c37-98f7-bc5104acd989-istio-podinfo\") pod \"e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c\" (UID: \"1680feb3-7324-4c37-98f7-bc5104acd989\") " pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" Nov 01 16:48:46.489855 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:46.489828 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqhp4\" (UniqueName: \"kubernetes.io/projected/1680feb3-7324-4c37-98f7-bc5104acd989-kube-api-access-xqhp4\") pod \"e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c\" (UID: \"1680feb3-7324-4c37-98f7-bc5104acd989\") " pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" Nov 01 16:48:46.490241 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:46.490224 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1680feb3-7324-4c37-98f7-bc5104acd989-istio-token\") pod \"e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c\" (UID: \"1680feb3-7324-4c37-98f7-bc5104acd989\") " pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" Nov 01 16:48:46.551375 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:46.551347 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" Nov 01 16:48:46.837197 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:46.837174 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c"] Nov 01 16:48:46.839387 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:48:46.839361 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1680feb3_7324_4c37_98f7_bc5104acd989.slice/crio-982775b09c0851158a3d5a675e00be9d482db45ee93f47e4877779ce9ced31ac WatchSource:0}: Error finding container 982775b09c0851158a3d5a675e00be9d482db45ee93f47e4877779ce9ced31ac: Status 404 returned error can't find the container with id 982775b09c0851158a3d5a675e00be9d482db45ee93f47e4877779ce9ced31ac Nov 01 16:48:46.841502 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:46.841471 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:48:46.841577 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:46.841534 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:48:46.841577 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:46.841562 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:48:47.520730 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:47.520691 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" event={"ID":"1680feb3-7324-4c37-98f7-bc5104acd989","Type":"ContainerStarted","Data":"306dfd2abcf54a7399d75fefef80fd397e5001624ab38d8e8867b0b8e0737faa"} Nov 01 16:48:47.521146 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:47.520772 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" event={"ID":"1680feb3-7324-4c37-98f7-bc5104acd989","Type":"ContainerStarted","Data":"982775b09c0851158a3d5a675e00be9d482db45ee93f47e4877779ce9ced31ac"} Nov 01 16:48:47.540795 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:47.540750 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" podStartSLOduration=1.540735988 podStartE2EDuration="1.540735988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:48:47.539361098 +0000 UTC m=+3660.871098749" watchObservedRunningTime="2026-11-01 16:48:47.540735988 +0000 UTC m=+3660.872473704" Nov 01 16:48:47.552398 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:47.552373 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" Nov 01 16:48:47.557441 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:47.557411 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" Nov 01 16:48:48.311191 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:48:48.311153 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:48:48.525873 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:48.525836 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" Nov 01 16:48:48.526984 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:48.526963 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" Nov 01 16:48:50.270794 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:50.270749 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f9b881a4-5d54654464-zqn8c"] Nov 01 16:48:50.275431 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:50.275402 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9b881a4-5d54654464-zqn8c" Nov 01 16:48:50.286214 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:50.286188 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f9b881a4-serving-cert\"" Nov 01 16:48:50.289875 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:50.289848 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f9b881a4-5d54654464-zqn8c"] Nov 01 16:48:50.302634 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:50.302606 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f9b881a4-54c985c9cc-w8w2f"] Nov 01 16:48:50.306429 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:50.306405 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f9b881a4-54c985c9cc-w8w2f" Nov 01 16:48:50.316506 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:50.316480 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f9b881a4-54c985c9cc-w8w2f"] Nov 01 16:48:50.323265 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:50.323231 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8xm4\" (UniqueName: \"kubernetes.io/projected/2e461c8c-2437-436a-be04-42eadbfe901f-kube-api-access-s8xm4\") pod \"payload-processing-e2e-ait-f9b881a4-54c985c9cc-w8w2f\" (UID: \"2e461c8c-2437-436a-be04-42eadbfe901f\") " pod="openshift-ingress/payload-processing-e2e-ait-f9b881a4-54c985c9cc-w8w2f" Nov 01 16:48:50.323401 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:50.323313 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e461c8c-2437-436a-be04-42eadbfe901f-plugins-config-volume\") pod \"payload-processing-e2e-ait-f9b881a4-54c985c9cc-w8w2f\" (UID: \"2e461c8c-2437-436a-be04-42eadbfe901f\") " pod="openshift-ingress/payload-processing-e2e-ait-f9b881a4-54c985c9cc-w8w2f" Nov 01 16:48:50.323461 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:50.323423 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f76mj\" (UniqueName: \"kubernetes.io/projected/c7692098-7ce2-4a8d-bca3-af994d1486e2-kube-api-access-f76mj\") pod \"maas-api-e2e-ait-f9b881a4-5d54654464-zqn8c\" (UID: \"c7692098-7ce2-4a8d-bca3-af994d1486e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9b881a4-5d54654464-zqn8c" Nov 01 16:48:50.323523 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:50.323472 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7692098-7ce2-4a8d-bca3-af994d1486e2-openshift-service-ca\") pod \"maas-api-e2e-ait-f9b881a4-5d54654464-zqn8c\" (UID: \"c7692098-7ce2-4a8d-bca3-af994d1486e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9b881a4-5d54654464-zqn8c" Nov 01 16:48:50.323523 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:50.323516 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7692098-7ce2-4a8d-bca3-af994d1486e2-metrics-certs\") pod \"maas-api-e2e-ait-f9b881a4-5d54654464-zqn8c\" (UID: \"c7692098-7ce2-4a8d-bca3-af994d1486e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9b881a4-5d54654464-zqn8c" Nov 01 16:48:50.323655 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:50.323630 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7692098-7ce2-4a8d-bca3-af994d1486e2-maas-api-tls\") pod \"maas-api-e2e-ait-f9b881a4-5d54654464-zqn8c\" (UID: \"c7692098-7ce2-4a8d-bca3-af994d1486e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9b881a4-5d54654464-zqn8c" Nov 01 16:48:50.343620 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:50.343564 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f9b881a4-dockercfg-cbsvl\"" Nov 01 16:48:50.407723 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:50.407692 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f9b881a4\"" Nov 01 16:48:50.424279 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:50.424248 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8xm4\" (UniqueName: \"kubernetes.io/projected/2e461c8c-2437-436a-be04-42eadbfe901f-kube-api-access-s8xm4\") pod \"payload-processing-e2e-ait-f9b881a4-54c985c9cc-w8w2f\" (UID: \"2e461c8c-2437-436a-be04-42eadbfe901f\") " pod="openshift-ingress/payload-processing-e2e-ait-f9b881a4-54c985c9cc-w8w2f" Nov 01 16:48:50.424409 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:50.424295 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e461c8c-2437-436a-be04-42eadbfe901f-plugins-config-volume\") pod \"payload-processing-e2e-ait-f9b881a4-54c985c9cc-w8w2f\" (UID: \"2e461c8c-2437-436a-be04-42eadbfe901f\") " pod="openshift-ingress/payload-processing-e2e-ait-f9b881a4-54c985c9cc-w8w2f" Nov 01 16:48:50.424409 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:50.424342 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f76mj\" (UniqueName: \"kubernetes.io/projected/c7692098-7ce2-4a8d-bca3-af994d1486e2-kube-api-access-f76mj\") pod \"maas-api-e2e-ait-f9b881a4-5d54654464-zqn8c\" (UID: \"c7692098-7ce2-4a8d-bca3-af994d1486e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9b881a4-5d54654464-zqn8c" Nov 01 16:48:50.424409 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:50.424375 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7692098-7ce2-4a8d-bca3-af994d1486e2-openshift-service-ca\") pod \"maas-api-e2e-ait-f9b881a4-5d54654464-zqn8c\" (UID: \"c7692098-7ce2-4a8d-bca3-af994d1486e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9b881a4-5d54654464-zqn8c" Nov 01 16:48:50.424409 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:50.424396 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7692098-7ce2-4a8d-bca3-af994d1486e2-metrics-certs\") pod \"maas-api-e2e-ait-f9b881a4-5d54654464-zqn8c\" (UID: \"c7692098-7ce2-4a8d-bca3-af994d1486e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9b881a4-5d54654464-zqn8c" Nov 01 16:48:50.424575 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:50.424525 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7692098-7ce2-4a8d-bca3-af994d1486e2-maas-api-tls\") pod \"maas-api-e2e-ait-f9b881a4-5d54654464-zqn8c\" (UID: \"c7692098-7ce2-4a8d-bca3-af994d1486e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9b881a4-5d54654464-zqn8c" Nov 01 16:48:50.425147 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:50.425122 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7692098-7ce2-4a8d-bca3-af994d1486e2-openshift-service-ca\") pod \"maas-api-e2e-ait-f9b881a4-5d54654464-zqn8c\" (UID: \"c7692098-7ce2-4a8d-bca3-af994d1486e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9b881a4-5d54654464-zqn8c" Nov 01 16:48:50.425267 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:50.425229 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e461c8c-2437-436a-be04-42eadbfe901f-plugins-config-volume\") pod \"payload-processing-e2e-ait-f9b881a4-54c985c9cc-w8w2f\" (UID: \"2e461c8c-2437-436a-be04-42eadbfe901f\") " pod="openshift-ingress/payload-processing-e2e-ait-f9b881a4-54c985c9cc-w8w2f" Nov 01 16:48:50.426883 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:50.426836 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7692098-7ce2-4a8d-bca3-af994d1486e2-maas-api-tls\") pod \"maas-api-e2e-ait-f9b881a4-5d54654464-zqn8c\" (UID: \"c7692098-7ce2-4a8d-bca3-af994d1486e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9b881a4-5d54654464-zqn8c" Nov 01 16:48:50.427094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:50.427070 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7692098-7ce2-4a8d-bca3-af994d1486e2-metrics-certs\") pod \"maas-api-e2e-ait-f9b881a4-5d54654464-zqn8c\" (UID: \"c7692098-7ce2-4a8d-bca3-af994d1486e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9b881a4-5d54654464-zqn8c" Nov 01 16:48:50.435148 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:50.435123 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f76mj\" (UniqueName: \"kubernetes.io/projected/c7692098-7ce2-4a8d-bca3-af994d1486e2-kube-api-access-f76mj\") pod \"maas-api-e2e-ait-f9b881a4-5d54654464-zqn8c\" (UID: \"c7692098-7ce2-4a8d-bca3-af994d1486e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9b881a4-5d54654464-zqn8c" Nov 01 16:48:50.435148 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:50.435142 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8xm4\" (UniqueName: \"kubernetes.io/projected/2e461c8c-2437-436a-be04-42eadbfe901f-kube-api-access-s8xm4\") pod \"payload-processing-e2e-ait-f9b881a4-54c985c9cc-w8w2f\" (UID: \"2e461c8c-2437-436a-be04-42eadbfe901f\") " pod="openshift-ingress/payload-processing-e2e-ait-f9b881a4-54c985c9cc-w8w2f" Nov 01 16:48:50.595969 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:50.595943 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9b881a4-5d54654464-zqn8c" Nov 01 16:48:50.625845 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:50.625811 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f9b881a4-54c985c9cc-w8w2f" Nov 01 16:48:50.883838 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:50.883396 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f9b881a4-5d54654464-zqn8c"] Nov 01 16:48:50.885060 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:48:50.885028 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7692098_7ce2_4a8d_bca3_af994d1486e2.slice/crio-80257f31bf5a2e923944b27ebcd23a6f14d0debf11ba9e00b940b24a1d05fe98 WatchSource:0}: Error finding container 80257f31bf5a2e923944b27ebcd23a6f14d0debf11ba9e00b940b24a1d05fe98: Status 404 returned error can't find the container with id 80257f31bf5a2e923944b27ebcd23a6f14d0debf11ba9e00b940b24a1d05fe98 Nov 01 16:48:50.886997 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:50.886976 4363 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 16:48:50.923392 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:50.923308 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f9b881a4-54c985c9cc-w8w2f"] Nov 01 16:48:51.544900 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:51.544866 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9b881a4-5d54654464-zqn8c" event={"ID":"c7692098-7ce2-4a8d-bca3-af994d1486e2","Type":"ContainerStarted","Data":"58491ec7bd6aa4dfa2bdb34330e482561f501838178a8c5ed2be3905dc9994d3"} Nov 01 16:48:51.545428 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:51.544927 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9b881a4-5d54654464-zqn8c" event={"ID":"c7692098-7ce2-4a8d-bca3-af994d1486e2","Type":"ContainerStarted","Data":"80257f31bf5a2e923944b27ebcd23a6f14d0debf11ba9e00b940b24a1d05fe98"} Nov 01 16:48:51.545428 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:51.544956 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9b881a4-5d54654464-zqn8c" Nov 01 16:48:51.550351 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:51.548588 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f9b881a4-54c985c9cc-w8w2f" event={"ID":"2e461c8c-2437-436a-be04-42eadbfe901f","Type":"ContainerStarted","Data":"ec989966880fde2be2cd1b6f9e87cbc48019f89e88038cb165d90e07873c9591"} Nov 01 16:48:51.550351 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:51.548654 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f9b881a4-54c985c9cc-w8w2f" event={"ID":"2e461c8c-2437-436a-be04-42eadbfe901f","Type":"ContainerStarted","Data":"a9f60f417144af57c8129d985025ce65103b5f3b373d8432de9e57a1f8c4394b"} Nov 01 16:48:51.550351 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:51.549246 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f9b881a4-54c985c9cc-w8w2f" Nov 01 16:48:51.562383 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:51.562285 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9b881a4-5d54654464-zqn8c" podStartSLOduration=1.144916201 podStartE2EDuration="1.562269018s" totalImagesPullingTime="417.352817ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:48:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:48:51.562098396 +0000 UTC m=+3664.893836047" watchObservedRunningTime="2026-11-01 16:48:51.562269018 +0000 UTC m=+3664.894006668" Nov 01 16:48:51.581031 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:51.580983 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f9b881a4-54c985c9cc-w8w2f" podStartSLOduration=1.580970276 podStartE2EDuration="1.580970276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:48:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:48:51.579494727 +0000 UTC m=+3664.911232378" watchObservedRunningTime="2026-11-01 16:48:51.580970276 +0000 UTC m=+3664.912707926" Nov 01 16:48:55.064424 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:55.064388 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-655f6fc5df-b8pd9"] Nov 01 16:48:55.064923 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:55.064638 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-655f6fc5df-b8pd9" podUID="086dd0e1-6a4f-45dd-9b53-f78094bda8ab" containerName="maas-api" containerID="cri-o://684d58f509a37ff570fa897e0325c5676ce73e96138ae58aa86bc2ac582b193c" gracePeriod=30 Nov 01 16:48:56.376837 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:56.376782 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-655f6fc5df-b8pd9" Nov 01 16:48:56.434087 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:56.434055 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/086dd0e1-6a4f-45dd-9b53-f78094bda8ab-metrics-certs\" (UniqueName: \"kubernetes.io/secret/086dd0e1-6a4f-45dd-9b53-f78094bda8ab-metrics-certs\") pod \"086dd0e1-6a4f-45dd-9b53-f78094bda8ab\" (UID: \"086dd0e1-6a4f-45dd-9b53-f78094bda8ab\") " Nov 01 16:48:56.434270 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:56.434175 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/086dd0e1-6a4f-45dd-9b53-f78094bda8ab-kube-api-access-kk2km\" (UniqueName: \"kubernetes.io/projected/086dd0e1-6a4f-45dd-9b53-f78094bda8ab-kube-api-access-kk2km\") pod \"086dd0e1-6a4f-45dd-9b53-f78094bda8ab\" (UID: \"086dd0e1-6a4f-45dd-9b53-f78094bda8ab\") " Nov 01 16:48:56.434270 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:56.434212 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/086dd0e1-6a4f-45dd-9b53-f78094bda8ab-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/086dd0e1-6a4f-45dd-9b53-f78094bda8ab-openshift-service-ca\") pod \"086dd0e1-6a4f-45dd-9b53-f78094bda8ab\" (UID: \"086dd0e1-6a4f-45dd-9b53-f78094bda8ab\") " Nov 01 16:48:56.434270 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:56.434244 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/086dd0e1-6a4f-45dd-9b53-f78094bda8ab-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/086dd0e1-6a4f-45dd-9b53-f78094bda8ab-maas-api-tls\") pod \"086dd0e1-6a4f-45dd-9b53-f78094bda8ab\" (UID: \"086dd0e1-6a4f-45dd-9b53-f78094bda8ab\") " Nov 01 16:48:56.434658 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:56.434628 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/086dd0e1-6a4f-45dd-9b53-f78094bda8ab-openshift-service-ca" pod "086dd0e1-6a4f-45dd-9b53-f78094bda8ab" (UID: "086dd0e1-6a4f-45dd-9b53-f78094bda8ab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:48:56.436419 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:56.436393 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/086dd0e1-6a4f-45dd-9b53-f78094bda8ab-kube-api-access-kk2km" pod "086dd0e1-6a4f-45dd-9b53-f78094bda8ab" (UID: "086dd0e1-6a4f-45dd-9b53-f78094bda8ab"). InnerVolumeSpecName "kube-api-access-kk2km". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:48:56.436537 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:56.436483 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/086dd0e1-6a4f-45dd-9b53-f78094bda8ab-maas-api-tls" pod "086dd0e1-6a4f-45dd-9b53-f78094bda8ab" (UID: "086dd0e1-6a4f-45dd-9b53-f78094bda8ab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:48:56.438638 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:56.438608 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/086dd0e1-6a4f-45dd-9b53-f78094bda8ab-metrics-certs" pod "086dd0e1-6a4f-45dd-9b53-f78094bda8ab" (UID: "086dd0e1-6a4f-45dd-9b53-f78094bda8ab"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:48:56.535638 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:56.535604 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk2km\" (UniqueName: \"kubernetes.io/projected/086dd0e1-6a4f-45dd-9b53-f78094bda8ab-kube-api-access-kk2km\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:48:56.535638 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:56.535635 4363 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/086dd0e1-6a4f-45dd-9b53-f78094bda8ab-openshift-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:48:56.535638 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:56.535647 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/086dd0e1-6a4f-45dd-9b53-f78094bda8ab-maas-api-tls\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:48:56.535888 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:56.535656 4363 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/086dd0e1-6a4f-45dd-9b53-f78094bda8ab-metrics-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:48:56.608313 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:56.608277 4363 generic.go:356] "Generic (PLEG): container finished" podID="086dd0e1-6a4f-45dd-9b53-f78094bda8ab" containerID="684d58f509a37ff570fa897e0325c5676ce73e96138ae58aa86bc2ac582b193c" exitCode=0 Nov 01 16:48:56.608524 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:56.608336 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-655f6fc5df-b8pd9" Nov 01 16:48:56.608524 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:56.608367 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-655f6fc5df-b8pd9" event={"ID":"086dd0e1-6a4f-45dd-9b53-f78094bda8ab","Type":"ContainerDied","Data":"684d58f509a37ff570fa897e0325c5676ce73e96138ae58aa86bc2ac582b193c"} Nov 01 16:48:56.608524 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:56.608446 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-655f6fc5df-b8pd9" event={"ID":"086dd0e1-6a4f-45dd-9b53-f78094bda8ab","Type":"ContainerDied","Data":"e3c39e19124bb109887ba9c093a0fed505e193289761eb9c1f81224082158cce"} Nov 01 16:48:56.608524 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:56.608480 4363 scope.go:122] "RemoveContainer" containerID="684d58f509a37ff570fa897e0325c5676ce73e96138ae58aa86bc2ac582b193c" Nov 01 16:48:56.619006 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:56.618991 4363 scope.go:122] "RemoveContainer" containerID="684d58f509a37ff570fa897e0325c5676ce73e96138ae58aa86bc2ac582b193c" Nov 01 16:48:56.619278 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:48:56.619258 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"684d58f509a37ff570fa897e0325c5676ce73e96138ae58aa86bc2ac582b193c\": container with ID starting with 684d58f509a37ff570fa897e0325c5676ce73e96138ae58aa86bc2ac582b193c not found: ID does not exist" containerID="684d58f509a37ff570fa897e0325c5676ce73e96138ae58aa86bc2ac582b193c" Nov 01 16:48:56.619351 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:56.619294 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"684d58f509a37ff570fa897e0325c5676ce73e96138ae58aa86bc2ac582b193c"} err="failed to get container status \"684d58f509a37ff570fa897e0325c5676ce73e96138ae58aa86bc2ac582b193c\": rpc error: code = NotFound desc = could not find container \"684d58f509a37ff570fa897e0325c5676ce73e96138ae58aa86bc2ac582b193c\": container with ID starting with 684d58f509a37ff570fa897e0325c5676ce73e96138ae58aa86bc2ac582b193c not found: ID does not exist" Nov 01 16:48:56.630647 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:56.630586 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-655f6fc5df-b8pd9"] Nov 01 16:48:56.632298 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:56.632279 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-655f6fc5df-b8pd9"] Nov 01 16:48:56.814700 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:56.814661 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="086dd0e1-6a4f-45dd-9b53-f78094bda8ab" path="/var/lib/kubelet/pods/086dd0e1-6a4f-45dd-9b53-f78094bda8ab/volumes" Nov 01 16:48:57.569525 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:48:57.569498 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9b881a4-5d54654464-zqn8c" Nov 01 16:49:00.920037 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:00.919992 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9b881a4-6pwbn"] Nov 01 16:49:00.920525 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:00.920431 4363 cpu_manager.go:414] "containerMap: removing container" podUID="086dd0e1-6a4f-45dd-9b53-f78094bda8ab" containerName="maas-api" Nov 01 16:49:00.920525 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:00.920443 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="086dd0e1-6a4f-45dd-9b53-f78094bda8ab" containerName="maas-api" Nov 01 16:49:00.920525 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:00.920505 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="086dd0e1-6a4f-45dd-9b53-f78094bda8ab" containerName="maas-api" Nov 01 16:49:00.925205 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:00.925183 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9b881a4-6pwbn" Nov 01 16:49:00.931235 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:00.931202 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9b881a4-6pwbn"] Nov 01 16:49:00.968694 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:00.968666 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:49:00.968864 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:00.968843 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vj67v\"" Nov 01 16:49:00.980207 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:00.980182 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc2ll\" (UniqueName: \"kubernetes.io/projected/bd69b355-238b-4566-a0bf-c7264fde6a08-kube-api-access-zc2ll\") pod \"maas-api-revoke-keys-e2e-ait-f9b881a4-6pwbn\" (UID: \"bd69b355-238b-4566-a0bf-c7264fde6a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9b881a4-6pwbn" Nov 01 16:49:00.980307 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:00.980240 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd69b355-238b-4566-a0bf-c7264fde6a08-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9b881a4-6pwbn\" (UID: \"bd69b355-238b-4566-a0bf-c7264fde6a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9b881a4-6pwbn" Nov 01 16:49:00.980349 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:00.980312 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd69b355-238b-4566-a0bf-c7264fde6a08-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9b881a4-6pwbn\" (UID: \"bd69b355-238b-4566-a0bf-c7264fde6a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9b881a4-6pwbn" Nov 01 16:49:01.081732 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:01.081706 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc2ll\" (UniqueName: \"kubernetes.io/projected/bd69b355-238b-4566-a0bf-c7264fde6a08-kube-api-access-zc2ll\") pod \"maas-api-revoke-keys-e2e-ait-f9b881a4-6pwbn\" (UID: \"bd69b355-238b-4566-a0bf-c7264fde6a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9b881a4-6pwbn" Nov 01 16:49:01.081857 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:01.081774 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd69b355-238b-4566-a0bf-c7264fde6a08-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9b881a4-6pwbn\" (UID: \"bd69b355-238b-4566-a0bf-c7264fde6a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9b881a4-6pwbn" Nov 01 16:49:01.081857 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:01.081812 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd69b355-238b-4566-a0bf-c7264fde6a08-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9b881a4-6pwbn\" (UID: \"bd69b355-238b-4566-a0bf-c7264fde6a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9b881a4-6pwbn" Nov 01 16:49:01.082659 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:01.082633 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd69b355-238b-4566-a0bf-c7264fde6a08-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9b881a4-6pwbn\" (UID: \"bd69b355-238b-4566-a0bf-c7264fde6a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9b881a4-6pwbn" Nov 01 16:49:01.084230 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:01.084210 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd69b355-238b-4566-a0bf-c7264fde6a08-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9b881a4-6pwbn\" (UID: \"bd69b355-238b-4566-a0bf-c7264fde6a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9b881a4-6pwbn" Nov 01 16:49:01.089281 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:01.089254 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc2ll\" (UniqueName: \"kubernetes.io/projected/bd69b355-238b-4566-a0bf-c7264fde6a08-kube-api-access-zc2ll\") pod \"maas-api-revoke-keys-e2e-ait-f9b881a4-6pwbn\" (UID: \"bd69b355-238b-4566-a0bf-c7264fde6a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9b881a4-6pwbn" Nov 01 16:49:01.242471 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:01.242385 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9b881a4-6pwbn" Nov 01 16:49:01.519969 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:01.519937 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9b881a4-6pwbn"] Nov 01 16:49:01.521519 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:49:01.521477 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd69b355_238b_4566_a0bf_c7264fde6a08.slice/crio-fe7f238838a7e4ce4e7b2747138daf14a17a0f1c94c01ae3577e5d93e24405a0 WatchSource:0}: Error finding container fe7f238838a7e4ce4e7b2747138daf14a17a0f1c94c01ae3577e5d93e24405a0: Status 404 returned error can't find the container with id fe7f238838a7e4ce4e7b2747138daf14a17a0f1c94c01ae3577e5d93e24405a0 Nov 01 16:49:01.644585 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:01.644542 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9b881a4-6pwbn" event={"ID":"bd69b355-238b-4566-a0bf-c7264fde6a08","Type":"ContainerStarted","Data":"fe7f238838a7e4ce4e7b2747138daf14a17a0f1c94c01ae3577e5d93e24405a0"} Nov 01 16:49:02.651460 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:02.651371 4363 generic.go:356] "Generic (PLEG): container finished" podID="bd69b355-238b-4566-a0bf-c7264fde6a08" containerID="e5cb07a5d49371434641d4b468afc67472e276fae25355370c00e3ce164b48ec" exitCode=0 Nov 01 16:49:02.652004 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:02.651452 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9b881a4-6pwbn" event={"ID":"bd69b355-238b-4566-a0bf-c7264fde6a08","Type":"ContainerDied","Data":"e5cb07a5d49371434641d4b468afc67472e276fae25355370c00e3ce164b48ec"} Nov 01 16:49:02.653143 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:02.653123 4363 generic.go:356] "Generic (PLEG): container finished" podID="1680feb3-7324-4c37-98f7-bc5104acd989" containerID="306dfd2abcf54a7399d75fefef80fd397e5001624ab38d8e8867b0b8e0737faa" exitCode=0 Nov 01 16:49:02.653291 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:02.653150 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" event={"ID":"1680feb3-7324-4c37-98f7-bc5104acd989","Type":"ContainerDied","Data":"306dfd2abcf54a7399d75fefef80fd397e5001624ab38d8e8867b0b8e0737faa"} Nov 01 16:49:02.653505 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:02.653491 4363 scope.go:122] "RemoveContainer" containerID="306dfd2abcf54a7399d75fefef80fd397e5001624ab38d8e8867b0b8e0737faa" Nov 01 16:49:02.654789 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:02.654760 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:49:02.654901 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:02.654825 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:49:02.654901 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:02.654858 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:49:03.526627 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:03.526596 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" Nov 01 16:49:03.601053 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:03.597361 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f9b881a4-54c985c9cc-w8w2f" Nov 01 16:49:03.669685 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:03.669641 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" event={"ID":"1680feb3-7324-4c37-98f7-bc5104acd989","Type":"ContainerStarted","Data":"2954a0463e20acdb312613f1681e4bde8054d161e2d02bb343feef110c4a19dd"} Nov 01 16:49:03.866296 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:03.866270 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9b881a4-6pwbn" Nov 01 16:49:03.922364 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:03.922325 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd69b355-238b-4566-a0bf-c7264fde6a08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd69b355-238b-4566-a0bf-c7264fde6a08-maas-api-openshift-ca\") pod \"bd69b355-238b-4566-a0bf-c7264fde6a08\" (UID: \"bd69b355-238b-4566-a0bf-c7264fde6a08\") " Nov 01 16:49:03.922563 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:03.922379 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd69b355-238b-4566-a0bf-c7264fde6a08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd69b355-238b-4566-a0bf-c7264fde6a08-maas-api-xks-ca\") pod \"bd69b355-238b-4566-a0bf-c7264fde6a08\" (UID: \"bd69b355-238b-4566-a0bf-c7264fde6a08\") " Nov 01 16:49:03.922563 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:03.922423 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd69b355-238b-4566-a0bf-c7264fde6a08-kube-api-access-zc2ll\" (UniqueName: \"kubernetes.io/projected/bd69b355-238b-4566-a0bf-c7264fde6a08-kube-api-access-zc2ll\") pod \"bd69b355-238b-4566-a0bf-c7264fde6a08\" (UID: \"bd69b355-238b-4566-a0bf-c7264fde6a08\") " Nov 01 16:49:03.922799 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:03.922770 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd69b355-238b-4566-a0bf-c7264fde6a08-maas-api-openshift-ca" pod "bd69b355-238b-4566-a0bf-c7264fde6a08" (UID: "bd69b355-238b-4566-a0bf-c7264fde6a08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:49:03.924691 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:03.924661 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd69b355-238b-4566-a0bf-c7264fde6a08-maas-api-xks-ca" pod "bd69b355-238b-4566-a0bf-c7264fde6a08" (UID: "bd69b355-238b-4566-a0bf-c7264fde6a08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:49:03.925277 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:03.925243 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd69b355-238b-4566-a0bf-c7264fde6a08-kube-api-access-zc2ll" pod "bd69b355-238b-4566-a0bf-c7264fde6a08" (UID: "bd69b355-238b-4566-a0bf-c7264fde6a08"). InnerVolumeSpecName "kube-api-access-zc2ll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:49:04.023538 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:04.023498 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd69b355-238b-4566-a0bf-c7264fde6a08-maas-api-openshift-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:49:04.023538 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:04.023535 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd69b355-238b-4566-a0bf-c7264fde6a08-maas-api-xks-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:49:04.023762 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:04.023549 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc2ll\" (UniqueName: \"kubernetes.io/projected/bd69b355-238b-4566-a0bf-c7264fde6a08-kube-api-access-zc2ll\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:49:04.551962 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:04.551923 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" Nov 01 16:49:04.553324 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:04.553294 4363 patch_prober.go:29] interesting pod/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.47:15021/healthz/ready\": dial tcp 10.134.1.47:15021: connect: connection refused" start-of-body= Nov 01 16:49:04.553472 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:04.553352 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" podUID="1680feb3-7324-4c37-98f7-bc5104acd989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.47:15021/healthz/ready\": dial tcp 10.134.1.47:15021: connect: connection refused" Nov 01 16:49:04.675463 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:04.675434 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9b881a4-6pwbn" Nov 01 16:49:04.675914 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:04.675470 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9b881a4-6pwbn" event={"ID":"bd69b355-238b-4566-a0bf-c7264fde6a08","Type":"ContainerDied","Data":"fe7f238838a7e4ce4e7b2747138daf14a17a0f1c94c01ae3577e5d93e24405a0"} Nov 01 16:49:04.675914 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:04.675518 4363 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe7f238838a7e4ce4e7b2747138daf14a17a0f1c94c01ae3577e5d93e24405a0" Nov 01 16:49:05.552734 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:05.552699 4363 patch_prober.go:29] interesting pod/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.47:15021/healthz/ready\": dial tcp 10.134.1.47:15021: connect: connection refused" start-of-body= Nov 01 16:49:05.552923 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:05.552753 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" podUID="1680feb3-7324-4c37-98f7-bc5104acd989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.47:15021/healthz/ready\": dial tcp 10.134.1.47:15021: connect: connection refused" Nov 01 16:49:06.552557 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:06.552525 4363 patch_prober.go:29] interesting pod/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.47:15021/healthz/ready\": dial tcp 10.134.1.47:15021: connect: connection refused" start-of-body= Nov 01 16:49:06.552965 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:06.552577 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" podUID="1680feb3-7324-4c37-98f7-bc5104acd989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.47:15021/healthz/ready\": dial tcp 10.134.1.47:15021: connect: connection refused" Nov 01 16:49:07.551857 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:07.551821 4363 patch_prober.go:29] interesting pod/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.47:15021/healthz/ready\": dial tcp 10.134.1.47:15021: connect: connection refused" start-of-body= Nov 01 16:49:07.552105 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:07.551882 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" podUID="1680feb3-7324-4c37-98f7-bc5104acd989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.47:15021/healthz/ready\": dial tcp 10.134.1.47:15021: connect: connection refused" Nov 01 16:49:08.552119 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:08.552083 4363 patch_prober.go:29] interesting pod/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.47:15021/healthz/ready\": dial tcp 10.134.1.47:15021: connect: connection refused" start-of-body= Nov 01 16:49:08.552534 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:08.552136 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" podUID="1680feb3-7324-4c37-98f7-bc5104acd989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.47:15021/healthz/ready\": dial tcp 10.134.1.47:15021: connect: connection refused" Nov 01 16:49:09.555846 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:09.555814 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" Nov 01 16:49:09.556356 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:09.556151 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" Nov 01 16:49:09.556777 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:09.556757 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" Nov 01 16:49:10.012389 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:10.012353 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f9b881a4-54c985c9cc-w8w2f"] Nov 01 16:49:10.443755 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:10.443715 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f9b881a4-5d54654464-zqn8c"] Nov 01 16:49:10.444011 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:10.443971 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9b881a4-5d54654464-zqn8c" podUID="c7692098-7ce2-4a8d-bca3-af994d1486e2" containerName="maas-api" containerID="cri-o://58491ec7bd6aa4dfa2bdb34330e482561f501838178a8c5ed2be3905dc9994d3" gracePeriod=30 Nov 01 16:49:10.616767 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:10.616732 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9b881a4-6pwbn"] Nov 01 16:49:10.618482 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:10.618458 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9b881a4-6pwbn"] Nov 01 16:49:10.736364 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:10.733068 4363 generic.go:356] "Generic (PLEG): container finished" podID="c7692098-7ce2-4a8d-bca3-af994d1486e2" containerID="58491ec7bd6aa4dfa2bdb34330e482561f501838178a8c5ed2be3905dc9994d3" exitCode=0 Nov 01 16:49:10.736364 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:10.733987 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9b881a4-5d54654464-zqn8c" event={"ID":"c7692098-7ce2-4a8d-bca3-af994d1486e2","Type":"ContainerDied","Data":"58491ec7bd6aa4dfa2bdb34330e482561f501838178a8c5ed2be3905dc9994d3"} Nov 01 16:49:10.769157 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:10.769130 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9b881a4-5d54654464-zqn8c" Nov 01 16:49:10.798804 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:10.798111 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd69b355-238b-4566-a0bf-c7264fde6a08" path="/var/lib/kubelet/pods/bd69b355-238b-4566-a0bf-c7264fde6a08/volumes" Nov 01 16:49:10.903828 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:10.903789 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7692098-7ce2-4a8d-bca3-af994d1486e2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7692098-7ce2-4a8d-bca3-af994d1486e2-metrics-certs\") pod \"c7692098-7ce2-4a8d-bca3-af994d1486e2\" (UID: \"c7692098-7ce2-4a8d-bca3-af994d1486e2\") " Nov 01 16:49:10.904035 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:10.903846 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7692098-7ce2-4a8d-bca3-af994d1486e2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7692098-7ce2-4a8d-bca3-af994d1486e2-openshift-service-ca\") pod \"c7692098-7ce2-4a8d-bca3-af994d1486e2\" (UID: \"c7692098-7ce2-4a8d-bca3-af994d1486e2\") " Nov 01 16:49:10.904035 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:10.903920 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7692098-7ce2-4a8d-bca3-af994d1486e2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7692098-7ce2-4a8d-bca3-af994d1486e2-maas-api-tls\") pod \"c7692098-7ce2-4a8d-bca3-af994d1486e2\" (UID: \"c7692098-7ce2-4a8d-bca3-af994d1486e2\") " Nov 01 16:49:10.904035 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:10.903969 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7692098-7ce2-4a8d-bca3-af994d1486e2-kube-api-access-f76mj\" (UniqueName: \"kubernetes.io/projected/c7692098-7ce2-4a8d-bca3-af994d1486e2-kube-api-access-f76mj\") pod \"c7692098-7ce2-4a8d-bca3-af994d1486e2\" (UID: \"c7692098-7ce2-4a8d-bca3-af994d1486e2\") " Nov 01 16:49:10.904528 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:10.904474 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7692098-7ce2-4a8d-bca3-af994d1486e2-openshift-service-ca" pod "c7692098-7ce2-4a8d-bca3-af994d1486e2" (UID: "c7692098-7ce2-4a8d-bca3-af994d1486e2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:49:10.907599 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:10.907568 4363 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7692098-7ce2-4a8d-bca3-af994d1486e2-openshift-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:49:10.909742 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:10.909710 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7692098-7ce2-4a8d-bca3-af994d1486e2-metrics-certs" pod "c7692098-7ce2-4a8d-bca3-af994d1486e2" (UID: "c7692098-7ce2-4a8d-bca3-af994d1486e2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:49:10.910092 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:10.910069 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7692098-7ce2-4a8d-bca3-af994d1486e2-maas-api-tls" pod "c7692098-7ce2-4a8d-bca3-af994d1486e2" (UID: "c7692098-7ce2-4a8d-bca3-af994d1486e2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:49:10.910370 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:10.910347 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7692098-7ce2-4a8d-bca3-af994d1486e2-kube-api-access-f76mj" pod "c7692098-7ce2-4a8d-bca3-af994d1486e2" (UID: "c7692098-7ce2-4a8d-bca3-af994d1486e2"). InnerVolumeSpecName "kube-api-access-f76mj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:49:11.009120 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:11.009037 4363 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7692098-7ce2-4a8d-bca3-af994d1486e2-metrics-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:49:11.009120 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:11.009071 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7692098-7ce2-4a8d-bca3-af994d1486e2-maas-api-tls\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:49:11.009120 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:11.009087 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f76mj\" (UniqueName: \"kubernetes.io/projected/c7692098-7ce2-4a8d-bca3-af994d1486e2-kube-api-access-f76mj\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:49:11.738826 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:11.738795 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9b881a4-5d54654464-zqn8c" Nov 01 16:49:11.738826 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:11.738808 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9b881a4-5d54654464-zqn8c" event={"ID":"c7692098-7ce2-4a8d-bca3-af994d1486e2","Type":"ContainerDied","Data":"80257f31bf5a2e923944b27ebcd23a6f14d0debf11ba9e00b940b24a1d05fe98"} Nov 01 16:49:11.739370 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:11.738863 4363 scope.go:122] "RemoveContainer" containerID="58491ec7bd6aa4dfa2bdb34330e482561f501838178a8c5ed2be3905dc9994d3" Nov 01 16:49:11.760597 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:11.760568 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f9b881a4-5d54654464-zqn8c"] Nov 01 16:49:11.764439 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:11.764414 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f9b881a4-5d54654464-zqn8c"] Nov 01 16:49:12.795405 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:12.795372 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7692098-7ce2-4a8d-bca3-af994d1486e2" path="/var/lib/kubelet/pods/c7692098-7ce2-4a8d-bca3-af994d1486e2/volumes" Nov 01 16:49:15.013577 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:15.013534 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f9b881a4-54c985c9cc-w8w2f" podUID="2e461c8c-2437-436a-be04-42eadbfe901f" containerName="payload-processing" containerID="cri-o://ec989966880fde2be2cd1b6f9e87cbc48019f89e88038cb165d90e07873c9591" gracePeriod=25 Nov 01 16:49:15.265500 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:15.265439 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f9b881a4-54c985c9cc-w8w2f" Nov 01 16:49:15.356381 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:15.356348 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e461c8c-2437-436a-be04-42eadbfe901f-kube-api-access-s8xm4\" (UniqueName: \"kubernetes.io/projected/2e461c8c-2437-436a-be04-42eadbfe901f-kube-api-access-s8xm4\") pod \"2e461c8c-2437-436a-be04-42eadbfe901f\" (UID: \"2e461c8c-2437-436a-be04-42eadbfe901f\") " Nov 01 16:49:15.356557 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:15.356488 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e461c8c-2437-436a-be04-42eadbfe901f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e461c8c-2437-436a-be04-42eadbfe901f-plugins-config-volume\") pod \"2e461c8c-2437-436a-be04-42eadbfe901f\" (UID: \"2e461c8c-2437-436a-be04-42eadbfe901f\") " Nov 01 16:49:15.356851 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:15.356828 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e461c8c-2437-436a-be04-42eadbfe901f-plugins-config-volume" pod "2e461c8c-2437-436a-be04-42eadbfe901f" (UID: "2e461c8c-2437-436a-be04-42eadbfe901f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:49:15.358488 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:15.358467 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e461c8c-2437-436a-be04-42eadbfe901f-kube-api-access-s8xm4" pod "2e461c8c-2437-436a-be04-42eadbfe901f" (UID: "2e461c8c-2437-436a-be04-42eadbfe901f"). InnerVolumeSpecName "kube-api-access-s8xm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:49:15.457970 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:15.457940 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8xm4\" (UniqueName: \"kubernetes.io/projected/2e461c8c-2437-436a-be04-42eadbfe901f-kube-api-access-s8xm4\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:49:15.457970 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:15.457967 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e461c8c-2437-436a-be04-42eadbfe901f-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:49:15.759707 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:15.759672 4363 generic.go:356] "Generic (PLEG): container finished" podID="2e461c8c-2437-436a-be04-42eadbfe901f" containerID="ec989966880fde2be2cd1b6f9e87cbc48019f89e88038cb165d90e07873c9591" exitCode=0 Nov 01 16:49:15.759918 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:15.759723 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f9b881a4-54c985c9cc-w8w2f" Nov 01 16:49:15.759918 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:15.759756 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f9b881a4-54c985c9cc-w8w2f" event={"ID":"2e461c8c-2437-436a-be04-42eadbfe901f","Type":"ContainerDied","Data":"ec989966880fde2be2cd1b6f9e87cbc48019f89e88038cb165d90e07873c9591"} Nov 01 16:49:15.759918 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:15.759813 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f9b881a4-54c985c9cc-w8w2f" event={"ID":"2e461c8c-2437-436a-be04-42eadbfe901f","Type":"ContainerDied","Data":"a9f60f417144af57c8129d985025ce65103b5f3b373d8432de9e57a1f8c4394b"} Nov 01 16:49:15.759918 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:15.759837 4363 scope.go:122] "RemoveContainer" containerID="ec989966880fde2be2cd1b6f9e87cbc48019f89e88038cb165d90e07873c9591" Nov 01 16:49:15.768727 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:15.768712 4363 scope.go:122] "RemoveContainer" containerID="ec989966880fde2be2cd1b6f9e87cbc48019f89e88038cb165d90e07873c9591" Nov 01 16:49:15.768969 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:49:15.768952 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec989966880fde2be2cd1b6f9e87cbc48019f89e88038cb165d90e07873c9591\": container with ID starting with ec989966880fde2be2cd1b6f9e87cbc48019f89e88038cb165d90e07873c9591 not found: ID does not exist" containerID="ec989966880fde2be2cd1b6f9e87cbc48019f89e88038cb165d90e07873c9591" Nov 01 16:49:15.769044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:15.768977 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec989966880fde2be2cd1b6f9e87cbc48019f89e88038cb165d90e07873c9591"} err="failed to get container status \"ec989966880fde2be2cd1b6f9e87cbc48019f89e88038cb165d90e07873c9591\": rpc error: code = NotFound desc = could not find container \"ec989966880fde2be2cd1b6f9e87cbc48019f89e88038cb165d90e07873c9591\": container with ID starting with ec989966880fde2be2cd1b6f9e87cbc48019f89e88038cb165d90e07873c9591 not found: ID does not exist" Nov 01 16:49:15.779777 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:15.779747 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f9b881a4-54c985c9cc-w8w2f"] Nov 01 16:49:15.784471 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:15.784450 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f9b881a4-54c985c9cc-w8w2f"] Nov 01 16:49:16.116068 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:16.113408 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c"] Nov 01 16:49:16.116068 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:16.113675 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" podUID="1680feb3-7324-4c37-98f7-bc5104acd989" containerName="istio-proxy" containerID="cri-o://2954a0463e20acdb312613f1681e4bde8054d161e2d02bb343feef110c4a19dd" gracePeriod=30 Nov 01 16:49:16.797000 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:16.796964 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e461c8c-2437-436a-be04-42eadbfe901f" path="/var/lib/kubelet/pods/2e461c8c-2437-436a-be04-42eadbfe901f/volumes" Nov 01 16:49:17.774627 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:17.774588 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz"] Nov 01 16:49:17.775293 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:17.775036 4363 cpu_manager.go:414] "containerMap: removing container" podUID="c7692098-7ce2-4a8d-bca3-af994d1486e2" containerName="maas-api" Nov 01 16:49:17.775293 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:17.775053 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7692098-7ce2-4a8d-bca3-af994d1486e2" containerName="maas-api" Nov 01 16:49:17.775293 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:17.775080 4363 cpu_manager.go:414] "containerMap: removing container" podUID="bd69b355-238b-4566-a0bf-c7264fde6a08" containerName="revoke-keys" Nov 01 16:49:17.775293 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:17.775085 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd69b355-238b-4566-a0bf-c7264fde6a08" containerName="revoke-keys" Nov 01 16:49:17.775293 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:17.775107 4363 cpu_manager.go:414] "containerMap: removing container" podUID="2e461c8c-2437-436a-be04-42eadbfe901f" containerName="payload-processing" Nov 01 16:49:17.775293 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:17.775112 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e461c8c-2437-436a-be04-42eadbfe901f" containerName="payload-processing" Nov 01 16:49:17.775293 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:17.775178 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e461c8c-2437-436a-be04-42eadbfe901f" containerName="payload-processing" Nov 01 16:49:17.775293 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:17.775189 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7692098-7ce2-4a8d-bca3-af994d1486e2" containerName="maas-api" Nov 01 16:49:17.775293 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:17.775197 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd69b355-238b-4566-a0bf-c7264fde6a08" containerName="revoke-keys" Nov 01 16:49:17.779556 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:17.779524 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" Nov 01 16:49:17.785967 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:17.785943 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-c6e992-openshift-default-dockercfg-jqr4s\"" Nov 01 16:49:17.789585 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:17.789559 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz"] Nov 01 16:49:17.879686 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:17.879650 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/163d3f72-1667-453f-b670-db2931af40a6-istio-data\") pod \"e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz\" (UID: \"163d3f72-1667-453f-b670-db2931af40a6\") " pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" Nov 01 16:49:17.879872 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:17.879692 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/163d3f72-1667-453f-b670-db2931af40a6-credential-socket\") pod \"e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz\" (UID: \"163d3f72-1667-453f-b670-db2931af40a6\") " pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" Nov 01 16:49:17.879872 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:17.879806 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t97fj\" (UniqueName: \"kubernetes.io/projected/163d3f72-1667-453f-b670-db2931af40a6-kube-api-access-t97fj\") pod \"e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz\" (UID: \"163d3f72-1667-453f-b670-db2931af40a6\") " pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" Nov 01 16:49:17.879997 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:17.879876 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/163d3f72-1667-453f-b670-db2931af40a6-istiod-ca-cert\") pod \"e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz\" (UID: \"163d3f72-1667-453f-b670-db2931af40a6\") " pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" Nov 01 16:49:17.879997 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:17.879966 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/163d3f72-1667-453f-b670-db2931af40a6-istio-envoy\") pod \"e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz\" (UID: \"163d3f72-1667-453f-b670-db2931af40a6\") " pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" Nov 01 16:49:17.880141 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:17.880038 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/163d3f72-1667-453f-b670-db2931af40a6-istio-token\") pod \"e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz\" (UID: \"163d3f72-1667-453f-b670-db2931af40a6\") " pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" Nov 01 16:49:17.880141 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:17.880070 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/163d3f72-1667-453f-b670-db2931af40a6-istio-podinfo\") pod \"e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz\" (UID: \"163d3f72-1667-453f-b670-db2931af40a6\") " pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" Nov 01 16:49:17.880241 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:17.880162 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/163d3f72-1667-453f-b670-db2931af40a6-workload-certs\") pod \"e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz\" (UID: \"163d3f72-1667-453f-b670-db2931af40a6\") " pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" Nov 01 16:49:17.880282 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:17.880249 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/163d3f72-1667-453f-b670-db2931af40a6-workload-socket\") pod \"e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz\" (UID: \"163d3f72-1667-453f-b670-db2931af40a6\") " pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" Nov 01 16:49:17.983048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:17.981642 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/163d3f72-1667-453f-b670-db2931af40a6-istiod-ca-cert\") pod \"e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz\" (UID: \"163d3f72-1667-453f-b670-db2931af40a6\") " pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" Nov 01 16:49:17.983048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:17.981713 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/163d3f72-1667-453f-b670-db2931af40a6-istio-envoy\") pod \"e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz\" (UID: \"163d3f72-1667-453f-b670-db2931af40a6\") " pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" Nov 01 16:49:17.983048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:17.981756 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/163d3f72-1667-453f-b670-db2931af40a6-istio-token\") pod \"e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz\" (UID: \"163d3f72-1667-453f-b670-db2931af40a6\") " pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" Nov 01 16:49:17.983048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:17.981784 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/163d3f72-1667-453f-b670-db2931af40a6-istio-podinfo\") pod \"e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz\" (UID: \"163d3f72-1667-453f-b670-db2931af40a6\") " pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" Nov 01 16:49:17.983048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:17.981846 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/163d3f72-1667-453f-b670-db2931af40a6-workload-certs\") pod \"e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz\" (UID: \"163d3f72-1667-453f-b670-db2931af40a6\") " pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" Nov 01 16:49:17.983048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:17.981923 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/163d3f72-1667-453f-b670-db2931af40a6-workload-socket\") pod \"e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz\" (UID: \"163d3f72-1667-453f-b670-db2931af40a6\") " pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" Nov 01 16:49:17.983048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:17.981964 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/163d3f72-1667-453f-b670-db2931af40a6-istio-data\") pod \"e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz\" (UID: \"163d3f72-1667-453f-b670-db2931af40a6\") " pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" Nov 01 16:49:17.983048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:17.981992 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/163d3f72-1667-453f-b670-db2931af40a6-credential-socket\") pod \"e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz\" (UID: \"163d3f72-1667-453f-b670-db2931af40a6\") " pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" Nov 01 16:49:17.983048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:17.982075 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t97fj\" (UniqueName: \"kubernetes.io/projected/163d3f72-1667-453f-b670-db2931af40a6-kube-api-access-t97fj\") pod \"e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz\" (UID: \"163d3f72-1667-453f-b670-db2931af40a6\") " pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" Nov 01 16:49:17.983048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:17.982958 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/163d3f72-1667-453f-b670-db2931af40a6-istiod-ca-cert\") pod \"e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz\" (UID: \"163d3f72-1667-453f-b670-db2931af40a6\") " pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" Nov 01 16:49:17.984399 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:17.984372 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/163d3f72-1667-453f-b670-db2931af40a6-workload-socket\") pod \"e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz\" (UID: \"163d3f72-1667-453f-b670-db2931af40a6\") " pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" Nov 01 16:49:17.986114 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:17.984813 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/163d3f72-1667-453f-b670-db2931af40a6-workload-certs\") pod \"e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz\" (UID: \"163d3f72-1667-453f-b670-db2931af40a6\") " pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" Nov 01 16:49:17.986237 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:17.984838 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/163d3f72-1667-453f-b670-db2931af40a6-istio-data\") pod \"e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz\" (UID: \"163d3f72-1667-453f-b670-db2931af40a6\") " pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" Nov 01 16:49:17.986237 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:17.984890 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/163d3f72-1667-453f-b670-db2931af40a6-credential-socket\") pod \"e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz\" (UID: \"163d3f72-1667-453f-b670-db2931af40a6\") " pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" Nov 01 16:49:17.987074 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:17.987051 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/163d3f72-1667-453f-b670-db2931af40a6-istio-envoy\") pod \"e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz\" (UID: \"163d3f72-1667-453f-b670-db2931af40a6\") " pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" Nov 01 16:49:17.987344 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:17.987324 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/163d3f72-1667-453f-b670-db2931af40a6-istio-podinfo\") pod \"e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz\" (UID: \"163d3f72-1667-453f-b670-db2931af40a6\") " pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" Nov 01 16:49:17.990209 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:17.990189 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t97fj\" (UniqueName: \"kubernetes.io/projected/163d3f72-1667-453f-b670-db2931af40a6-kube-api-access-t97fj\") pod \"e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz\" (UID: \"163d3f72-1667-453f-b670-db2931af40a6\") " pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" Nov 01 16:49:17.990300 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:17.990185 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/163d3f72-1667-453f-b670-db2931af40a6-istio-token\") pod \"e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz\" (UID: \"163d3f72-1667-453f-b670-db2931af40a6\") " pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" Nov 01 16:49:18.101645 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:18.101610 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" Nov 01 16:49:18.309453 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:49:18.309417 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:49:18.381519 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:18.381496 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz"] Nov 01 16:49:18.383720 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:49:18.383692 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod163d3f72_1667_453f_b670_db2931af40a6.slice/crio-0803cdcc4a3b8b293ff7bce22dba4e26af58e0dbfc408dbbe9dcab11dacdbe71 WatchSource:0}: Error finding container 0803cdcc4a3b8b293ff7bce22dba4e26af58e0dbfc408dbbe9dcab11dacdbe71: Status 404 returned error can't find the container with id 0803cdcc4a3b8b293ff7bce22dba4e26af58e0dbfc408dbbe9dcab11dacdbe71 Nov 01 16:49:18.386155 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:18.386122 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:49:18.386255 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:18.386184 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:49:18.386255 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:18.386214 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:49:18.796317 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:18.796228 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" event={"ID":"163d3f72-1667-453f-b670-db2931af40a6","Type":"ContainerStarted","Data":"f9ee4eb0037d67ee364bf9b2812ca028bf0060b5ed4339122b44a278d7e86f67"} Nov 01 16:49:18.796317 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:18.796265 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" event={"ID":"163d3f72-1667-453f-b670-db2931af40a6","Type":"ContainerStarted","Data":"0803cdcc4a3b8b293ff7bce22dba4e26af58e0dbfc408dbbe9dcab11dacdbe71"} Nov 01 16:49:18.809544 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:18.809498 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" podStartSLOduration=1.809481249 podStartE2EDuration="1.809481249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:49:18.808990263 +0000 UTC m=+3692.140727918" watchObservedRunningTime="2026-11-01 16:49:18.809481249 +0000 UTC m=+3692.141218901" Nov 01 16:49:19.102202 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:19.102169 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" Nov 01 16:49:19.107222 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:19.107196 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" Nov 01 16:49:19.795323 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:19.795290 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" Nov 01 16:49:19.796364 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:19.796343 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" Nov 01 16:49:21.368647 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:21.368622 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" Nov 01 16:49:21.519186 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:21.519090 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1680feb3-7324-4c37-98f7-bc5104acd989-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1680feb3-7324-4c37-98f7-bc5104acd989-istio-podinfo\") pod \"1680feb3-7324-4c37-98f7-bc5104acd989\" (UID: \"1680feb3-7324-4c37-98f7-bc5104acd989\") " Nov 01 16:49:21.519186 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:21.519133 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1680feb3-7324-4c37-98f7-bc5104acd989-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1680feb3-7324-4c37-98f7-bc5104acd989-workload-certs\") pod \"1680feb3-7324-4c37-98f7-bc5104acd989\" (UID: \"1680feb3-7324-4c37-98f7-bc5104acd989\") " Nov 01 16:49:21.519186 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:21.519166 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1680feb3-7324-4c37-98f7-bc5104acd989-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1680feb3-7324-4c37-98f7-bc5104acd989-credential-socket\") pod \"1680feb3-7324-4c37-98f7-bc5104acd989\" (UID: \"1680feb3-7324-4c37-98f7-bc5104acd989\") " Nov 01 16:49:21.519502 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:21.519249 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1680feb3-7324-4c37-98f7-bc5104acd989-kube-api-access-xqhp4\" (UniqueName: \"kubernetes.io/projected/1680feb3-7324-4c37-98f7-bc5104acd989-kube-api-access-xqhp4\") pod \"1680feb3-7324-4c37-98f7-bc5104acd989\" (UID: \"1680feb3-7324-4c37-98f7-bc5104acd989\") " Nov 01 16:49:21.519502 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:21.519307 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1680feb3-7324-4c37-98f7-bc5104acd989-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1680feb3-7324-4c37-98f7-bc5104acd989-istiod-ca-cert\") pod \"1680feb3-7324-4c37-98f7-bc5104acd989\" (UID: \"1680feb3-7324-4c37-98f7-bc5104acd989\") " Nov 01 16:49:21.519502 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:21.519353 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1680feb3-7324-4c37-98f7-bc5104acd989-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1680feb3-7324-4c37-98f7-bc5104acd989-istio-data\") pod \"1680feb3-7324-4c37-98f7-bc5104acd989\" (UID: \"1680feb3-7324-4c37-98f7-bc5104acd989\") " Nov 01 16:49:21.519502 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:21.519398 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1680feb3-7324-4c37-98f7-bc5104acd989-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1680feb3-7324-4c37-98f7-bc5104acd989-istio-envoy\") pod \"1680feb3-7324-4c37-98f7-bc5104acd989\" (UID: \"1680feb3-7324-4c37-98f7-bc5104acd989\") " Nov 01 16:49:21.519502 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:21.519439 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1680feb3-7324-4c37-98f7-bc5104acd989-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1680feb3-7324-4c37-98f7-bc5104acd989-workload-socket\") pod \"1680feb3-7324-4c37-98f7-bc5104acd989\" (UID: \"1680feb3-7324-4c37-98f7-bc5104acd989\") " Nov 01 16:49:21.519502 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:21.519500 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1680feb3-7324-4c37-98f7-bc5104acd989-istio-token\" (UniqueName: \"kubernetes.io/projected/1680feb3-7324-4c37-98f7-bc5104acd989-istio-token\") pod \"1680feb3-7324-4c37-98f7-bc5104acd989\" (UID: \"1680feb3-7324-4c37-98f7-bc5104acd989\") " Nov 01 16:49:21.519824 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:21.519412 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1680feb3-7324-4c37-98f7-bc5104acd989-workload-certs" pod "1680feb3-7324-4c37-98f7-bc5104acd989" (UID: "1680feb3-7324-4c37-98f7-bc5104acd989"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:49:21.519824 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:21.519472 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1680feb3-7324-4c37-98f7-bc5104acd989-credential-socket" pod "1680feb3-7324-4c37-98f7-bc5104acd989" (UID: "1680feb3-7324-4c37-98f7-bc5104acd989"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:49:21.519824 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:21.519684 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1680feb3-7324-4c37-98f7-bc5104acd989-workload-socket" pod "1680feb3-7324-4c37-98f7-bc5104acd989" (UID: "1680feb3-7324-4c37-98f7-bc5104acd989"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:49:21.519824 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:21.519707 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1680feb3-7324-4c37-98f7-bc5104acd989-istio-data" pod "1680feb3-7324-4c37-98f7-bc5104acd989" (UID: "1680feb3-7324-4c37-98f7-bc5104acd989"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:49:21.520044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:21.519864 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1680feb3-7324-4c37-98f7-bc5104acd989-istiod-ca-cert" pod "1680feb3-7324-4c37-98f7-bc5104acd989" (UID: "1680feb3-7324-4c37-98f7-bc5104acd989"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:49:21.520044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:21.519934 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1680feb3-7324-4c37-98f7-bc5104acd989-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:49:21.520044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:21.519959 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1680feb3-7324-4c37-98f7-bc5104acd989-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:49:21.520044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:21.519978 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1680feb3-7324-4c37-98f7-bc5104acd989-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:49:21.520044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:21.519990 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1680feb3-7324-4c37-98f7-bc5104acd989-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:49:21.520044 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:21.520005 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1680feb3-7324-4c37-98f7-bc5104acd989-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:49:21.521439 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:21.521416 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1680feb3-7324-4c37-98f7-bc5104acd989-istio-podinfo" pod "1680feb3-7324-4c37-98f7-bc5104acd989" (UID: "1680feb3-7324-4c37-98f7-bc5104acd989"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:49:21.521618 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:21.521597 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1680feb3-7324-4c37-98f7-bc5104acd989-istio-token" pod "1680feb3-7324-4c37-98f7-bc5104acd989" (UID: "1680feb3-7324-4c37-98f7-bc5104acd989"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:49:21.521839 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:21.521812 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1680feb3-7324-4c37-98f7-bc5104acd989-istio-envoy" pod "1680feb3-7324-4c37-98f7-bc5104acd989" (UID: "1680feb3-7324-4c37-98f7-bc5104acd989"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:49:21.521949 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:21.521838 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1680feb3-7324-4c37-98f7-bc5104acd989-kube-api-access-xqhp4" pod "1680feb3-7324-4c37-98f7-bc5104acd989" (UID: "1680feb3-7324-4c37-98f7-bc5104acd989"). InnerVolumeSpecName "kube-api-access-xqhp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:49:21.621478 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:21.621443 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1680feb3-7324-4c37-98f7-bc5104acd989-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:49:21.621478 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:21.621473 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1680feb3-7324-4c37-98f7-bc5104acd989-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:49:21.621478 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:21.621483 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1680feb3-7324-4c37-98f7-bc5104acd989-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:49:21.621713 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:21.621493 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqhp4\" (UniqueName: \"kubernetes.io/projected/1680feb3-7324-4c37-98f7-bc5104acd989-kube-api-access-xqhp4\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:49:21.806766 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:21.806681 4363 generic.go:356] "Generic (PLEG): container finished" podID="1680feb3-7324-4c37-98f7-bc5104acd989" containerID="2954a0463e20acdb312613f1681e4bde8054d161e2d02bb343feef110c4a19dd" exitCode=0 Nov 01 16:49:21.806766 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:21.806723 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" Nov 01 16:49:21.806977 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:21.806761 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" event={"ID":"1680feb3-7324-4c37-98f7-bc5104acd989","Type":"ContainerDied","Data":"2954a0463e20acdb312613f1681e4bde8054d161e2d02bb343feef110c4a19dd"} Nov 01 16:49:21.806977 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:21.806807 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c" event={"ID":"1680feb3-7324-4c37-98f7-bc5104acd989","Type":"ContainerDied","Data":"982775b09c0851158a3d5a675e00be9d482db45ee93f47e4877779ce9ced31ac"} Nov 01 16:49:21.806977 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:21.806828 4363 scope.go:122] "RemoveContainer" containerID="2954a0463e20acdb312613f1681e4bde8054d161e2d02bb343feef110c4a19dd" Nov 01 16:49:21.816237 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:21.816136 4363 scope.go:122] "RemoveContainer" containerID="306dfd2abcf54a7399d75fefef80fd397e5001624ab38d8e8867b0b8e0737faa" Nov 01 16:49:21.823907 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:21.823888 4363 scope.go:122] "RemoveContainer" containerID="2954a0463e20acdb312613f1681e4bde8054d161e2d02bb343feef110c4a19dd" Nov 01 16:49:21.824198 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:49:21.824182 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2954a0463e20acdb312613f1681e4bde8054d161e2d02bb343feef110c4a19dd\": container with ID starting with 2954a0463e20acdb312613f1681e4bde8054d161e2d02bb343feef110c4a19dd not found: ID does not exist" containerID="2954a0463e20acdb312613f1681e4bde8054d161e2d02bb343feef110c4a19dd" Nov 01 16:49:21.824255 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:21.824206 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2954a0463e20acdb312613f1681e4bde8054d161e2d02bb343feef110c4a19dd"} err="failed to get container status \"2954a0463e20acdb312613f1681e4bde8054d161e2d02bb343feef110c4a19dd\": rpc error: code = NotFound desc = could not find container \"2954a0463e20acdb312613f1681e4bde8054d161e2d02bb343feef110c4a19dd\": container with ID starting with 2954a0463e20acdb312613f1681e4bde8054d161e2d02bb343feef110c4a19dd not found: ID does not exist" Nov 01 16:49:21.824255 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:21.824224 4363 scope.go:122] "RemoveContainer" containerID="306dfd2abcf54a7399d75fefef80fd397e5001624ab38d8e8867b0b8e0737faa" Nov 01 16:49:21.824470 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:49:21.824452 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"306dfd2abcf54a7399d75fefef80fd397e5001624ab38d8e8867b0b8e0737faa\": container with ID starting with 306dfd2abcf54a7399d75fefef80fd397e5001624ab38d8e8867b0b8e0737faa not found: ID does not exist" containerID="306dfd2abcf54a7399d75fefef80fd397e5001624ab38d8e8867b0b8e0737faa" Nov 01 16:49:21.824514 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:21.824478 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"306dfd2abcf54a7399d75fefef80fd397e5001624ab38d8e8867b0b8e0737faa"} err="failed to get container status \"306dfd2abcf54a7399d75fefef80fd397e5001624ab38d8e8867b0b8e0737faa\": rpc error: code = NotFound desc = could not find container \"306dfd2abcf54a7399d75fefef80fd397e5001624ab38d8e8867b0b8e0737faa\": container with ID starting with 306dfd2abcf54a7399d75fefef80fd397e5001624ab38d8e8867b0b8e0737faa not found: ID does not exist" Nov 01 16:49:21.829384 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:21.829362 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c"] Nov 01 16:49:21.832942 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:21.832923 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f9b881a4-openshift-default-56bcf548ff-rcf7c"] Nov 01 16:49:22.259192 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.259155 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c6e992-796b4d6f6-qz4g6"] Nov 01 16:49:22.259947 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.259923 4363 cpu_manager.go:414] "containerMap: removing container" podUID="1680feb3-7324-4c37-98f7-bc5104acd989" containerName="istio-proxy" Nov 01 16:49:22.259995 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.259953 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1680feb3-7324-4c37-98f7-bc5104acd989" containerName="istio-proxy" Nov 01 16:49:22.260158 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.260141 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="1680feb3-7324-4c37-98f7-bc5104acd989" containerName="istio-proxy" Nov 01 16:49:22.260202 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.260163 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="1680feb3-7324-4c37-98f7-bc5104acd989" containerName="istio-proxy" Nov 01 16:49:22.265582 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.265556 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c6e992-796b4d6f6-qz4g6" Nov 01 16:49:22.276804 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.273598 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 16:49:22.276804 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.273837 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-c6e992-serving-cert\"" Nov 01 16:49:22.276804 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.273850 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-l5k8h\"" Nov 01 16:49:22.279589 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.279570 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 16:49:22.279690 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.279628 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 16:49:22.279832 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.279810 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c6e992-796b4d6f6-qz4g6"] Nov 01 16:49:22.298679 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.298647 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-c6e992-55b58d7f9f-zzkqk"] Nov 01 16:49:22.299339 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.299318 4363 cpu_manager.go:414] "containerMap: removing container" podUID="1680feb3-7324-4c37-98f7-bc5104acd989" containerName="istio-proxy" Nov 01 16:49:22.299410 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.299343 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1680feb3-7324-4c37-98f7-bc5104acd989" containerName="istio-proxy" Nov 01 16:49:22.303074 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.303054 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-c6e992-55b58d7f9f-zzkqk" Nov 01 16:49:22.311682 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.311655 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-c6e992-55b58d7f9f-zzkqk"] Nov 01 16:49:22.319820 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.319791 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-c6e992-6d6f87545f-k7nxk"] Nov 01 16:49:22.323646 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.323622 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c6e992-6d6f87545f-k7nxk" Nov 01 16:49:22.342008 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.341877 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-c6e992-6d6f87545f-k7nxk"] Nov 01 16:49:22.354314 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.354291 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-c6e992-dockercfg-rgxld\"" Nov 01 16:49:22.370109 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.370085 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-c6e992\"" Nov 01 16:49:22.431313 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.431273 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5h5s\" (UniqueName: \"kubernetes.io/projected/f2a6315e-bd0c-4e81-8889-bd959a700d72-kube-api-access-j5h5s\") pod \"payload-processing-e2e-authcfg-http-c6e992-55b58d7f9f-zzkqk\" (UID: \"f2a6315e-bd0c-4e81-8889-bd959a700d72\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-c6e992-55b58d7f9f-zzkqk" Nov 01 16:49:22.431313 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.431322 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4ed1cc4-b328-4aea-a329-e9704aee1afc-metrics-certs\") pod \"maas-api-e2e-authcfg-http-c6e992-796b4d6f6-qz4g6\" (UID: \"f4ed1cc4-b328-4aea-a329-e9704aee1afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c6e992-796b4d6f6-qz4g6" Nov 01 16:49:22.431550 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.431344 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4ed1cc4-b328-4aea-a329-e9704aee1afc-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-c6e992-796b4d6f6-qz4g6\" (UID: \"f4ed1cc4-b328-4aea-a329-e9704aee1afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c6e992-796b4d6f6-qz4g6" Nov 01 16:49:22.431550 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.431406 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4ed1cc4-b328-4aea-a329-e9704aee1afc-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-c6e992-796b4d6f6-qz4g6\" (UID: \"f4ed1cc4-b328-4aea-a329-e9704aee1afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c6e992-796b4d6f6-qz4g6" Nov 01 16:49:22.431550 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.431525 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mv4t\" (UniqueName: \"kubernetes.io/projected/f4ed1cc4-b328-4aea-a329-e9704aee1afc-kube-api-access-5mv4t\") pod \"maas-api-e2e-authcfg-http-c6e992-796b4d6f6-qz4g6\" (UID: \"f4ed1cc4-b328-4aea-a329-e9704aee1afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c6e992-796b4d6f6-qz4g6" Nov 01 16:49:22.431715 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.431592 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8524c9f3-7421-4d7c-9675-3fffd24ce754-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-c6e992-6d6f87545f-k7nxk\" (UID: \"8524c9f3-7421-4d7c-9675-3fffd24ce754\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c6e992-6d6f87545f-k7nxk" Nov 01 16:49:22.431757 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.431713 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvkt5\" (UniqueName: \"kubernetes.io/projected/8524c9f3-7421-4d7c-9675-3fffd24ce754-kube-api-access-jvkt5\") pod \"payload-pre-processing-e2e-authcfg-http-c6e992-6d6f87545f-k7nxk\" (UID: \"8524c9f3-7421-4d7c-9675-3fffd24ce754\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c6e992-6d6f87545f-k7nxk" Nov 01 16:49:22.431757 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.431743 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2a6315e-bd0c-4e81-8889-bd959a700d72-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-c6e992-55b58d7f9f-zzkqk\" (UID: \"f2a6315e-bd0c-4e81-8889-bd959a700d72\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-c6e992-55b58d7f9f-zzkqk" Nov 01 16:49:22.533096 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.532973 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8524c9f3-7421-4d7c-9675-3fffd24ce754-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-c6e992-6d6f87545f-k7nxk\" (UID: \"8524c9f3-7421-4d7c-9675-3fffd24ce754\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c6e992-6d6f87545f-k7nxk" Nov 01 16:49:22.533096 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.533074 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvkt5\" (UniqueName: \"kubernetes.io/projected/8524c9f3-7421-4d7c-9675-3fffd24ce754-kube-api-access-jvkt5\") pod \"payload-pre-processing-e2e-authcfg-http-c6e992-6d6f87545f-k7nxk\" (UID: \"8524c9f3-7421-4d7c-9675-3fffd24ce754\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c6e992-6d6f87545f-k7nxk" Nov 01 16:49:22.533357 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.533113 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2a6315e-bd0c-4e81-8889-bd959a700d72-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-c6e992-55b58d7f9f-zzkqk\" (UID: \"f2a6315e-bd0c-4e81-8889-bd959a700d72\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-c6e992-55b58d7f9f-zzkqk" Nov 01 16:49:22.533357 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.533195 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5h5s\" (UniqueName: \"kubernetes.io/projected/f2a6315e-bd0c-4e81-8889-bd959a700d72-kube-api-access-j5h5s\") pod \"payload-processing-e2e-authcfg-http-c6e992-55b58d7f9f-zzkqk\" (UID: \"f2a6315e-bd0c-4e81-8889-bd959a700d72\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-c6e992-55b58d7f9f-zzkqk" Nov 01 16:49:22.533357 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.533227 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4ed1cc4-b328-4aea-a329-e9704aee1afc-metrics-certs\") pod \"maas-api-e2e-authcfg-http-c6e992-796b4d6f6-qz4g6\" (UID: \"f4ed1cc4-b328-4aea-a329-e9704aee1afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c6e992-796b4d6f6-qz4g6" Nov 01 16:49:22.533357 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.533258 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4ed1cc4-b328-4aea-a329-e9704aee1afc-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-c6e992-796b4d6f6-qz4g6\" (UID: \"f4ed1cc4-b328-4aea-a329-e9704aee1afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c6e992-796b4d6f6-qz4g6" Nov 01 16:49:22.533357 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.533285 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4ed1cc4-b328-4aea-a329-e9704aee1afc-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-c6e992-796b4d6f6-qz4g6\" (UID: \"f4ed1cc4-b328-4aea-a329-e9704aee1afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c6e992-796b4d6f6-qz4g6" Nov 01 16:49:22.533357 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.533355 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mv4t\" (UniqueName: \"kubernetes.io/projected/f4ed1cc4-b328-4aea-a329-e9704aee1afc-kube-api-access-5mv4t\") pod \"maas-api-e2e-authcfg-http-c6e992-796b4d6f6-qz4g6\" (UID: \"f4ed1cc4-b328-4aea-a329-e9704aee1afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c6e992-796b4d6f6-qz4g6" Nov 01 16:49:22.534513 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.534487 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8524c9f3-7421-4d7c-9675-3fffd24ce754-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-c6e992-6d6f87545f-k7nxk\" (UID: \"8524c9f3-7421-4d7c-9675-3fffd24ce754\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c6e992-6d6f87545f-k7nxk" Nov 01 16:49:22.535384 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.535355 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2a6315e-bd0c-4e81-8889-bd959a700d72-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-c6e992-55b58d7f9f-zzkqk\" (UID: \"f2a6315e-bd0c-4e81-8889-bd959a700d72\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-c6e992-55b58d7f9f-zzkqk" Nov 01 16:49:22.535713 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.535689 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4ed1cc4-b328-4aea-a329-e9704aee1afc-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-c6e992-796b4d6f6-qz4g6\" (UID: \"f4ed1cc4-b328-4aea-a329-e9704aee1afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c6e992-796b4d6f6-qz4g6" Nov 01 16:49:22.537741 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.537720 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4ed1cc4-b328-4aea-a329-e9704aee1afc-metrics-certs\") pod \"maas-api-e2e-authcfg-http-c6e992-796b4d6f6-qz4g6\" (UID: \"f4ed1cc4-b328-4aea-a329-e9704aee1afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c6e992-796b4d6f6-qz4g6" Nov 01 16:49:22.537836 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.537767 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4ed1cc4-b328-4aea-a329-e9704aee1afc-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-c6e992-796b4d6f6-qz4g6\" (UID: \"f4ed1cc4-b328-4aea-a329-e9704aee1afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c6e992-796b4d6f6-qz4g6" Nov 01 16:49:22.541441 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.541409 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mv4t\" (UniqueName: \"kubernetes.io/projected/f4ed1cc4-b328-4aea-a329-e9704aee1afc-kube-api-access-5mv4t\") pod \"maas-api-e2e-authcfg-http-c6e992-796b4d6f6-qz4g6\" (UID: \"f4ed1cc4-b328-4aea-a329-e9704aee1afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c6e992-796b4d6f6-qz4g6" Nov 01 16:49:22.542465 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.542441 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvkt5\" (UniqueName: \"kubernetes.io/projected/8524c9f3-7421-4d7c-9675-3fffd24ce754-kube-api-access-jvkt5\") pod \"payload-pre-processing-e2e-authcfg-http-c6e992-6d6f87545f-k7nxk\" (UID: \"8524c9f3-7421-4d7c-9675-3fffd24ce754\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c6e992-6d6f87545f-k7nxk" Nov 01 16:49:22.543080 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.543057 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5h5s\" (UniqueName: \"kubernetes.io/projected/f2a6315e-bd0c-4e81-8889-bd959a700d72-kube-api-access-j5h5s\") pod \"payload-processing-e2e-authcfg-http-c6e992-55b58d7f9f-zzkqk\" (UID: \"f2a6315e-bd0c-4e81-8889-bd959a700d72\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-c6e992-55b58d7f9f-zzkqk" Nov 01 16:49:22.588492 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.588466 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c6e992-796b4d6f6-qz4g6" Nov 01 16:49:22.623879 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.623851 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-c6e992-55b58d7f9f-zzkqk" Nov 01 16:49:22.643045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.642979 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c6e992-6d6f87545f-k7nxk" Nov 01 16:49:22.812237 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.812154 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1680feb3-7324-4c37-98f7-bc5104acd989" path="/var/lib/kubelet/pods/1680feb3-7324-4c37-98f7-bc5104acd989/volumes" Nov 01 16:49:22.873783 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.873748 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c6e992-796b4d6f6-qz4g6"] Nov 01 16:49:22.877748 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:49:22.877719 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4ed1cc4_b328_4aea_a329_e9704aee1afc.slice/crio-94109389cfba35ba5093633864e3cdb07b8f7fc6e262f5b28d20073a2e8f669f WatchSource:0}: Error finding container 94109389cfba35ba5093633864e3cdb07b8f7fc6e262f5b28d20073a2e8f669f: Status 404 returned error can't find the container with id 94109389cfba35ba5093633864e3cdb07b8f7fc6e262f5b28d20073a2e8f669f Nov 01 16:49:22.933978 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.933945 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-c6e992-55b58d7f9f-zzkqk"] Nov 01 16:49:22.934348 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:49:22.934315 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2a6315e_bd0c_4e81_8889_bd959a700d72.slice/crio-a4cd2f355beb8fb672c353212cb2a768dfcbee16dd214d06b87b7709d82faf47 WatchSource:0}: Error finding container a4cd2f355beb8fb672c353212cb2a768dfcbee16dd214d06b87b7709d82faf47: Status 404 returned error can't find the container with id a4cd2f355beb8fb672c353212cb2a768dfcbee16dd214d06b87b7709d82faf47 Nov 01 16:49:22.955765 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:22.955738 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-c6e992-6d6f87545f-k7nxk"] Nov 01 16:49:22.962217 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:49:22.962186 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8524c9f3_7421_4d7c_9675_3fffd24ce754.slice/crio-df675437260d7bc28a62f9bec0da7a627b8f3f5d049ff1ba874a63d0eb6b37de WatchSource:0}: Error finding container df675437260d7bc28a62f9bec0da7a627b8f3f5d049ff1ba874a63d0eb6b37de: Status 404 returned error can't find the container with id df675437260d7bc28a62f9bec0da7a627b8f3f5d049ff1ba874a63d0eb6b37de Nov 01 16:49:23.850361 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:23.850076 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-c6e992-55b58d7f9f-zzkqk" event={"ID":"f2a6315e-bd0c-4e81-8889-bd959a700d72","Type":"ContainerStarted","Data":"11381a071bdbf42bd36a86d826dc069d4d048c3dc5707485511a381360fcab68"} Nov 01 16:49:23.850361 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:23.850149 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-c6e992-55b58d7f9f-zzkqk" event={"ID":"f2a6315e-bd0c-4e81-8889-bd959a700d72","Type":"ContainerStarted","Data":"a4cd2f355beb8fb672c353212cb2a768dfcbee16dd214d06b87b7709d82faf47"} Nov 01 16:49:23.851100 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:23.851069 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-c6e992-55b58d7f9f-zzkqk" Nov 01 16:49:23.853059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:23.852771 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c6e992-6d6f87545f-k7nxk" event={"ID":"8524c9f3-7421-4d7c-9675-3fffd24ce754","Type":"ContainerStarted","Data":"01a5edf1657884f934fe6e756dad8138ec580fab2ce0d68f7714c3b44929537f"} Nov 01 16:49:23.853059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:23.852825 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c6e992-6d6f87545f-k7nxk" event={"ID":"8524c9f3-7421-4d7c-9675-3fffd24ce754","Type":"ContainerStarted","Data":"df675437260d7bc28a62f9bec0da7a627b8f3f5d049ff1ba874a63d0eb6b37de"} Nov 01 16:49:23.853059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:23.852848 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c6e992-6d6f87545f-k7nxk" Nov 01 16:49:23.854562 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:23.854535 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c6e992-796b4d6f6-qz4g6" event={"ID":"f4ed1cc4-b328-4aea-a329-e9704aee1afc","Type":"ContainerStarted","Data":"12b41849dae611c023a8e2cef5649cb9899d6f77d31a9c58a9a6361148d501aa"} Nov 01 16:49:23.854667 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:23.854599 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c6e992-796b4d6f6-qz4g6" Nov 01 16:49:23.854667 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:23.854624 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c6e992-796b4d6f6-qz4g6" event={"ID":"f4ed1cc4-b328-4aea-a329-e9704aee1afc","Type":"ContainerStarted","Data":"94109389cfba35ba5093633864e3cdb07b8f7fc6e262f5b28d20073a2e8f669f"} Nov 01 16:49:23.868560 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:23.868516 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-c6e992-55b58d7f9f-zzkqk" podStartSLOduration=1.868504874 podStartE2EDuration="1.868504874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:49:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:49:23.867172192 +0000 UTC m=+3697.198909842" watchObservedRunningTime="2026-11-01 16:49:23.868504874 +0000 UTC m=+3697.200242524" Nov 01 16:49:23.887073 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:23.886988 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c6e992-796b4d6f6-qz4g6" podStartSLOduration=1.446322572 podStartE2EDuration="1.886970326s" totalImagesPullingTime="440.647754ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:49:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:49:23.883367141 +0000 UTC m=+3697.215104796" watchObservedRunningTime="2026-11-01 16:49:23.886970326 +0000 UTC m=+3697.218707977" Nov 01 16:49:23.902958 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:23.902902 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c6e992-6d6f87545f-k7nxk" podStartSLOduration=1.902885497 podStartE2EDuration="1.902885497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:49:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:49:23.899515374 +0000 UTC m=+3697.231253025" watchObservedRunningTime="2026-11-01 16:49:23.902885497 +0000 UTC m=+3697.234623153" Nov 01 16:49:29.865174 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:29.865140 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c6e992-796b4d6f6-qz4g6" Nov 01 16:49:34.796499 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:34.796468 4363 patch_prober.go:29] interesting pod/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" start-of-body= Nov 01 16:49:34.797045 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:34.796550 4363 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" podUID="163d3f72-1667-453f-b670-db2931af40a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" Nov 01 16:49:34.863600 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:34.863571 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c6e992-6d6f87545f-k7nxk" Nov 01 16:49:34.980140 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:34.980110 4363 generic.go:356] "Generic (PLEG): container finished" podID="163d3f72-1667-453f-b670-db2931af40a6" containerID="f9ee4eb0037d67ee364bf9b2812ca028bf0060b5ed4339122b44a278d7e86f67" exitCode=0 Nov 01 16:49:34.980310 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:34.980160 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" event={"ID":"163d3f72-1667-453f-b670-db2931af40a6","Type":"ContainerDied","Data":"f9ee4eb0037d67ee364bf9b2812ca028bf0060b5ed4339122b44a278d7e86f67"} Nov 01 16:49:34.980536 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:34.980522 4363 scope.go:122] "RemoveContainer" containerID="f9ee4eb0037d67ee364bf9b2812ca028bf0060b5ed4339122b44a278d7e86f67" Nov 01 16:49:34.981835 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:34.981792 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:49:34.982047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:34.981917 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:49:34.982185 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:34.982064 4363 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:49:35.867863 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:35.867830 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-c6e992-55b58d7f9f-zzkqk" Nov 01 16:49:36.007982 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:36.007942 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" event={"ID":"163d3f72-1667-453f-b670-db2931af40a6","Type":"ContainerStarted","Data":"198559252af38eb2a6eaaab3838de747f2aae11ce8b851823ca78df354a3f4ab"} Nov 01 16:49:36.107041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:36.106662 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" Nov 01 16:49:36.107240 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:36.107173 4363 patch_prober.go:29] interesting pod/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" start-of-body= Nov 01 16:49:36.107304 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:36.107236 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" podUID="163d3f72-1667-453f-b670-db2931af40a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" Nov 01 16:49:37.109039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:37.108349 4363 patch_prober.go:29] interesting pod/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" start-of-body= Nov 01 16:49:37.109039 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:37.108418 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" podUID="163d3f72-1667-453f-b670-db2931af40a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" Nov 01 16:49:38.102458 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:38.102417 4363 patch_prober.go:29] interesting pod/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" start-of-body= Nov 01 16:49:38.102761 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:38.102732 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" podUID="163d3f72-1667-453f-b670-db2931af40a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" Nov 01 16:49:39.104041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:39.103431 4363 patch_prober.go:29] interesting pod/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" start-of-body= Nov 01 16:49:39.104041 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:39.103494 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" podUID="163d3f72-1667-453f-b670-db2931af40a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" Nov 01 16:49:40.102539 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:40.102497 4363 patch_prober.go:29] interesting pod/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" start-of-body= Nov 01 16:49:40.102760 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:40.102568 4363 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" podUID="163d3f72-1667-453f-b670-db2931af40a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" Nov 01 16:49:41.171654 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:41.171604 4363 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" Nov 01 16:49:41.172529 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:41.172502 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" Nov 01 16:49:41.173450 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:49:41.173427 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" Nov 01 16:49:48.308603 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:49:48.308565 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:50:18.308319 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:50:18.308239 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:50:48.308702 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:50:48.308660 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:50:55.766452 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:50:55.766417 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c6e992-2mp4p"] Nov 01 16:50:55.769894 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:50:55.769875 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c6e992-2mp4p" Nov 01 16:50:55.775647 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:50:55.775622 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:50:55.776381 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:50:55.776359 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vj67v\"" Nov 01 16:50:55.794613 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:50:55.794587 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c6e992-2mp4p"] Nov 01 16:50:55.798505 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:50:55.798479 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a1c7334-39f1-45c2-9722-a8039b9bdfe5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-c6e992-2mp4p\" (UID: \"1a1c7334-39f1-45c2-9722-a8039b9bdfe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c6e992-2mp4p" Nov 01 16:50:55.798645 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:50:55.798526 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvsvw\" (UniqueName: \"kubernetes.io/projected/1a1c7334-39f1-45c2-9722-a8039b9bdfe5-kube-api-access-gvsvw\") pod \"maas-api-revoke-keys-e2e-authcfg-http-c6e992-2mp4p\" (UID: \"1a1c7334-39f1-45c2-9722-a8039b9bdfe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c6e992-2mp4p" Nov 01 16:50:55.798645 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:50:55.798625 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a1c7334-39f1-45c2-9722-a8039b9bdfe5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-c6e992-2mp4p\" (UID: \"1a1c7334-39f1-45c2-9722-a8039b9bdfe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c6e992-2mp4p" Nov 01 16:50:55.899333 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:50:55.899298 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a1c7334-39f1-45c2-9722-a8039b9bdfe5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-c6e992-2mp4p\" (UID: \"1a1c7334-39f1-45c2-9722-a8039b9bdfe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c6e992-2mp4p" Nov 01 16:50:55.899520 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:50:55.899429 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvsvw\" (UniqueName: \"kubernetes.io/projected/1a1c7334-39f1-45c2-9722-a8039b9bdfe5-kube-api-access-gvsvw\") pod \"maas-api-revoke-keys-e2e-authcfg-http-c6e992-2mp4p\" (UID: \"1a1c7334-39f1-45c2-9722-a8039b9bdfe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c6e992-2mp4p" Nov 01 16:50:55.899520 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:50:55.899485 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a1c7334-39f1-45c2-9722-a8039b9bdfe5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-c6e992-2mp4p\" (UID: \"1a1c7334-39f1-45c2-9722-a8039b9bdfe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c6e992-2mp4p" Nov 01 16:50:55.900156 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:50:55.900127 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a1c7334-39f1-45c2-9722-a8039b9bdfe5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-c6e992-2mp4p\" (UID: \"1a1c7334-39f1-45c2-9722-a8039b9bdfe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c6e992-2mp4p" Nov 01 16:50:55.902063 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:50:55.902043 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a1c7334-39f1-45c2-9722-a8039b9bdfe5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-c6e992-2mp4p\" (UID: \"1a1c7334-39f1-45c2-9722-a8039b9bdfe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c6e992-2mp4p" Nov 01 16:50:55.909927 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:50:55.909891 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvsvw\" (UniqueName: \"kubernetes.io/projected/1a1c7334-39f1-45c2-9722-a8039b9bdfe5-kube-api-access-gvsvw\") pod \"maas-api-revoke-keys-e2e-authcfg-http-c6e992-2mp4p\" (UID: \"1a1c7334-39f1-45c2-9722-a8039b9bdfe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c6e992-2mp4p" Nov 01 16:50:56.083392 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:50:56.083311 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c6e992-2mp4p" Nov 01 16:50:56.364482 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:50:56.364434 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c6e992-2mp4p"] Nov 01 16:50:56.366864 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:50:56.366838 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a1c7334_39f1_45c2_9722_a8039b9bdfe5.slice/crio-3902b6b42ecb6b7ae875b382c475d2112c8dbc6229fe00212d62b0306dd7db36 WatchSource:0}: Error finding container 3902b6b42ecb6b7ae875b382c475d2112c8dbc6229fe00212d62b0306dd7db36: Status 404 returned error can't find the container with id 3902b6b42ecb6b7ae875b382c475d2112c8dbc6229fe00212d62b0306dd7db36 Nov 01 16:50:56.428798 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:50:56.428771 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c6e992-2mp4p" event={"ID":"1a1c7334-39f1-45c2-9722-a8039b9bdfe5","Type":"ContainerStarted","Data":"3902b6b42ecb6b7ae875b382c475d2112c8dbc6229fe00212d62b0306dd7db36"} Nov 01 16:50:57.436386 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:50:57.436295 4363 generic.go:356] "Generic (PLEG): container finished" podID="1a1c7334-39f1-45c2-9722-a8039b9bdfe5" containerID="09e90de6866c31e1fd070cf9adcaa9a2f881d3838645d4c7587dca5c3c76be4e" exitCode=0 Nov 01 16:50:57.436386 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:50:57.436367 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c6e992-2mp4p" event={"ID":"1a1c7334-39f1-45c2-9722-a8039b9bdfe5","Type":"ContainerDied","Data":"09e90de6866c31e1fd070cf9adcaa9a2f881d3838645d4c7587dca5c3c76be4e"} Nov 01 16:50:58.575853 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:50:58.575822 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c6e992-2mp4p" Nov 01 16:50:58.626245 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:50:58.626217 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a1c7334-39f1-45c2-9722-a8039b9bdfe5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a1c7334-39f1-45c2-9722-a8039b9bdfe5-maas-api-openshift-ca\") pod \"1a1c7334-39f1-45c2-9722-a8039b9bdfe5\" (UID: \"1a1c7334-39f1-45c2-9722-a8039b9bdfe5\") " Nov 01 16:50:58.626420 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:50:58.626305 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a1c7334-39f1-45c2-9722-a8039b9bdfe5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a1c7334-39f1-45c2-9722-a8039b9bdfe5-maas-api-xks-ca\") pod \"1a1c7334-39f1-45c2-9722-a8039b9bdfe5\" (UID: \"1a1c7334-39f1-45c2-9722-a8039b9bdfe5\") " Nov 01 16:50:58.626420 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:50:58.626393 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a1c7334-39f1-45c2-9722-a8039b9bdfe5-kube-api-access-gvsvw\" (UniqueName: \"kubernetes.io/projected/1a1c7334-39f1-45c2-9722-a8039b9bdfe5-kube-api-access-gvsvw\") pod \"1a1c7334-39f1-45c2-9722-a8039b9bdfe5\" (UID: \"1a1c7334-39f1-45c2-9722-a8039b9bdfe5\") " Nov 01 16:50:58.628575 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:50:58.626662 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a1c7334-39f1-45c2-9722-a8039b9bdfe5-maas-api-openshift-ca" pod "1a1c7334-39f1-45c2-9722-a8039b9bdfe5" (UID: "1a1c7334-39f1-45c2-9722-a8039b9bdfe5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:50:58.628575 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:50:58.626780 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a1c7334-39f1-45c2-9722-a8039b9bdfe5-maas-api-openshift-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:50:58.628733 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:50:58.628696 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a1c7334-39f1-45c2-9722-a8039b9bdfe5-maas-api-xks-ca" pod "1a1c7334-39f1-45c2-9722-a8039b9bdfe5" (UID: "1a1c7334-39f1-45c2-9722-a8039b9bdfe5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:50:58.629090 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:50:58.629067 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a1c7334-39f1-45c2-9722-a8039b9bdfe5-kube-api-access-gvsvw" pod "1a1c7334-39f1-45c2-9722-a8039b9bdfe5" (UID: "1a1c7334-39f1-45c2-9722-a8039b9bdfe5"). InnerVolumeSpecName "kube-api-access-gvsvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:50:58.728340 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:50:58.728260 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvsvw\" (UniqueName: \"kubernetes.io/projected/1a1c7334-39f1-45c2-9722-a8039b9bdfe5-kube-api-access-gvsvw\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:50:58.728340 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:50:58.728291 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a1c7334-39f1-45c2-9722-a8039b9bdfe5-maas-api-xks-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:50:59.446836 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:50:59.446809 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c6e992-2mp4p" Nov 01 16:50:59.447038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:50:59.446805 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c6e992-2mp4p" event={"ID":"1a1c7334-39f1-45c2-9722-a8039b9bdfe5","Type":"ContainerDied","Data":"3902b6b42ecb6b7ae875b382c475d2112c8dbc6229fe00212d62b0306dd7db36"} Nov 01 16:50:59.447038 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:50:59.446934 4363 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3902b6b42ecb6b7ae875b382c475d2112c8dbc6229fe00212d62b0306dd7db36" Nov 01 16:51:05.922243 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:05.922210 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-c6e992-55b58d7f9f-zzkqk"] Nov 01 16:51:05.934315 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:05.934281 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-c6e992-6d6f87545f-k7nxk"] Nov 01 16:51:10.923919 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:10.923859 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-c6e992-55b58d7f9f-zzkqk" podUID="f2a6315e-bd0c-4e81-8889-bd959a700d72" containerName="payload-processing" containerID="cri-o://11381a071bdbf42bd36a86d826dc069d4d048c3dc5707485511a381360fcab68" gracePeriod=25 Nov 01 16:51:10.935005 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:10.934969 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c6e992-6d6f87545f-k7nxk" podUID="8524c9f3-7421-4d7c-9675-3fffd24ce754" containerName="payload-pre-processing" containerID="cri-o://01a5edf1657884f934fe6e756dad8138ec580fab2ce0d68f7714c3b44929537f" gracePeriod=25 Nov 01 16:51:11.186354 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.186276 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-c6e992-55b58d7f9f-zzkqk" Nov 01 16:51:11.217988 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.217954 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c6e992-6d6f87545f-k7nxk" Nov 01 16:51:11.261290 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.261260 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2a6315e-bd0c-4e81-8889-bd959a700d72-kube-api-access-j5h5s\" (UniqueName: \"kubernetes.io/projected/f2a6315e-bd0c-4e81-8889-bd959a700d72-kube-api-access-j5h5s\") pod \"f2a6315e-bd0c-4e81-8889-bd959a700d72\" (UID: \"f2a6315e-bd0c-4e81-8889-bd959a700d72\") " Nov 01 16:51:11.261459 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.261319 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2a6315e-bd0c-4e81-8889-bd959a700d72-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2a6315e-bd0c-4e81-8889-bd959a700d72-plugins-config-volume\") pod \"f2a6315e-bd0c-4e81-8889-bd959a700d72\" (UID: \"f2a6315e-bd0c-4e81-8889-bd959a700d72\") " Nov 01 16:51:11.261723 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.261701 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2a6315e-bd0c-4e81-8889-bd959a700d72-plugins-config-volume" pod "f2a6315e-bd0c-4e81-8889-bd959a700d72" (UID: "f2a6315e-bd0c-4e81-8889-bd959a700d72"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:51:11.263460 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.263434 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2a6315e-bd0c-4e81-8889-bd959a700d72-kube-api-access-j5h5s" pod "f2a6315e-bd0c-4e81-8889-bd959a700d72" (UID: "f2a6315e-bd0c-4e81-8889-bd959a700d72"). InnerVolumeSpecName "kube-api-access-j5h5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:51:11.362414 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.362383 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8524c9f3-7421-4d7c-9675-3fffd24ce754-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8524c9f3-7421-4d7c-9675-3fffd24ce754-plugins-config-volume\") pod \"8524c9f3-7421-4d7c-9675-3fffd24ce754\" (UID: \"8524c9f3-7421-4d7c-9675-3fffd24ce754\") " Nov 01 16:51:11.362591 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.362459 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8524c9f3-7421-4d7c-9675-3fffd24ce754-kube-api-access-jvkt5\" (UniqueName: \"kubernetes.io/projected/8524c9f3-7421-4d7c-9675-3fffd24ce754-kube-api-access-jvkt5\") pod \"8524c9f3-7421-4d7c-9675-3fffd24ce754\" (UID: \"8524c9f3-7421-4d7c-9675-3fffd24ce754\") " Nov 01 16:51:11.362654 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.362640 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5h5s\" (UniqueName: \"kubernetes.io/projected/f2a6315e-bd0c-4e81-8889-bd959a700d72-kube-api-access-j5h5s\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:51:11.362697 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.362656 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2a6315e-bd0c-4e81-8889-bd959a700d72-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:51:11.362858 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.362826 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8524c9f3-7421-4d7c-9675-3fffd24ce754-plugins-config-volume" pod "8524c9f3-7421-4d7c-9675-3fffd24ce754" (UID: "8524c9f3-7421-4d7c-9675-3fffd24ce754"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:51:11.364597 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.364576 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8524c9f3-7421-4d7c-9675-3fffd24ce754-kube-api-access-jvkt5" pod "8524c9f3-7421-4d7c-9675-3fffd24ce754" (UID: "8524c9f3-7421-4d7c-9675-3fffd24ce754"). InnerVolumeSpecName "kube-api-access-jvkt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:51:11.422279 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.422237 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c6e992-796b4d6f6-qz4g6"] Nov 01 16:51:11.422502 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.422471 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c6e992-796b4d6f6-qz4g6" podUID="f4ed1cc4-b328-4aea-a329-e9704aee1afc" containerName="maas-api" containerID="cri-o://12b41849dae611c023a8e2cef5649cb9899d6f77d31a9c58a9a6361148d501aa" gracePeriod=30 Nov 01 16:51:11.463476 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.463408 4363 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8524c9f3-7421-4d7c-9675-3fffd24ce754-plugins-config-volume\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:51:11.463476 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.463436 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvkt5\" (UniqueName: \"kubernetes.io/projected/8524c9f3-7421-4d7c-9675-3fffd24ce754-kube-api-access-jvkt5\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:51:11.507869 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.507837 4363 generic.go:356] "Generic (PLEG): container finished" podID="f2a6315e-bd0c-4e81-8889-bd959a700d72" containerID="11381a071bdbf42bd36a86d826dc069d4d048c3dc5707485511a381360fcab68" exitCode=0 Nov 01 16:51:11.508090 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.507894 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-c6e992-55b58d7f9f-zzkqk" Nov 01 16:51:11.508090 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.507922 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-c6e992-55b58d7f9f-zzkqk" event={"ID":"f2a6315e-bd0c-4e81-8889-bd959a700d72","Type":"ContainerDied","Data":"11381a071bdbf42bd36a86d826dc069d4d048c3dc5707485511a381360fcab68"} Nov 01 16:51:11.508090 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.507977 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-c6e992-55b58d7f9f-zzkqk" event={"ID":"f2a6315e-bd0c-4e81-8889-bd959a700d72","Type":"ContainerDied","Data":"a4cd2f355beb8fb672c353212cb2a768dfcbee16dd214d06b87b7709d82faf47"} Nov 01 16:51:11.508090 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.508007 4363 scope.go:122] "RemoveContainer" containerID="11381a071bdbf42bd36a86d826dc069d4d048c3dc5707485511a381360fcab68" Nov 01 16:51:11.509528 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.509501 4363 generic.go:356] "Generic (PLEG): container finished" podID="8524c9f3-7421-4d7c-9675-3fffd24ce754" containerID="01a5edf1657884f934fe6e756dad8138ec580fab2ce0d68f7714c3b44929537f" exitCode=0 Nov 01 16:51:11.509653 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.509544 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c6e992-6d6f87545f-k7nxk" Nov 01 16:51:11.509653 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.509585 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c6e992-6d6f87545f-k7nxk" event={"ID":"8524c9f3-7421-4d7c-9675-3fffd24ce754","Type":"ContainerDied","Data":"01a5edf1657884f934fe6e756dad8138ec580fab2ce0d68f7714c3b44929537f"} Nov 01 16:51:11.509653 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.509628 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c6e992-6d6f87545f-k7nxk" event={"ID":"8524c9f3-7421-4d7c-9675-3fffd24ce754","Type":"ContainerDied","Data":"df675437260d7bc28a62f9bec0da7a627b8f3f5d049ff1ba874a63d0eb6b37de"} Nov 01 16:51:11.536546 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.536414 4363 scope.go:122] "RemoveContainer" containerID="11381a071bdbf42bd36a86d826dc069d4d048c3dc5707485511a381360fcab68" Nov 01 16:51:11.536841 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:51:11.536819 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11381a071bdbf42bd36a86d826dc069d4d048c3dc5707485511a381360fcab68\": container with ID starting with 11381a071bdbf42bd36a86d826dc069d4d048c3dc5707485511a381360fcab68 not found: ID does not exist" containerID="11381a071bdbf42bd36a86d826dc069d4d048c3dc5707485511a381360fcab68" Nov 01 16:51:11.536954 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.536853 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11381a071bdbf42bd36a86d826dc069d4d048c3dc5707485511a381360fcab68"} err="failed to get container status \"11381a071bdbf42bd36a86d826dc069d4d048c3dc5707485511a381360fcab68\": rpc error: code = NotFound desc = could not find container \"11381a071bdbf42bd36a86d826dc069d4d048c3dc5707485511a381360fcab68\": container with ID starting with 11381a071bdbf42bd36a86d826dc069d4d048c3dc5707485511a381360fcab68 not found: ID does not exist" Nov 01 16:51:11.536954 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.536876 4363 scope.go:122] "RemoveContainer" containerID="01a5edf1657884f934fe6e756dad8138ec580fab2ce0d68f7714c3b44929537f" Nov 01 16:51:11.539209 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.539167 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-c6e992-6d6f87545f-k7nxk"] Nov 01 16:51:11.542587 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.542563 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-c6e992-6d6f87545f-k7nxk"] Nov 01 16:51:11.547337 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.547320 4363 scope.go:122] "RemoveContainer" containerID="01a5edf1657884f934fe6e756dad8138ec580fab2ce0d68f7714c3b44929537f" Nov 01 16:51:11.547633 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:51:11.547613 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01a5edf1657884f934fe6e756dad8138ec580fab2ce0d68f7714c3b44929537f\": container with ID starting with 01a5edf1657884f934fe6e756dad8138ec580fab2ce0d68f7714c3b44929537f not found: ID does not exist" containerID="01a5edf1657884f934fe6e756dad8138ec580fab2ce0d68f7714c3b44929537f" Nov 01 16:51:11.547750 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.547640 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01a5edf1657884f934fe6e756dad8138ec580fab2ce0d68f7714c3b44929537f"} err="failed to get container status \"01a5edf1657884f934fe6e756dad8138ec580fab2ce0d68f7714c3b44929537f\": rpc error: code = NotFound desc = could not find container \"01a5edf1657884f934fe6e756dad8138ec580fab2ce0d68f7714c3b44929537f\": container with ID starting with 01a5edf1657884f934fe6e756dad8138ec580fab2ce0d68f7714c3b44929537f not found: ID does not exist" Nov 01 16:51:11.572561 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.572527 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-c6e992-55b58d7f9f-zzkqk"] Nov 01 16:51:11.587047 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.586263 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-c6e992-55b58d7f9f-zzkqk"] Nov 01 16:51:11.714141 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.713984 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c6e992-796b4d6f6-qz4g6" Nov 01 16:51:11.764905 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.764871 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c6e992-2mp4p"] Nov 01 16:51:11.767398 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.767367 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4ed1cc4-b328-4aea-a329-e9704aee1afc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4ed1cc4-b328-4aea-a329-e9704aee1afc-metrics-certs\") pod \"f4ed1cc4-b328-4aea-a329-e9704aee1afc\" (UID: \"f4ed1cc4-b328-4aea-a329-e9704aee1afc\") " Nov 01 16:51:11.767556 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.767489 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4ed1cc4-b328-4aea-a329-e9704aee1afc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4ed1cc4-b328-4aea-a329-e9704aee1afc-maas-api-tls\") pod \"f4ed1cc4-b328-4aea-a329-e9704aee1afc\" (UID: \"f4ed1cc4-b328-4aea-a329-e9704aee1afc\") " Nov 01 16:51:11.767556 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.767525 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4ed1cc4-b328-4aea-a329-e9704aee1afc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4ed1cc4-b328-4aea-a329-e9704aee1afc-openshift-service-ca\") pod \"f4ed1cc4-b328-4aea-a329-e9704aee1afc\" (UID: \"f4ed1cc4-b328-4aea-a329-e9704aee1afc\") " Nov 01 16:51:11.767688 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.767578 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4ed1cc4-b328-4aea-a329-e9704aee1afc-kube-api-access-5mv4t\" (UniqueName: \"kubernetes.io/projected/f4ed1cc4-b328-4aea-a329-e9704aee1afc-kube-api-access-5mv4t\") pod \"f4ed1cc4-b328-4aea-a329-e9704aee1afc\" (UID: \"f4ed1cc4-b328-4aea-a329-e9704aee1afc\") " Nov 01 16:51:11.768236 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.768195 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4ed1cc4-b328-4aea-a329-e9704aee1afc-openshift-service-ca" pod "f4ed1cc4-b328-4aea-a329-e9704aee1afc" (UID: "f4ed1cc4-b328-4aea-a329-e9704aee1afc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:51:11.768457 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.768435 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-c6e992-2mp4p"] Nov 01 16:51:11.770107 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.770082 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4ed1cc4-b328-4aea-a329-e9704aee1afc-metrics-certs" pod "f4ed1cc4-b328-4aea-a329-e9704aee1afc" (UID: "f4ed1cc4-b328-4aea-a329-e9704aee1afc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:51:11.770349 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.770335 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4ed1cc4-b328-4aea-a329-e9704aee1afc-kube-api-access-5mv4t" pod "f4ed1cc4-b328-4aea-a329-e9704aee1afc" (UID: "f4ed1cc4-b328-4aea-a329-e9704aee1afc"). InnerVolumeSpecName "kube-api-access-5mv4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:51:11.770624 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.770611 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4ed1cc4-b328-4aea-a329-e9704aee1afc-maas-api-tls" pod "f4ed1cc4-b328-4aea-a329-e9704aee1afc" (UID: "f4ed1cc4-b328-4aea-a329-e9704aee1afc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:51:11.870948 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.870834 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4ed1cc4-b328-4aea-a329-e9704aee1afc-maas-api-tls\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:51:11.870948 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.870878 4363 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4ed1cc4-b328-4aea-a329-e9704aee1afc-openshift-service-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:51:11.870948 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.870894 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mv4t\" (UniqueName: \"kubernetes.io/projected/f4ed1cc4-b328-4aea-a329-e9704aee1afc-kube-api-access-5mv4t\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:51:11.870948 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:11.870913 4363 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4ed1cc4-b328-4aea-a329-e9704aee1afc-metrics-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:51:12.515918 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:12.515876 4363 generic.go:356] "Generic (PLEG): container finished" podID="f4ed1cc4-b328-4aea-a329-e9704aee1afc" containerID="12b41849dae611c023a8e2cef5649cb9899d6f77d31a9c58a9a6361148d501aa" exitCode=0 Nov 01 16:51:12.516464 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:12.515933 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c6e992-796b4d6f6-qz4g6" Nov 01 16:51:12.516464 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:12.515984 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c6e992-796b4d6f6-qz4g6" event={"ID":"f4ed1cc4-b328-4aea-a329-e9704aee1afc","Type":"ContainerDied","Data":"12b41849dae611c023a8e2cef5649cb9899d6f77d31a9c58a9a6361148d501aa"} Nov 01 16:51:12.516464 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:12.516058 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c6e992-796b4d6f6-qz4g6" event={"ID":"f4ed1cc4-b328-4aea-a329-e9704aee1afc","Type":"ContainerDied","Data":"94109389cfba35ba5093633864e3cdb07b8f7fc6e262f5b28d20073a2e8f669f"} Nov 01 16:51:12.516464 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:12.516093 4363 scope.go:122] "RemoveContainer" containerID="12b41849dae611c023a8e2cef5649cb9899d6f77d31a9c58a9a6361148d501aa" Nov 01 16:51:12.525944 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:12.525928 4363 scope.go:122] "RemoveContainer" containerID="12b41849dae611c023a8e2cef5649cb9899d6f77d31a9c58a9a6361148d501aa" Nov 01 16:51:12.526250 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:51:12.526223 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12b41849dae611c023a8e2cef5649cb9899d6f77d31a9c58a9a6361148d501aa\": container with ID starting with 12b41849dae611c023a8e2cef5649cb9899d6f77d31a9c58a9a6361148d501aa not found: ID does not exist" containerID="12b41849dae611c023a8e2cef5649cb9899d6f77d31a9c58a9a6361148d501aa" Nov 01 16:51:12.526326 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:12.526255 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12b41849dae611c023a8e2cef5649cb9899d6f77d31a9c58a9a6361148d501aa"} err="failed to get container status \"12b41849dae611c023a8e2cef5649cb9899d6f77d31a9c58a9a6361148d501aa\": rpc error: code = NotFound desc = could not find container \"12b41849dae611c023a8e2cef5649cb9899d6f77d31a9c58a9a6361148d501aa\": container with ID starting with 12b41849dae611c023a8e2cef5649cb9899d6f77d31a9c58a9a6361148d501aa not found: ID does not exist" Nov 01 16:51:12.559518 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:12.559488 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c6e992-796b4d6f6-qz4g6"] Nov 01 16:51:12.574706 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:12.574681 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c6e992-796b4d6f6-qz4g6"] Nov 01 16:51:12.797153 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:12.797081 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a1c7334-39f1-45c2-9722-a8039b9bdfe5" path="/var/lib/kubelet/pods/1a1c7334-39f1-45c2-9722-a8039b9bdfe5/volumes" Nov 01 16:51:12.797450 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:12.797436 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8524c9f3-7421-4d7c-9675-3fffd24ce754" path="/var/lib/kubelet/pods/8524c9f3-7421-4d7c-9675-3fffd24ce754/volumes" Nov 01 16:51:12.797774 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:12.797762 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2a6315e-bd0c-4e81-8889-bd959a700d72" path="/var/lib/kubelet/pods/f2a6315e-bd0c-4e81-8889-bd959a700d72/volumes" Nov 01 16:51:12.798518 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:12.798498 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4ed1cc4-b328-4aea-a329-e9704aee1afc" path="/var/lib/kubelet/pods/f4ed1cc4-b328-4aea-a329-e9704aee1afc/volumes" Nov 01 16:51:18.308538 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:51:18.308498 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:51:18.602692 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:18.602662 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz"] Nov 01 16:51:18.602917 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:18.602890 4363 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" podUID="163d3f72-1667-453f-b670-db2931af40a6" containerName="istio-proxy" containerID="cri-o://198559252af38eb2a6eaaab3838de747f2aae11ce8b851823ca78df354a3f4ab" gracePeriod=30 Nov 01 16:51:23.822060 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:51:23.820443 4363 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod163d3f72_1667_453f_b670_db2931af40a6.slice/crio-198559252af38eb2a6eaaab3838de747f2aae11ce8b851823ca78df354a3f4ab.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod163d3f72_1667_453f_b670_db2931af40a6.slice/crio-conmon-198559252af38eb2a6eaaab3838de747f2aae11ce8b851823ca78df354a3f4ab.scope\": RecentStats: unable to find data in memory cache]" Nov 01 16:51:24.178700 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:24.178669 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" Nov 01 16:51:24.259753 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:24.259721 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/163d3f72-1667-453f-b670-db2931af40a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/163d3f72-1667-453f-b670-db2931af40a6-istiod-ca-cert\") pod \"163d3f72-1667-453f-b670-db2931af40a6\" (UID: \"163d3f72-1667-453f-b670-db2931af40a6\") " Nov 01 16:51:24.259946 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:24.259808 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/163d3f72-1667-453f-b670-db2931af40a6-istio-token\" (UniqueName: \"kubernetes.io/projected/163d3f72-1667-453f-b670-db2931af40a6-istio-token\") pod \"163d3f72-1667-453f-b670-db2931af40a6\" (UID: \"163d3f72-1667-453f-b670-db2931af40a6\") " Nov 01 16:51:24.259946 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:24.259835 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/163d3f72-1667-453f-b670-db2931af40a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/163d3f72-1667-453f-b670-db2931af40a6-credential-socket\") pod \"163d3f72-1667-453f-b670-db2931af40a6\" (UID: \"163d3f72-1667-453f-b670-db2931af40a6\") " Nov 01 16:51:24.259946 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:24.259922 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/163d3f72-1667-453f-b670-db2931af40a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/163d3f72-1667-453f-b670-db2931af40a6-istio-podinfo\") pod \"163d3f72-1667-453f-b670-db2931af40a6\" (UID: \"163d3f72-1667-453f-b670-db2931af40a6\") " Nov 01 16:51:24.260872 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:24.259990 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/163d3f72-1667-453f-b670-db2931af40a6-kube-api-access-t97fj\" (UniqueName: \"kubernetes.io/projected/163d3f72-1667-453f-b670-db2931af40a6-kube-api-access-t97fj\") pod \"163d3f72-1667-453f-b670-db2931af40a6\" (UID: \"163d3f72-1667-453f-b670-db2931af40a6\") " Nov 01 16:51:24.260872 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:24.260050 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/163d3f72-1667-453f-b670-db2931af40a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/163d3f72-1667-453f-b670-db2931af40a6-istio-data\") pod \"163d3f72-1667-453f-b670-db2931af40a6\" (UID: \"163d3f72-1667-453f-b670-db2931af40a6\") " Nov 01 16:51:24.260872 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:24.260083 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/163d3f72-1667-453f-b670-db2931af40a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/163d3f72-1667-453f-b670-db2931af40a6-workload-certs\") pod \"163d3f72-1667-453f-b670-db2931af40a6\" (UID: \"163d3f72-1667-453f-b670-db2931af40a6\") " Nov 01 16:51:24.260872 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:24.260137 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/163d3f72-1667-453f-b670-db2931af40a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/163d3f72-1667-453f-b670-db2931af40a6-istio-envoy\") pod \"163d3f72-1667-453f-b670-db2931af40a6\" (UID: \"163d3f72-1667-453f-b670-db2931af40a6\") " Nov 01 16:51:24.260872 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:24.260158 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163d3f72-1667-453f-b670-db2931af40a6-credential-socket" pod "163d3f72-1667-453f-b670-db2931af40a6" (UID: "163d3f72-1667-453f-b670-db2931af40a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:51:24.260872 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:24.260186 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/163d3f72-1667-453f-b670-db2931af40a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/163d3f72-1667-453f-b670-db2931af40a6-workload-socket\") pod \"163d3f72-1667-453f-b670-db2931af40a6\" (UID: \"163d3f72-1667-453f-b670-db2931af40a6\") " Nov 01 16:51:24.260872 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:24.260238 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/163d3f72-1667-453f-b670-db2931af40a6-istiod-ca-cert" pod "163d3f72-1667-453f-b670-db2931af40a6" (UID: "163d3f72-1667-453f-b670-db2931af40a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:51:24.260872 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:24.260373 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163d3f72-1667-453f-b670-db2931af40a6-workload-certs" pod "163d3f72-1667-453f-b670-db2931af40a6" (UID: "163d3f72-1667-453f-b670-db2931af40a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:51:24.260872 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:24.260555 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163d3f72-1667-453f-b670-db2931af40a6-istio-data" pod "163d3f72-1667-453f-b670-db2931af40a6" (UID: "163d3f72-1667-453f-b670-db2931af40a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:51:24.260872 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:24.260566 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163d3f72-1667-453f-b670-db2931af40a6-workload-socket" pod "163d3f72-1667-453f-b670-db2931af40a6" (UID: "163d3f72-1667-453f-b670-db2931af40a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:51:24.260872 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:24.260637 4363 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/163d3f72-1667-453f-b670-db2931af40a6-workload-certs\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:51:24.260872 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:24.260655 4363 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/163d3f72-1667-453f-b670-db2931af40a6-istiod-ca-cert\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:51:24.260872 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:24.260668 4363 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/163d3f72-1667-453f-b670-db2931af40a6-credential-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:51:24.262686 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:24.262657 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/163d3f72-1667-453f-b670-db2931af40a6-istio-podinfo" pod "163d3f72-1667-453f-b670-db2931af40a6" (UID: "163d3f72-1667-453f-b670-db2931af40a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:51:24.262807 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:24.262789 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/163d3f72-1667-453f-b670-db2931af40a6-istio-token" pod "163d3f72-1667-453f-b670-db2931af40a6" (UID: "163d3f72-1667-453f-b670-db2931af40a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:51:24.262875 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:24.262807 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/163d3f72-1667-453f-b670-db2931af40a6-kube-api-access-t97fj" pod "163d3f72-1667-453f-b670-db2931af40a6" (UID: "163d3f72-1667-453f-b670-db2931af40a6"). InnerVolumeSpecName "kube-api-access-t97fj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:51:24.262942 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:24.262922 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163d3f72-1667-453f-b670-db2931af40a6-istio-envoy" pod "163d3f72-1667-453f-b670-db2931af40a6" (UID: "163d3f72-1667-453f-b670-db2931af40a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:51:24.361190 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:24.361157 4363 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/163d3f72-1667-453f-b670-db2931af40a6-istio-token\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:51:24.361190 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:24.361185 4363 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/163d3f72-1667-453f-b670-db2931af40a6-istio-podinfo\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:51:24.361190 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:24.361195 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t97fj\" (UniqueName: \"kubernetes.io/projected/163d3f72-1667-453f-b670-db2931af40a6-kube-api-access-t97fj\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:51:24.361428 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:24.361205 4363 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/163d3f72-1667-453f-b670-db2931af40a6-istio-data\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:51:24.361428 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:24.361213 4363 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/163d3f72-1667-453f-b670-db2931af40a6-istio-envoy\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:51:24.361428 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:24.361221 4363 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/163d3f72-1667-453f-b670-db2931af40a6-workload-socket\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:51:24.634099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:24.633997 4363 generic.go:356] "Generic (PLEG): container finished" podID="163d3f72-1667-453f-b670-db2931af40a6" containerID="198559252af38eb2a6eaaab3838de747f2aae11ce8b851823ca78df354a3f4ab" exitCode=0 Nov 01 16:51:24.634099 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:24.634054 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" Nov 01 16:51:24.634302 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:24.634048 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" event={"ID":"163d3f72-1667-453f-b670-db2931af40a6","Type":"ContainerDied","Data":"198559252af38eb2a6eaaab3838de747f2aae11ce8b851823ca78df354a3f4ab"} Nov 01 16:51:24.634302 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:24.634170 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz" event={"ID":"163d3f72-1667-453f-b670-db2931af40a6","Type":"ContainerDied","Data":"0803cdcc4a3b8b293ff7bce22dba4e26af58e0dbfc408dbbe9dcab11dacdbe71"} Nov 01 16:51:24.634302 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:24.634194 4363 scope.go:122] "RemoveContainer" containerID="198559252af38eb2a6eaaab3838de747f2aae11ce8b851823ca78df354a3f4ab" Nov 01 16:51:24.644059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:24.644042 4363 scope.go:122] "RemoveContainer" containerID="f9ee4eb0037d67ee364bf9b2812ca028bf0060b5ed4339122b44a278d7e86f67" Nov 01 16:51:24.652324 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:24.652303 4363 scope.go:122] "RemoveContainer" containerID="198559252af38eb2a6eaaab3838de747f2aae11ce8b851823ca78df354a3f4ab" Nov 01 16:51:24.652580 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:51:24.652562 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"198559252af38eb2a6eaaab3838de747f2aae11ce8b851823ca78df354a3f4ab\": container with ID starting with 198559252af38eb2a6eaaab3838de747f2aae11ce8b851823ca78df354a3f4ab not found: ID does not exist" containerID="198559252af38eb2a6eaaab3838de747f2aae11ce8b851823ca78df354a3f4ab" Nov 01 16:51:24.652636 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:24.652591 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"198559252af38eb2a6eaaab3838de747f2aae11ce8b851823ca78df354a3f4ab"} err="failed to get container status \"198559252af38eb2a6eaaab3838de747f2aae11ce8b851823ca78df354a3f4ab\": rpc error: code = NotFound desc = could not find container \"198559252af38eb2a6eaaab3838de747f2aae11ce8b851823ca78df354a3f4ab\": container with ID starting with 198559252af38eb2a6eaaab3838de747f2aae11ce8b851823ca78df354a3f4ab not found: ID does not exist" Nov 01 16:51:24.652636 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:24.652616 4363 scope.go:122] "RemoveContainer" containerID="f9ee4eb0037d67ee364bf9b2812ca028bf0060b5ed4339122b44a278d7e86f67" Nov 01 16:51:24.652867 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:51:24.652845 4363 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9ee4eb0037d67ee364bf9b2812ca028bf0060b5ed4339122b44a278d7e86f67\": container with ID starting with f9ee4eb0037d67ee364bf9b2812ca028bf0060b5ed4339122b44a278d7e86f67 not found: ID does not exist" containerID="f9ee4eb0037d67ee364bf9b2812ca028bf0060b5ed4339122b44a278d7e86f67" Nov 01 16:51:24.652910 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:24.652879 4363 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9ee4eb0037d67ee364bf9b2812ca028bf0060b5ed4339122b44a278d7e86f67"} err="failed to get container status \"f9ee4eb0037d67ee364bf9b2812ca028bf0060b5ed4339122b44a278d7e86f67\": rpc error: code = NotFound desc = could not find container \"f9ee4eb0037d67ee364bf9b2812ca028bf0060b5ed4339122b44a278d7e86f67\": container with ID starting with f9ee4eb0037d67ee364bf9b2812ca028bf0060b5ed4339122b44a278d7e86f67 not found: ID does not exist" Nov 01 16:51:24.656832 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:24.656806 4363 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz"] Nov 01 16:51:24.661637 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:24.661616 4363 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-c6e992-openshift-default-6868c659d9-ncmwz"] Nov 01 16:51:24.795455 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:24.795422 4363 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="163d3f72-1667-453f-b670-db2931af40a6" path="/var/lib/kubelet/pods/163d3f72-1667-453f-b670-db2931af40a6/volumes" Nov 01 16:51:30.318841 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:30.318807 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-5xppx"] Nov 01 16:51:30.319352 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:30.319237 4363 cpu_manager.go:414] "containerMap: removing container" podUID="8524c9f3-7421-4d7c-9675-3fffd24ce754" containerName="payload-pre-processing" Nov 01 16:51:30.319352 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:30.319249 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8524c9f3-7421-4d7c-9675-3fffd24ce754" containerName="payload-pre-processing" Nov 01 16:51:30.319352 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:30.319262 4363 cpu_manager.go:414] "containerMap: removing container" podUID="f2a6315e-bd0c-4e81-8889-bd959a700d72" containerName="payload-processing" Nov 01 16:51:30.319352 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:30.319268 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2a6315e-bd0c-4e81-8889-bd959a700d72" containerName="payload-processing" Nov 01 16:51:30.319352 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:30.319279 4363 cpu_manager.go:414] "containerMap: removing container" podUID="163d3f72-1667-453f-b670-db2931af40a6" containerName="istio-proxy" Nov 01 16:51:30.319352 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:30.319284 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="163d3f72-1667-453f-b670-db2931af40a6" containerName="istio-proxy" Nov 01 16:51:30.319352 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:30.319292 4363 cpu_manager.go:414] "containerMap: removing container" podUID="163d3f72-1667-453f-b670-db2931af40a6" containerName="istio-proxy" Nov 01 16:51:30.319352 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:30.319297 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="163d3f72-1667-453f-b670-db2931af40a6" containerName="istio-proxy" Nov 01 16:51:30.319352 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:30.319304 4363 cpu_manager.go:414] "containerMap: removing container" podUID="1a1c7334-39f1-45c2-9722-a8039b9bdfe5" containerName="revoke-keys" Nov 01 16:51:30.319352 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:30.319308 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a1c7334-39f1-45c2-9722-a8039b9bdfe5" containerName="revoke-keys" Nov 01 16:51:30.319352 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:30.319321 4363 cpu_manager.go:414] "containerMap: removing container" podUID="f4ed1cc4-b328-4aea-a329-e9704aee1afc" containerName="maas-api" Nov 01 16:51:30.319352 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:30.319326 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4ed1cc4-b328-4aea-a329-e9704aee1afc" containerName="maas-api" Nov 01 16:51:30.319751 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:30.319380 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4ed1cc4-b328-4aea-a329-e9704aee1afc" containerName="maas-api" Nov 01 16:51:30.319751 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:30.319392 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="8524c9f3-7421-4d7c-9675-3fffd24ce754" containerName="payload-pre-processing" Nov 01 16:51:30.319751 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:30.319400 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2a6315e-bd0c-4e81-8889-bd959a700d72" containerName="payload-processing" Nov 01 16:51:30.319751 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:30.319406 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="163d3f72-1667-453f-b670-db2931af40a6" containerName="istio-proxy" Nov 01 16:51:30.319751 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:30.319411 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="163d3f72-1667-453f-b670-db2931af40a6" containerName="istio-proxy" Nov 01 16:51:30.319751 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:30.319416 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a1c7334-39f1-45c2-9722-a8039b9bdfe5" containerName="revoke-keys" Nov 01 16:51:30.322659 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:30.322640 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-5xppx" Nov 01 16:51:30.329141 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:30.329110 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 16:51:30.329141 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:30.329130 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 16:51:30.330231 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:30.329507 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:51:30.330231 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:30.329875 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vj67v\"" Nov 01 16:51:30.331118 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:30.331094 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-5xppx"] Nov 01 16:51:30.379203 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:30.379172 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05b6b7a4-23d1-402c-88a4-1633e911e55c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-5xppx\" (UID: \"05b6b7a4-23d1-402c-88a4-1633e911e55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-5xppx" Nov 01 16:51:30.379373 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:30.379225 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2272\" (UniqueName: \"kubernetes.io/projected/05b6b7a4-23d1-402c-88a4-1633e911e55c-kube-api-access-b2272\") pod \"maas-api-revoke-sub-f03629db-5xppx\" (UID: \"05b6b7a4-23d1-402c-88a4-1633e911e55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-5xppx" Nov 01 16:51:30.379373 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:30.379248 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05b6b7a4-23d1-402c-88a4-1633e911e55c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-5xppx\" (UID: \"05b6b7a4-23d1-402c-88a4-1633e911e55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-5xppx" Nov 01 16:51:30.480868 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:30.480835 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05b6b7a4-23d1-402c-88a4-1633e911e55c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-5xppx\" (UID: \"05b6b7a4-23d1-402c-88a4-1633e911e55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-5xppx" Nov 01 16:51:30.481083 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:30.480889 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2272\" (UniqueName: \"kubernetes.io/projected/05b6b7a4-23d1-402c-88a4-1633e911e55c-kube-api-access-b2272\") pod \"maas-api-revoke-sub-f03629db-5xppx\" (UID: \"05b6b7a4-23d1-402c-88a4-1633e911e55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-5xppx" Nov 01 16:51:30.481083 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:30.480912 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05b6b7a4-23d1-402c-88a4-1633e911e55c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-5xppx\" (UID: \"05b6b7a4-23d1-402c-88a4-1633e911e55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-5xppx" Nov 01 16:51:30.481596 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:30.481569 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05b6b7a4-23d1-402c-88a4-1633e911e55c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-5xppx\" (UID: \"05b6b7a4-23d1-402c-88a4-1633e911e55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-5xppx" Nov 01 16:51:30.483297 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:30.483268 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05b6b7a4-23d1-402c-88a4-1633e911e55c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-5xppx\" (UID: \"05b6b7a4-23d1-402c-88a4-1633e911e55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-5xppx" Nov 01 16:51:30.490361 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:30.490338 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2272\" (UniqueName: \"kubernetes.io/projected/05b6b7a4-23d1-402c-88a4-1633e911e55c-kube-api-access-b2272\") pod \"maas-api-revoke-sub-f03629db-5xppx\" (UID: \"05b6b7a4-23d1-402c-88a4-1633e911e55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-5xppx" Nov 01 16:51:30.637946 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:30.637914 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-5xppx" Nov 01 16:51:30.915154 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:30.915074 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-5xppx"] Nov 01 16:51:30.918174 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:51:30.918143 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05b6b7a4_23d1_402c_88a4_1633e911e55c.slice/crio-3cf323fecd55767b6984bd899ba87a5dcad13d507b2f86398bd7d5f51b0313b5 WatchSource:0}: Error finding container 3cf323fecd55767b6984bd899ba87a5dcad13d507b2f86398bd7d5f51b0313b5: Status 404 returned error can't find the container with id 3cf323fecd55767b6984bd899ba87a5dcad13d507b2f86398bd7d5f51b0313b5 Nov 01 16:51:31.704758 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:31.704723 4363 generic.go:356] "Generic (PLEG): container finished" podID="05b6b7a4-23d1-402c-88a4-1633e911e55c" containerID="aaea38c92f21d9eea6898e97bdf7d2901117d0df995082e6780a3a4f320069b4" exitCode=0 Nov 01 16:51:31.705154 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:31.704809 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-5xppx" event={"ID":"05b6b7a4-23d1-402c-88a4-1633e911e55c","Type":"ContainerDied","Data":"aaea38c92f21d9eea6898e97bdf7d2901117d0df995082e6780a3a4f320069b4"} Nov 01 16:51:31.705154 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:31.704899 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-5xppx" event={"ID":"05b6b7a4-23d1-402c-88a4-1633e911e55c","Type":"ContainerStarted","Data":"3cf323fecd55767b6984bd899ba87a5dcad13d507b2f86398bd7d5f51b0313b5"} Nov 01 16:51:32.841646 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:32.841620 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-5xppx" Nov 01 16:51:32.904204 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:32.904175 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05b6b7a4-23d1-402c-88a4-1633e911e55c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05b6b7a4-23d1-402c-88a4-1633e911e55c-maas-api-openshift-ca\") pod \"05b6b7a4-23d1-402c-88a4-1633e911e55c\" (UID: \"05b6b7a4-23d1-402c-88a4-1633e911e55c\") " Nov 01 16:51:32.904389 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:32.904253 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05b6b7a4-23d1-402c-88a4-1633e911e55c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05b6b7a4-23d1-402c-88a4-1633e911e55c-maas-api-xks-ca\") pod \"05b6b7a4-23d1-402c-88a4-1633e911e55c\" (UID: \"05b6b7a4-23d1-402c-88a4-1633e911e55c\") " Nov 01 16:51:32.904389 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:32.904307 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05b6b7a4-23d1-402c-88a4-1633e911e55c-kube-api-access-b2272\" (UniqueName: \"kubernetes.io/projected/05b6b7a4-23d1-402c-88a4-1633e911e55c-kube-api-access-b2272\") pod \"05b6b7a4-23d1-402c-88a4-1633e911e55c\" (UID: \"05b6b7a4-23d1-402c-88a4-1633e911e55c\") " Nov 01 16:51:32.904626 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:32.904592 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05b6b7a4-23d1-402c-88a4-1633e911e55c-maas-api-openshift-ca" pod "05b6b7a4-23d1-402c-88a4-1633e911e55c" (UID: "05b6b7a4-23d1-402c-88a4-1633e911e55c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:51:32.906425 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:32.906398 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05b6b7a4-23d1-402c-88a4-1633e911e55c-maas-api-xks-ca" pod "05b6b7a4-23d1-402c-88a4-1633e911e55c" (UID: "05b6b7a4-23d1-402c-88a4-1633e911e55c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:51:32.906535 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:32.906475 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05b6b7a4-23d1-402c-88a4-1633e911e55c-kube-api-access-b2272" pod "05b6b7a4-23d1-402c-88a4-1633e911e55c" (UID: "05b6b7a4-23d1-402c-88a4-1633e911e55c"). InnerVolumeSpecName "kube-api-access-b2272". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:51:33.005587 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:33.005524 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05b6b7a4-23d1-402c-88a4-1633e911e55c-maas-api-xks-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:51:33.005587 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:33.005549 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2272\" (UniqueName: \"kubernetes.io/projected/05b6b7a4-23d1-402c-88a4-1633e911e55c-kube-api-access-b2272\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:51:33.005587 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:33.005558 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05b6b7a4-23d1-402c-88a4-1633e911e55c-maas-api-openshift-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:51:33.715538 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:33.715510 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-5xppx" Nov 01 16:51:33.715695 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:33.715558 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-5xppx" event={"ID":"05b6b7a4-23d1-402c-88a4-1633e911e55c","Type":"ContainerDied","Data":"3cf323fecd55767b6984bd899ba87a5dcad13d507b2f86398bd7d5f51b0313b5"} Nov 01 16:51:33.715695 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:33.715605 4363 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cf323fecd55767b6984bd899ba87a5dcad13d507b2f86398bd7d5f51b0313b5" Nov 01 16:51:46.951776 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:46.951742 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7qk22"] Nov 01 16:51:46.952242 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:46.952223 4363 cpu_manager.go:414] "containerMap: removing container" podUID="05b6b7a4-23d1-402c-88a4-1633e911e55c" containerName="revoke-keys" Nov 01 16:51:46.952290 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:46.952246 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05b6b7a4-23d1-402c-88a4-1633e911e55c" containerName="revoke-keys" Nov 01 16:51:46.952359 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:46.952348 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="05b6b7a4-23d1-402c-88a4-1633e911e55c" containerName="revoke-keys" Nov 01 16:51:46.955801 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:46.955779 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7qk22" Nov 01 16:51:46.966491 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:46.966379 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 16:51:46.966491 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:46.966483 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 16:51:46.966647 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:46.966612 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vj67v\"" Nov 01 16:51:46.966702 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:46.966678 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:51:46.974475 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:46.974365 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7qk22"] Nov 01 16:51:47.044596 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:47.044560 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0d4136e-09f3-4d13-8652-e41e0d794390-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-7qk22\" (UID: \"e0d4136e-09f3-4d13-8652-e41e0d794390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7qk22" Nov 01 16:51:47.044796 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:47.044628 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbmbf\" (UniqueName: \"kubernetes.io/projected/e0d4136e-09f3-4d13-8652-e41e0d794390-kube-api-access-xbmbf\") pod \"maas-api-revoke-sub-0165d7d2-7qk22\" (UID: \"e0d4136e-09f3-4d13-8652-e41e0d794390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7qk22" Nov 01 16:51:47.044876 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:47.044797 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0d4136e-09f3-4d13-8652-e41e0d794390-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-7qk22\" (UID: \"e0d4136e-09f3-4d13-8652-e41e0d794390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7qk22" Nov 01 16:51:47.145988 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:47.145955 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0d4136e-09f3-4d13-8652-e41e0d794390-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-7qk22\" (UID: \"e0d4136e-09f3-4d13-8652-e41e0d794390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7qk22" Nov 01 16:51:47.146197 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:47.145995 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0d4136e-09f3-4d13-8652-e41e0d794390-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-7qk22\" (UID: \"e0d4136e-09f3-4d13-8652-e41e0d794390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7qk22" Nov 01 16:51:47.146197 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:47.146058 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbmbf\" (UniqueName: \"kubernetes.io/projected/e0d4136e-09f3-4d13-8652-e41e0d794390-kube-api-access-xbmbf\") pod \"maas-api-revoke-sub-0165d7d2-7qk22\" (UID: \"e0d4136e-09f3-4d13-8652-e41e0d794390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7qk22" Nov 01 16:51:47.148295 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:47.148274 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 16:51:47.148389 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:47.148323 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0d4136e-09f3-4d13-8652-e41e0d794390-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-7qk22\" (UID: \"e0d4136e-09f3-4d13-8652-e41e0d794390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7qk22" Nov 01 16:51:47.157527 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:47.157503 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0d4136e-09f3-4d13-8652-e41e0d794390-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-7qk22\" (UID: \"e0d4136e-09f3-4d13-8652-e41e0d794390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7qk22" Nov 01 16:51:47.158131 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:47.158115 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 16:51:47.168356 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:47.168337 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbmbf\" (UniqueName: \"kubernetes.io/projected/e0d4136e-09f3-4d13-8652-e41e0d794390-kube-api-access-xbmbf\") pod \"maas-api-revoke-sub-0165d7d2-7qk22\" (UID: \"e0d4136e-09f3-4d13-8652-e41e0d794390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7qk22" Nov 01 16:51:47.280509 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:47.280437 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7qk22" Nov 01 16:51:47.569395 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:47.569307 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7qk22"] Nov 01 16:51:47.573507 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:51:47.573475 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0d4136e_09f3_4d13_8652_e41e0d794390.slice/crio-9b5dba204d910dd8f4bc4633ccb9ff031ceeeabcd4c7f52966bd01c74d279247 WatchSource:0}: Error finding container 9b5dba204d910dd8f4bc4633ccb9ff031ceeeabcd4c7f52966bd01c74d279247: Status 404 returned error can't find the container with id 9b5dba204d910dd8f4bc4633ccb9ff031ceeeabcd4c7f52966bd01c74d279247 Nov 01 16:51:47.811981 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:47.811947 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7qk22" event={"ID":"e0d4136e-09f3-4d13-8652-e41e0d794390","Type":"ContainerStarted","Data":"9b5dba204d910dd8f4bc4633ccb9ff031ceeeabcd4c7f52966bd01c74d279247"} Nov 01 16:51:48.307855 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:51:48.307771 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:51:48.817840 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:48.817803 4363 generic.go:356] "Generic (PLEG): container finished" podID="e0d4136e-09f3-4d13-8652-e41e0d794390" containerID="8c6d17c9e33984a131cd314eeb4885d905fe9fee8155cff8fafb993202b950d6" exitCode=0 Nov 01 16:51:48.818028 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:48.817889 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7qk22" event={"ID":"e0d4136e-09f3-4d13-8652-e41e0d794390","Type":"ContainerDied","Data":"8c6d17c9e33984a131cd314eeb4885d905fe9fee8155cff8fafb993202b950d6"} Nov 01 16:51:49.957121 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:49.957094 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7qk22" Nov 01 16:51:50.073057 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:50.072969 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0d4136e-09f3-4d13-8652-e41e0d794390-kube-api-access-xbmbf\" (UniqueName: \"kubernetes.io/projected/e0d4136e-09f3-4d13-8652-e41e0d794390-kube-api-access-xbmbf\") pod \"e0d4136e-09f3-4d13-8652-e41e0d794390\" (UID: \"e0d4136e-09f3-4d13-8652-e41e0d794390\") " Nov 01 16:51:50.073194 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:50.073066 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0d4136e-09f3-4d13-8652-e41e0d794390-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0d4136e-09f3-4d13-8652-e41e0d794390-maas-api-openshift-ca\") pod \"e0d4136e-09f3-4d13-8652-e41e0d794390\" (UID: \"e0d4136e-09f3-4d13-8652-e41e0d794390\") " Nov 01 16:51:50.073194 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:50.073111 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0d4136e-09f3-4d13-8652-e41e0d794390-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0d4136e-09f3-4d13-8652-e41e0d794390-maas-api-xks-ca\") pod \"e0d4136e-09f3-4d13-8652-e41e0d794390\" (UID: \"e0d4136e-09f3-4d13-8652-e41e0d794390\") " Nov 01 16:51:50.073453 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:50.073430 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0d4136e-09f3-4d13-8652-e41e0d794390-maas-api-openshift-ca" pod "e0d4136e-09f3-4d13-8652-e41e0d794390" (UID: "e0d4136e-09f3-4d13-8652-e41e0d794390"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:51:50.075191 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:50.075167 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0d4136e-09f3-4d13-8652-e41e0d794390-maas-api-xks-ca" pod "e0d4136e-09f3-4d13-8652-e41e0d794390" (UID: "e0d4136e-09f3-4d13-8652-e41e0d794390"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:51:50.075314 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:50.075233 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0d4136e-09f3-4d13-8652-e41e0d794390-kube-api-access-xbmbf" pod "e0d4136e-09f3-4d13-8652-e41e0d794390" (UID: "e0d4136e-09f3-4d13-8652-e41e0d794390"). InnerVolumeSpecName "kube-api-access-xbmbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:51:50.174555 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:50.174523 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbmbf\" (UniqueName: \"kubernetes.io/projected/e0d4136e-09f3-4d13-8652-e41e0d794390-kube-api-access-xbmbf\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:51:50.174555 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:50.174552 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0d4136e-09f3-4d13-8652-e41e0d794390-maas-api-openshift-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:51:50.174555 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:50.174562 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0d4136e-09f3-4d13-8652-e41e0d794390-maas-api-xks-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:51:50.827894 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:50.827868 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7qk22" Nov 01 16:51:50.828067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:50.827907 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7qk22" event={"ID":"e0d4136e-09f3-4d13-8652-e41e0d794390","Type":"ContainerDied","Data":"9b5dba204d910dd8f4bc4633ccb9ff031ceeeabcd4c7f52966bd01c74d279247"} Nov 01 16:51:50.828067 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:50.827951 4363 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b5dba204d910dd8f4bc4633ccb9ff031ceeeabcd4c7f52966bd01c74d279247" Nov 01 16:51:57.780189 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:57.780151 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zr5hk"] Nov 01 16:51:57.780732 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:57.780540 4363 cpu_manager.go:414] "containerMap: removing container" podUID="e0d4136e-09f3-4d13-8652-e41e0d794390" containerName="revoke-keys" Nov 01 16:51:57.780732 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:57.780552 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0d4136e-09f3-4d13-8652-e41e0d794390" containerName="revoke-keys" Nov 01 16:51:57.780732 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:57.780638 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0d4136e-09f3-4d13-8652-e41e0d794390" containerName="revoke-keys" Nov 01 16:51:57.783824 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:57.783801 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zr5hk" Nov 01 16:51:57.791701 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:57.791681 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vj67v\"" Nov 01 16:51:57.793651 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:57.793629 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 16:51:57.796648 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:57.796616 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zr5hk"] Nov 01 16:51:57.796796 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:57.796776 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 16:51:57.894492 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:57.894461 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:51:57.946702 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:57.946670 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7edc58f-19b3-410d-82d9-55fdbf2fb024-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-zr5hk\" (UID: \"f7edc58f-19b3-410d-82d9-55fdbf2fb024\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zr5hk" Nov 01 16:51:57.946881 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:57.946707 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlq6t\" (UniqueName: \"kubernetes.io/projected/f7edc58f-19b3-410d-82d9-55fdbf2fb024-kube-api-access-wlq6t\") pod \"maas-api-revoke-sub-6b19338f-zr5hk\" (UID: \"f7edc58f-19b3-410d-82d9-55fdbf2fb024\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zr5hk" Nov 01 16:51:57.946881 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:57.946846 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7edc58f-19b3-410d-82d9-55fdbf2fb024-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-zr5hk\" (UID: \"f7edc58f-19b3-410d-82d9-55fdbf2fb024\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zr5hk" Nov 01 16:51:58.048120 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:58.048000 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7edc58f-19b3-410d-82d9-55fdbf2fb024-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-zr5hk\" (UID: \"f7edc58f-19b3-410d-82d9-55fdbf2fb024\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zr5hk" Nov 01 16:51:58.048120 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:58.048067 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlq6t\" (UniqueName: \"kubernetes.io/projected/f7edc58f-19b3-410d-82d9-55fdbf2fb024-kube-api-access-wlq6t\") pod \"maas-api-revoke-sub-6b19338f-zr5hk\" (UID: \"f7edc58f-19b3-410d-82d9-55fdbf2fb024\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zr5hk" Nov 01 16:51:58.048387 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:58.048153 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7edc58f-19b3-410d-82d9-55fdbf2fb024-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-zr5hk\" (UID: \"f7edc58f-19b3-410d-82d9-55fdbf2fb024\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zr5hk" Nov 01 16:51:58.048787 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:58.048762 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7edc58f-19b3-410d-82d9-55fdbf2fb024-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-zr5hk\" (UID: \"f7edc58f-19b3-410d-82d9-55fdbf2fb024\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zr5hk" Nov 01 16:51:58.050497 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:58.050473 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7edc58f-19b3-410d-82d9-55fdbf2fb024-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-zr5hk\" (UID: \"f7edc58f-19b3-410d-82d9-55fdbf2fb024\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zr5hk" Nov 01 16:51:58.058403 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:58.058380 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlq6t\" (UniqueName: \"kubernetes.io/projected/f7edc58f-19b3-410d-82d9-55fdbf2fb024-kube-api-access-wlq6t\") pod \"maas-api-revoke-sub-6b19338f-zr5hk\" (UID: \"f7edc58f-19b3-410d-82d9-55fdbf2fb024\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zr5hk" Nov 01 16:51:58.097303 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:58.097282 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zr5hk" Nov 01 16:51:58.384503 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:58.384477 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zr5hk"] Nov 01 16:51:58.386377 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:51:58.386347 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7edc58f_19b3_410d_82d9_55fdbf2fb024.slice/crio-bd637205306f38843676b83f8c4d28721d51b9cb99904803a67854fb2bbd3915 WatchSource:0}: Error finding container bd637205306f38843676b83f8c4d28721d51b9cb99904803a67854fb2bbd3915: Status 404 returned error can't find the container with id bd637205306f38843676b83f8c4d28721d51b9cb99904803a67854fb2bbd3915 Nov 01 16:51:58.865431 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:58.865393 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zr5hk" event={"ID":"f7edc58f-19b3-410d-82d9-55fdbf2fb024","Type":"ContainerStarted","Data":"29528c8433a410e4c36b72211ee768d38388147bab37f245f46dc3943609856d"} Nov 01 16:51:58.865842 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:58.865454 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zr5hk" event={"ID":"f7edc58f-19b3-410d-82d9-55fdbf2fb024","Type":"ContainerStarted","Data":"bd637205306f38843676b83f8c4d28721d51b9cb99904803a67854fb2bbd3915"} Nov 01 16:51:58.888327 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:58.888247 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zr5hk" podStartSLOduration=1.88822728 podStartE2EDuration="1.88822728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:51:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:51:58.88438745 +0000 UTC m=+3852.216125103" watchObservedRunningTime="2026-11-01 16:51:58.88822728 +0000 UTC m=+3852.219964931" Nov 01 16:51:59.875880 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:59.875846 4363 generic.go:356] "Generic (PLEG): container finished" podID="f7edc58f-19b3-410d-82d9-55fdbf2fb024" containerID="29528c8433a410e4c36b72211ee768d38388147bab37f245f46dc3943609856d" exitCode=0 Nov 01 16:51:59.876372 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:51:59.875947 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zr5hk" event={"ID":"f7edc58f-19b3-410d-82d9-55fdbf2fb024","Type":"ContainerDied","Data":"29528c8433a410e4c36b72211ee768d38388147bab37f245f46dc3943609856d"} Nov 01 16:52:01.018714 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:01.018682 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zr5hk" Nov 01 16:52:01.179964 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:01.179875 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7edc58f-19b3-410d-82d9-55fdbf2fb024-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7edc58f-19b3-410d-82d9-55fdbf2fb024-maas-api-openshift-ca\") pod \"f7edc58f-19b3-410d-82d9-55fdbf2fb024\" (UID: \"f7edc58f-19b3-410d-82d9-55fdbf2fb024\") " Nov 01 16:52:01.179964 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:01.179912 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7edc58f-19b3-410d-82d9-55fdbf2fb024-kube-api-access-wlq6t\" (UniqueName: \"kubernetes.io/projected/f7edc58f-19b3-410d-82d9-55fdbf2fb024-kube-api-access-wlq6t\") pod \"f7edc58f-19b3-410d-82d9-55fdbf2fb024\" (UID: \"f7edc58f-19b3-410d-82d9-55fdbf2fb024\") " Nov 01 16:52:01.180365 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:01.179970 4363 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7edc58f-19b3-410d-82d9-55fdbf2fb024-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7edc58f-19b3-410d-82d9-55fdbf2fb024-maas-api-xks-ca\") pod \"f7edc58f-19b3-410d-82d9-55fdbf2fb024\" (UID: \"f7edc58f-19b3-410d-82d9-55fdbf2fb024\") " Nov 01 16:52:01.180365 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:01.180260 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7edc58f-19b3-410d-82d9-55fdbf2fb024-maas-api-openshift-ca" pod "f7edc58f-19b3-410d-82d9-55fdbf2fb024" (UID: "f7edc58f-19b3-410d-82d9-55fdbf2fb024"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:52:01.182127 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:01.182099 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7edc58f-19b3-410d-82d9-55fdbf2fb024-maas-api-xks-ca" pod "f7edc58f-19b3-410d-82d9-55fdbf2fb024" (UID: "f7edc58f-19b3-410d-82d9-55fdbf2fb024"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:52:01.182222 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:01.182174 4363 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7edc58f-19b3-410d-82d9-55fdbf2fb024-kube-api-access-wlq6t" pod "f7edc58f-19b3-410d-82d9-55fdbf2fb024" (UID: "f7edc58f-19b3-410d-82d9-55fdbf2fb024"). InnerVolumeSpecName "kube-api-access-wlq6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:52:01.281156 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:01.281115 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7edc58f-19b3-410d-82d9-55fdbf2fb024-maas-api-openshift-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:52:01.281156 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:01.281147 4363 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlq6t\" (UniqueName: \"kubernetes.io/projected/f7edc58f-19b3-410d-82d9-55fdbf2fb024-kube-api-access-wlq6t\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:52:01.281156 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:01.281156 4363 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7edc58f-19b3-410d-82d9-55fdbf2fb024-maas-api-xks-ca\") on node \"ip-10-0-128-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:52:01.887511 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:01.887477 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zr5hk" event={"ID":"f7edc58f-19b3-410d-82d9-55fdbf2fb024","Type":"ContainerDied","Data":"bd637205306f38843676b83f8c4d28721d51b9cb99904803a67854fb2bbd3915"} Nov 01 16:52:01.887742 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:01.887530 4363 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd637205306f38843676b83f8c4d28721d51b9cb99904803a67854fb2bbd3915" Nov 01 16:52:01.887742 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:01.887489 4363 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zr5hk" Nov 01 16:52:18.308436 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:52:18.308400 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:52:19.979312 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:19.979272 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-j7vgt_2e85c52e-56da-4d66-913f-12bafed607c8/manager/0.log" Nov 01 16:52:20.439992 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:20.439959 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-646677ff99-v7mvq_372f3c14-1225-451c-ae57-e0b295b31364/manager/0.log" Nov 01 16:52:22.154357 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:22.154326 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-7qk22_e0d4136e-09f3-4d13-8652-e41e0d794390/revoke-keys/0.log" Nov 01 16:52:22.698209 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:22.698174 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-zr5hk_f7edc58f-19b3-410d-82d9-55fdbf2fb024/revoke-keys/0.log" Nov 01 16:52:23.052124 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:23.052051 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-5xppx_05b6b7a4-23d1-402c-88a4-1633e911e55c/revoke-keys/0.log" Nov 01 16:52:25.506816 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:25.506777 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-ztr6x_cfe104c9-1ec9-4536-a292-5fbe4bfb849c/manager/0.log" Nov 01 16:52:26.849875 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:26.849848 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-b9q8c_9208aac6-2f13-4f8b-ac7a-3453908209be/discovery/0.log" Nov 01 16:52:27.020708 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:27.020680 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6759f4f87c-mhx92_19531d99-b1bc-4632-9ced-ec214a278244/kube-auth-proxy/0.log" Nov 01 16:52:27.380888 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:27.380855 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-65xj6_97e26e65-03fd-4fad-99dd-a4d334b883e3/istio-proxy/0.log" Nov 01 16:52:27.570140 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:27.570112 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-8s26k_e0d02b9e-d3ce-4b70-9773-301792717809/istio-proxy/0.log" Nov 01 16:52:28.130356 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:28.130324 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-hmxgm_38489f4e-73ae-416a-a240-3e4942ffea4a/istio-proxy/0.log" Nov 01 16:52:28.345133 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:28.345099 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-lfl9n_0108f2e6-1b6d-4c70-a898-35c26d039d2a/istio-proxy/0.log" Nov 01 16:52:29.607245 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:29.607210 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-l5qnm_dae26a45-6a59-4c32-a565-704e50c5f271/payload-processing/0.log" Nov 01 16:52:30.770826 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:30.770793 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr_305de5fb-52ed-401a-89fe-b7d802472f5c/main/0.log" Nov 01 16:52:31.261304 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:31.261273 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-8h5j2_14cafac8-b130-4780-aa9e-e60215541376/main/0.log" Nov 01 16:52:31.452180 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:31.452151 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k_336c4cd8-6126-4431-930e-57f9b7ccf09d/main/0.log" Nov 01 16:52:31.634000 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:31.633969 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-qncjz_7c0c2696-09a8-4726-98ca-d8331d86e830/main/0.log" Nov 01 16:52:31.817398 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:31.817370 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-frphn_fa3886b7-e5e1-4f64-93d4-a097f446a835/main/0.log" Nov 01 16:52:42.074746 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:42.074713 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-j7vgt_2e85c52e-56da-4d66-913f-12bafed607c8/manager/0.log" Nov 01 16:52:42.484089 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:42.484053 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-646677ff99-v7mvq_372f3c14-1225-451c-ae57-e0b295b31364/manager/0.log" Nov 01 16:52:43.962805 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:43.962770 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-7qk22_e0d4136e-09f3-4d13-8652-e41e0d794390/revoke-keys/0.log" Nov 01 16:52:44.472410 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:44.472383 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-zr5hk_f7edc58f-19b3-410d-82d9-55fdbf2fb024/revoke-keys/0.log" Nov 01 16:52:44.818115 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:44.818007 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-5xppx_05b6b7a4-23d1-402c-88a4-1633e911e55c/revoke-keys/0.log" Nov 01 16:52:46.802614 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:46.802583 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-ztr6x_cfe104c9-1ec9-4536-a292-5fbe4bfb849c/manager/0.log" Nov 01 16:52:47.931980 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:47.931938 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-b9q8c_9208aac6-2f13-4f8b-ac7a-3453908209be/discovery/0.log" Nov 01 16:52:48.092199 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:48.092170 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtwhw_8254aafa-dd5b-4150-87e4-d165dab6bcb5/ovn-acl-logging/0.log" Nov 01 16:52:48.094106 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:48.094081 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jtwhw_8254aafa-dd5b-4150-87e4-d165dab6bcb5/ovn-acl-logging/0.log" Nov 01 16:52:48.109918 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:48.109894 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6759f4f87c-mhx92_19531d99-b1bc-4632-9ced-ec214a278244/kube-auth-proxy/0.log" Nov 01 16:52:48.309407 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:52:48.309323 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:52:48.465544 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:48.465507 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-65xj6_97e26e65-03fd-4fad-99dd-a4d334b883e3/istio-proxy/0.log" Nov 01 16:52:48.644726 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:48.644698 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-8s26k_e0d02b9e-d3ce-4b70-9773-301792717809/istio-proxy/0.log" Nov 01 16:52:49.180861 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:49.180824 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-hmxgm_38489f4e-73ae-416a-a240-3e4942ffea4a/istio-proxy/0.log" Nov 01 16:52:49.362974 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:49.362934 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-lfl9n_0108f2e6-1b6d-4c70-a898-35c26d039d2a/istio-proxy/0.log" Nov 01 16:52:50.622880 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:50.622838 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-l5qnm_dae26a45-6a59-4c32-a565-704e50c5f271/payload-processing/0.log" Nov 01 16:52:51.312542 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:51.312507 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-7nrxr_305de5fb-52ed-401a-89fe-b7d802472f5c/main/0.log" Nov 01 16:52:51.831132 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:51.831085 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-8h5j2_14cafac8-b130-4780-aa9e-e60215541376/main/0.log" Nov 01 16:52:52.013359 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:52.013331 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rg65k_336c4cd8-6126-4431-930e-57f9b7ccf09d/main/0.log" Nov 01 16:52:52.191208 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:52.191179 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-qncjz_7c0c2696-09a8-4726-98ca-d8331d86e830/main/0.log" Nov 01 16:52:52.371470 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:52:52.371439 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-frphn_fa3886b7-e5e1-4f64-93d4-a097f446a835/main/0.log" Nov 01 16:53:18.307910 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:53:18.307822 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:53:30.137093 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:53:30.137044 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-646677ff99-v7mvq_372f3c14-1225-451c-ae57-e0b295b31364/manager/0.log" Nov 01 16:53:30.392316 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:53:30.392224 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-646677ff99-v7mvq_372f3c14-1225-451c-ae57-e0b295b31364/manager/0.log" Nov 01 16:53:31.026624 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:53:31.026595 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-l5qnm_dae26a45-6a59-4c32-a565-704e50c5f271/payload-processing/0.log" Nov 01 16:53:31.201059 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:53:31.201032 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-l5qnm_dae26a45-6a59-4c32-a565-704e50c5f271/payload-processing/0.log" Nov 01 16:53:48.308502 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:53:48.308459 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:54:17.237272 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:17.237232 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-646677ff99-v7mvq_372f3c14-1225-451c-ae57-e0b295b31364/manager/0.log" Nov 01 16:54:17.481541 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:17.481506 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-646677ff99-v7mvq_372f3c14-1225-451c-ae57-e0b295b31364/manager/0.log" Nov 01 16:54:18.116593 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:18.116560 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-l5qnm_dae26a45-6a59-4c32-a565-704e50c5f271/payload-processing/0.log" Nov 01 16:54:18.284458 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:18.284423 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-l5qnm_dae26a45-6a59-4c32-a565-704e50c5f271/payload-processing/0.log" Nov 01 16:54:18.308244 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:54:18.308209 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor" Nov 01 16:54:22.183533 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:22.183501 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ckz9h_c7838987-12dc-47bc-9b39-a87ffd11ddaf/global-pull-secret-syncer/0.log" Nov 01 16:54:22.463690 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:22.463611 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-d7slg_f13a984b-8706-43d5-859f-abbfcaf58abf/konnectivity-agent/0.log" Nov 01 16:54:22.552293 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:22.552262 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-238.us-east-2.compute.internal_ae861e52d07eb0e1937b2b22576e5cfd/haproxy/0.log" Nov 01 16:54:31.402248 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:31.402157 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-ztr6x_cfe104c9-1ec9-4536-a292-5fbe4bfb849c/manager/0.log" Nov 01 16:54:32.679493 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:32.679460 4363 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-c79r4/perf-node-gather-daemonset-cp4nv"] Nov 01 16:54:32.680048 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:32.679974 4363 cpu_manager.go:414] "containerMap: removing container" podUID="f7edc58f-19b3-410d-82d9-55fdbf2fb024" containerName="revoke-keys" Nov 01 16:54:32.680275 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:32.680236 4363 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7edc58f-19b3-410d-82d9-55fdbf2fb024" containerName="revoke-keys" Nov 01 16:54:32.680469 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:32.680355 4363 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7edc58f-19b3-410d-82d9-55fdbf2fb024" containerName="revoke-keys" Nov 01 16:54:32.683629 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:32.683601 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-cp4nv" Nov 01 16:54:32.694094 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:32.694068 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c79r4/perf-node-gather-daemonset-cp4nv"] Nov 01 16:54:32.694239 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:32.694184 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c79r4\"/\"openshift-service-ca.crt\"" Nov 01 16:54:32.695139 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:32.695115 4363 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-c79r4\"/\"default-dockercfg-fqmbm\"" Nov 01 16:54:32.771383 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:32.771350 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p47rw\" (UniqueName: \"kubernetes.io/projected/aa04830b-7b55-48c2-911f-9276d29440fe-kube-api-access-p47rw\") pod \"perf-node-gather-daemonset-cp4nv\" (UID: \"aa04830b-7b55-48c2-911f-9276d29440fe\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-cp4nv" Nov 01 16:54:32.771543 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:32.771391 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aa04830b-7b55-48c2-911f-9276d29440fe-podres\") pod \"perf-node-gather-daemonset-cp4nv\" (UID: \"aa04830b-7b55-48c2-911f-9276d29440fe\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-cp4nv" Nov 01 16:54:32.771543 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:32.771456 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa04830b-7b55-48c2-911f-9276d29440fe-sys\") pod \"perf-node-gather-daemonset-cp4nv\" (UID: \"aa04830b-7b55-48c2-911f-9276d29440fe\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-cp4nv" Nov 01 16:54:32.771543 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:32.771517 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aa04830b-7b55-48c2-911f-9276d29440fe-proc\") pod \"perf-node-gather-daemonset-cp4nv\" (UID: \"aa04830b-7b55-48c2-911f-9276d29440fe\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-cp4nv" Nov 01 16:54:32.771543 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:32.771535 4363 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aa04830b-7b55-48c2-911f-9276d29440fe-lib-modules\") pod \"perf-node-gather-daemonset-cp4nv\" (UID: \"aa04830b-7b55-48c2-911f-9276d29440fe\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-cp4nv" Nov 01 16:54:32.790292 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:32.790261 4363 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c79r4\"/\"kube-root-ca.crt\"" Nov 01 16:54:32.818201 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:32.818172 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-8b544_7adfe5b2-8d23-4150-a89d-d8725623a6c0/kube-state-metrics/0.log" Nov 01 16:54:32.837476 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:32.837445 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-8b544_7adfe5b2-8d23-4150-a89d-d8725623a6c0/kube-rbac-proxy-main/0.log" Nov 01 16:54:32.860518 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:32.860498 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-8b544_7adfe5b2-8d23-4150-a89d-d8725623a6c0/kube-rbac-proxy-self/0.log" Nov 01 16:54:32.871963 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:32.871941 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p47rw\" (UniqueName: \"kubernetes.io/projected/aa04830b-7b55-48c2-911f-9276d29440fe-kube-api-access-p47rw\") pod \"perf-node-gather-daemonset-cp4nv\" (UID: \"aa04830b-7b55-48c2-911f-9276d29440fe\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-cp4nv" Nov 01 16:54:32.872087 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:32.871981 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aa04830b-7b55-48c2-911f-9276d29440fe-podres\") pod \"perf-node-gather-daemonset-cp4nv\" (UID: \"aa04830b-7b55-48c2-911f-9276d29440fe\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-cp4nv" Nov 01 16:54:32.872087 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:32.872043 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa04830b-7b55-48c2-911f-9276d29440fe-sys\") pod \"perf-node-gather-daemonset-cp4nv\" (UID: \"aa04830b-7b55-48c2-911f-9276d29440fe\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-cp4nv" Nov 01 16:54:32.872087 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:32.872070 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aa04830b-7b55-48c2-911f-9276d29440fe-proc\") pod \"perf-node-gather-daemonset-cp4nv\" (UID: \"aa04830b-7b55-48c2-911f-9276d29440fe\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-cp4nv" Nov 01 16:54:32.872255 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:32.872094 4363 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aa04830b-7b55-48c2-911f-9276d29440fe-lib-modules\") pod \"perf-node-gather-daemonset-cp4nv\" (UID: \"aa04830b-7b55-48c2-911f-9276d29440fe\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-cp4nv" Nov 01 16:54:32.872255 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:32.872153 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa04830b-7b55-48c2-911f-9276d29440fe-sys\") pod \"perf-node-gather-daemonset-cp4nv\" (UID: \"aa04830b-7b55-48c2-911f-9276d29440fe\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-cp4nv" Nov 01 16:54:32.872255 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:32.872166 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aa04830b-7b55-48c2-911f-9276d29440fe-podres\") pod \"perf-node-gather-daemonset-cp4nv\" (UID: \"aa04830b-7b55-48c2-911f-9276d29440fe\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-cp4nv" Nov 01 16:54:32.872255 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:32.872177 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aa04830b-7b55-48c2-911f-9276d29440fe-proc\") pod \"perf-node-gather-daemonset-cp4nv\" (UID: \"aa04830b-7b55-48c2-911f-9276d29440fe\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-cp4nv" Nov 01 16:54:32.872255 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:32.872207 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aa04830b-7b55-48c2-911f-9276d29440fe-lib-modules\") pod \"perf-node-gather-daemonset-cp4nv\" (UID: \"aa04830b-7b55-48c2-911f-9276d29440fe\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-cp4nv" Nov 01 16:54:32.879451 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:32.879427 4363 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p47rw\" (UniqueName: \"kubernetes.io/projected/aa04830b-7b55-48c2-911f-9276d29440fe-kube-api-access-p47rw\") pod \"perf-node-gather-daemonset-cp4nv\" (UID: \"aa04830b-7b55-48c2-911f-9276d29440fe\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-cp4nv" Nov 01 16:54:33.001101 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:33.000999 4363 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-cp4nv" Nov 01 16:54:33.042945 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:33.042908 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pwkct_9c69f277-d3e5-457d-a4f9-328d96b681d3/node-exporter/0.log" Nov 01 16:54:33.067628 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:33.067606 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pwkct_9c69f277-d3e5-457d-a4f9-328d96b681d3/kube-rbac-proxy/0.log" Nov 01 16:54:33.093417 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:33.093393 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pwkct_9c69f277-d3e5-457d-a4f9-328d96b681d3/init-textfile/0.log" Nov 01 16:54:33.205287 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:33.205258 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-jqtqm_457afc43-0d86-4348-9348-29b4fa1931af/kube-rbac-proxy-main/0.log" Nov 01 16:54:33.230386 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:33.230358 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-jqtqm_457afc43-0d86-4348-9348-29b4fa1931af/kube-rbac-proxy-self/0.log" Nov 01 16:54:33.256387 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:33.256313 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-jqtqm_457afc43-0d86-4348-9348-29b4fa1931af/openshift-state-metrics/0.log" Nov 01 16:54:33.278400 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:33.278312 4363 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c79r4/perf-node-gather-daemonset-cp4nv"] Nov 01 16:54:33.280836 ip-10-0-128-238 kubenswrapper[4363]: W1101 16:54:33.280807 4363 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podaa04830b_7b55_48c2_911f_9276d29440fe.slice/crio-c3f0a78fad2ae90323f07f67b52ad6f3c8a586074a5e047de6826781e38e8d1c WatchSource:0}: Error finding container c3f0a78fad2ae90323f07f67b52ad6f3c8a586074a5e047de6826781e38e8d1c: Status 404 returned error can't find the container with id c3f0a78fad2ae90323f07f67b52ad6f3c8a586074a5e047de6826781e38e8d1c Nov 01 16:54:33.500967 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:33.500936 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-qt7lp_cc6b2feb-cc68-454c-9805-6db660e294b8/prometheus-operator/0.log" Nov 01 16:54:33.518988 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:33.518916 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-qt7lp_cc6b2feb-cc68-454c-9805-6db660e294b8/kube-rbac-proxy/0.log" Nov 01 16:54:33.549082 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:33.549053 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-lsfq4_32a5cc8c-047c-4616-a273-2bd712bef330/prometheus-operator-admission-webhook/0.log" Nov 01 16:54:33.636561 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:33.636530 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-cp4nv" event={"ID":"aa04830b-7b55-48c2-911f-9276d29440fe","Type":"ContainerStarted","Data":"e40f0d59fbc5e5642f92250da5078484a7917374b34554ddc2bf81dd0d921410"} Nov 01 16:54:33.636763 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:33.636590 4363 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-cp4nv" event={"ID":"aa04830b-7b55-48c2-911f-9276d29440fe","Type":"ContainerStarted","Data":"c3f0a78fad2ae90323f07f67b52ad6f3c8a586074a5e047de6826781e38e8d1c"} Nov 01 16:54:33.636763 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:33.636626 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-cp4nv" Nov 01 16:54:33.654026 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:33.653955 4363 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-cp4nv" podStartSLOduration=1.653936238 podStartE2EDuration="1.653936238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 16:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:54:33.651533336 +0000 UTC m=+4006.983270987" watchObservedRunningTime="2026-11-01 16:54:33.653936238 +0000 UTC m=+4006.985673890" Nov 01 16:54:33.677120 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:33.677091 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-777b758c86-h4jsj_88d3d141-d4fa-4033-a3cb-372752fa2c67/thanos-query/0.log" Nov 01 16:54:33.700832 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:33.700808 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-777b758c86-h4jsj_88d3d141-d4fa-4033-a3cb-372752fa2c67/kube-rbac-proxy-web/0.log" Nov 01 16:54:33.724291 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:33.724252 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-777b758c86-h4jsj_88d3d141-d4fa-4033-a3cb-372752fa2c67/kube-rbac-proxy/0.log" Nov 01 16:54:33.748542 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:33.748519 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-777b758c86-h4jsj_88d3d141-d4fa-4033-a3cb-372752fa2c67/prom-label-proxy/0.log" Nov 01 16:54:33.772498 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:33.772423 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-777b758c86-h4jsj_88d3d141-d4fa-4033-a3cb-372752fa2c67/kube-rbac-proxy-rules/0.log" Nov 01 16:54:33.802614 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:33.802584 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-777b758c86-h4jsj_88d3d141-d4fa-4033-a3cb-372752fa2c67/kube-rbac-proxy-metrics/0.log" Nov 01 16:54:36.102759 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:36.102728 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-67647cbf64-wn65v_4fdc9534-ef36-455b-9ea0-ea4bb28d28e6/console/0.log" Nov 01 16:54:36.156248 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:36.156199 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-8vq2s_455d1f7d-3a33-43de-8561-6439dcab4589/download-server/0.log" Nov 01 16:54:38.060118 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:38.060088 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d9spc_c51a9296-d37f-4161-936d-c1de0a2c2c82/dns/0.log" Nov 01 16:54:38.081081 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:38.081056 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d9spc_c51a9296-d37f-4161-936d-c1de0a2c2c82/kube-rbac-proxy/0.log" Nov 01 16:54:38.241871 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:38.241841 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hkkmh_667a7db5-cf73-424c-9d18-53bfbc5f65db/dns-node-resolver/0.log" Nov 01 16:54:38.905285 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:38.905257 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-85755df759-sswt8_ca3f5966-c817-45e8-b1f9-d5a8815ae2e1/registry/0.log" Nov 01 16:54:38.931261 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:38.931227 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-44klx_56e8fbab-06d3-4421-9995-822247d94d90/node-ca/0.log" Nov 01 16:54:39.651400 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:39.651371 4363 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-cp4nv" Nov 01 16:54:43.123131 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:43.123089 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-b9q8c_9208aac6-2f13-4f8b-ac7a-3453908209be/discovery/0.log" Nov 01 16:54:43.682361 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:43.682328 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6759f4f87c-mhx92_19531d99-b1bc-4632-9ced-ec214a278244/kube-auth-proxy/0.log" Nov 01 16:54:43.750698 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:43.750666 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-65xj6_97e26e65-03fd-4fad-99dd-a4d334b883e3/istio-proxy/0.log" Nov 01 16:54:43.786867 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:43.786834 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-8s26k_e0d02b9e-d3ce-4b70-9773-301792717809/istio-proxy/0.log" Nov 01 16:54:43.891452 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:43.891416 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-hmxgm_38489f4e-73ae-416a-a240-3e4942ffea4a/istio-proxy/0.log" Nov 01 16:54:43.927175 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:43.927147 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-lfl9n_0108f2e6-1b6d-4c70-a898-35c26d039d2a/istio-proxy/0.log" Nov 01 16:54:44.208981 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:44.208950 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-l5qnm_dae26a45-6a59-4c32-a565-704e50c5f271/payload-processing/0.log" Nov 01 16:54:46.292381 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:46.292347 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-b9q8c_9208aac6-2f13-4f8b-ac7a-3453908209be/discovery/0.log" Nov 01 16:54:46.303606 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:46.303579 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6759f4f87c-mhx92_19531d99-b1bc-4632-9ced-ec214a278244/kube-auth-proxy/0.log" Nov 01 16:54:46.379132 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:46.379100 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-65xj6_97e26e65-03fd-4fad-99dd-a4d334b883e3/istio-proxy/0.log" Nov 01 16:54:46.401680 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:46.401657 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-8s26k_e0d02b9e-d3ce-4b70-9773-301792717809/istio-proxy/0.log" Nov 01 16:54:46.455963 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:46.455939 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-hmxgm_38489f4e-73ae-416a-a240-3e4942ffea4a/istio-proxy/0.log" Nov 01 16:54:46.475835 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:46.475804 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-lfl9n_0108f2e6-1b6d-4c70-a898-35c26d039d2a/istio-proxy/0.log" Nov 01 16:54:46.605407 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:46.605383 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-l5qnm_dae26a45-6a59-4c32-a565-704e50c5f271/payload-processing/0.log" Nov 01 16:54:46.627112 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:46.627089 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sjjtf_2d417d8c-560b-4038-9971-76592795cc92/serve-healthcheck-canary/0.log" Nov 01 16:54:47.390699 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:47.390665 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vw5gf_d0820c10-7071-4f70-82e6-fdbf2c0d6c76/kube-rbac-proxy/0.log" Nov 01 16:54:47.414122 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:47.414097 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vw5gf_d0820c10-7071-4f70-82e6-fdbf2c0d6c76/exporter/0.log" Nov 01 16:54:47.438748 ip-10-0-128-238 kubenswrapper[4363]: I1101 16:54:47.438720 4363 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vw5gf_d0820c10-7071-4f70-82e6-fdbf2c0d6c76/extractor/0.log" Nov 01 16:54:48.308029 ip-10-0-128-238 kubenswrapper[4363]: E1101 16:54:48.307983 4363 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vw5gf" containerName="extractor"